Re: [Gluster-devel] Adding xxhash to gluster code base
On Tue, Jun 27, 2017 at 8:46 PM, Niels de Vos wrote: > On Tue, Jun 27, 2017 at 08:09:25AM -0400, Kaleb S. KEITHLEY wrote: > > > > xxhash doesn't seem to change much. Last update to the non-test code was > six > > months ago. > > > > bundling giant (for some definition of giant) packages/projects would be > > bad. bundling two (three if you count the test) C files doesn't seem too > bad > > when you consider that there are already three or four packages in fedora > > (perl, python, R-digest, ghc (gnu haskell) that have implementations of > > xxhash or murmur but didn't bother to package a C implementation and use > it. > > I prefer to have as little maintenance components in the Gluster sources > as we can. The maintenance burdon is already very high. The number of > changes to xxhash seem limited, but we still need someone to track and > pay attention to them. > I agree that someone should maintain it, and we should add it to MAINTAINERS file (or some other place, where we are tracking the dependencies). For now, Kotresh will be looking into keeping these changes up-to-date with upstream xxhash project, along with me. > > > I'd be for packaging it in Fedora rather than bundling it in gluster. But > > then we get to "carry" it in rhgs as we do with userspace-rcu. > > We should descide what the most maintainable solution is. Having package > maintainers with the explicit task to keep xxhash updated and current is > apealing to me. Merging (even small) projects into the Gluster codebase > will add more maintenance need to the project members. Therefor I have a > strong preference to use xxhash (or an other library) that is provided > by distributions. The more common the library is, the better it will be > maintained without our (Gluster Community's) help. > > While this is desirable, we didn't see any library available for xxhash ( http://cyan4973.github.io/xxHash/) in our distro. I would recommend taking these patches with TODO to use library in future when its available, and continue to have xxhash in 'contrib/'. It is not new for us to take code from different libraries and use it for our need and maintain only that part (eg. libfuse). Lets treat this as similar setup. Regards, Amar > Niels > > > > On 06/27/2017 04:08 AM, Niels de Vos wrote: > > > On Tue, Jun 27, 2017 at 12:25:11PM +0530, Kotresh Hiremath Ravishankar > wrote: > > > > Hi, > > > > > > > > We were looking for faster non-cryptographic hash to be used for the > > > > gfid2path infra [1] > > > > The initial testing was done with md5 128bit checksum which was a > slow, > > > > cryptographic hash > > > > and using it makes software not complaint to FIPS [2] > > > > > > > > On searching online a bit we found out xxhash [3] seems to be faster > from > > > > the results of > > > > benchmark tests shared and lot of projects use it. So we have > decided to us > > > > xxHash > > > > and added following files to gluster code base with the patch [4] > > > > > > > > BSD 2-Clause License: > > > > contrib/xxhash/xxhash.c > > > > contrib/xxhash/xxhash.h > > > > > > > > GPL v2 License: > > > > tests/utils/xxhsum.c > > > > > > > > NOTE: We have ignored the code guideline check for these files as > > > > maintaining it > > > > further becomes difficult. > > > > > > > > Please comment on the same if there are any issues around it. > > > > > > How performance critical is the hashing for gfid2path? > > > > > > What is the plan to keep these files maintained? At minimal we need to > > > add these files to MAINTAINERS and the maintainers need to cherry-pick > > > updates and bugfixes from the original project. The few patches a year > > > makes this a recurring task that should not be forgoten. It would be > > > much better to use this as an external library that is provided by the > > > distributions. We already rely on OpenSSL, does this library not > provide > > > an alternative 'FIPS approved' hashing that performs reasonably well? > > > > > > Some distributions are very strict on bundling external projects, and > we > > > need to inform the packagers about the additions so that they can > handle > > > it correctly. Adding an external project to contrib/ should be > mentioned > > > in the release notes at the very least. > > > > > > Note that none of the symbols of any public functions in Gluster may > > > collide with functions in standard distribution libraries. This causes > > > for regular problems with gfapi applications. All exposed symbols that > > > get imported in contrib/ should have a gf_ prefix. > > > > > > Thanks, > > > Niels > > > > > > > > > > > > > > [1] Issue: https://github.com/gluster/glusterfs/issues/139 > > > > [2] https://en.wikipedia.org/wiki/Federal_Information_ > Processing_Standards > > > > [3] http://cyan4973.github.io/xxHash/ > > > > [4] https://review.gluster.org/#/c/17488/10 > > > > > > > > > > > > > > > > -- > > > > Thanks and Regards, > > > > Kotresh H R and Aravinda VK > > > -- A
Re: [Gluster-devel] Scripts to help RCA quota accounting issues
On Thu, Jun 22, 2017 at 4:56 PM, Sanoj Unnikrishnan
wrote:
> I have written some scripts that may help RCA quota accounting related
> issue in future.
> Please use them when necessary.
>
> 1) Below script will compare accounting done by 'du' with that done by
> quota and compare them.
> Requires input : mountpoint and volname.
> ouput: /tmp/gluster_files.tar
>
> cd
> du -h | head -n -1 | tr -d '.' |awk '{ for (i = 2; i <= NF; i++) {
> printf("%s ", $i);} print "" }' > /tmp/gluster_1
> cat /tmp/gluster_1 | sed 's/ $//' | sed 's/ /\\ /g' | sed 's/(/\\(/g' | sed
> 's/)/\\)/g' |xargs gluster v quota list > /tmp/gluster_2
> du -h | head -n -1 |awk '{ for (i = 2; i <= NF; i++) { printf("%s %s", $i,
> $1);} print "" }' | tr -d '.' > /tmp/gluster_3
> cat /tmp/gluster_2 /tmp/gluster_3 | sort > /tmp/gluster_4
> find . -type d > /tmp/gluster_5
> tar -cvf /tmp/gluster_files.tar /tmp/gluster_*
>
>
> Can you send this as patch ? Keep the script in
'extras/quota/good-script-name.sh'
> 2) To recusively get the quota xattr on a FS tree use:
> https://gist.github.com/sanoj-unnikrishnan/740b177cbe9c3277123cf08d875a6b
> f8
>
>
Cool, even this can get into project if you wish. A blog post on how using
this helped maintaining Quota feature would help other admins IMO.
Regards,
Amar
> Regards,
> Sanoj
>
>
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>
--
Amar Tumballi (amarts)
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Adding xxhash to gluster code base
On Wed, Jun 28, 2017 at 12:51:07PM +0530, Amar Tumballi wrote: > On Tue, Jun 27, 2017 at 8:46 PM, Niels de Vos wrote: > > > On Tue, Jun 27, 2017 at 08:09:25AM -0400, Kaleb S. KEITHLEY wrote: > > > > > > xxhash doesn't seem to change much. Last update to the non-test code was > > six > > > months ago. > > > > > > bundling giant (for some definition of giant) packages/projects would be > > > bad. bundling two (three if you count the test) C files doesn't seem too > > bad > > > when you consider that there are already three or four packages in fedora > > > (perl, python, R-digest, ghc (gnu haskell) that have implementations of > > > xxhash or murmur but didn't bother to package a C implementation and use > > it. > > > > I prefer to have as little maintenance components in the Gluster sources > > as we can. The maintenance burdon is already very high. The number of > > changes to xxhash seem limited, but we still need someone to track and > > pay attention to them. > > > > I agree that someone should maintain it, and we should add it to > MAINTAINERS file > (or some other place, where we are tracking the dependencies). > > For now, Kotresh will be looking into keeping these changes up-to-date with > upstream xxhash project, along with me. Kotresh as maintainer/owner, and Aravinda as peer? > > > I'd be for packaging it in Fedora rather than bundling it in gluster. But > > > then we get to "carry" it in rhgs as we do with userspace-rcu. > > > > We should descide what the most maintainable solution is. Having package > > maintainers with the explicit task to keep xxhash updated and current is > > apealing to me. Merging (even small) projects into the Gluster codebase > > will add more maintenance need to the project members. Therefor I have a > > strong preference to use xxhash (or an other library) that is provided > > by distributions. The more common the library is, the better it will be > > maintained without our (Gluster Community's) help. > > > > > While this is desirable, we didn't see any library available for xxhash ( > http://cyan4973.github.io/xxHash/) in our distro. > > I would recommend taking these patches with TODO to use library in future > when its available, and continue to have xxhash in 'contrib/'. It is not > new for us to take code from different libraries and use it for our need > and maintain only that part (eg. libfuse). Lets treat this as similar setup. Yes, if there is no suitable alternative available in the majority of distributions, this is the only sensible approach. Much of the code in contrib/ is not maintained at all. We should prevent this from happening with new code and assigning an owner/maintainer and peer(s) just like for other components is a must. Thanks, Niels > > Regards, > Amar > > > > > > > Niels > > > > > > > On 06/27/2017 04:08 AM, Niels de Vos wrote: > > > > On Tue, Jun 27, 2017 at 12:25:11PM +0530, Kotresh Hiremath Ravishankar > > wrote: > > > > > Hi, > > > > > > > > > > We were looking for faster non-cryptographic hash to be used for the > > > > > gfid2path infra [1] > > > > > The initial testing was done with md5 128bit checksum which was a > > slow, > > > > > cryptographic hash > > > > > and using it makes software not complaint to FIPS [2] > > > > > > > > > > On searching online a bit we found out xxhash [3] seems to be faster > > from > > > > > the results of > > > > > benchmark tests shared and lot of projects use it. So we have > > decided to us > > > > > xxHash > > > > > and added following files to gluster code base with the patch [4] > > > > > > > > > > BSD 2-Clause License: > > > > > contrib/xxhash/xxhash.c > > > > > contrib/xxhash/xxhash.h > > > > > > > > > > GPL v2 License: > > > > > tests/utils/xxhsum.c > > > > > > > > > > NOTE: We have ignored the code guideline check for these files as > > > > > maintaining it > > > > > further becomes difficult. > > > > > > > > > > Please comment on the same if there are any issues around it. > > > > > > > > How performance critical is the hashing for gfid2path? > > > > > > > > What is the plan to keep these files maintained? At minimal we need to > > > > add these files to MAINTAINERS and the maintainers need to cherry-pick > > > > updates and bugfixes from the original project. The few patches a year > > > > makes this a recurring task that should not be forgoten. It would be > > > > much better to use this as an external library that is provided by the > > > > distributions. We already rely on OpenSSL, does this library not > > provide > > > > an alternative 'FIPS approved' hashing that performs reasonably well? > > > > > > > > Some distributions are very strict on bundling external projects, and > > we > > > > need to inform the packagers about the additions so that they can > > handle > > > > it correctly. Adding an external project to contrib/ should be > > mentioned > > > > in the release notes at the very least. > > > > > > > > Note that none of the symbol
Re: [Gluster-devel] Adding xxhash to gluster code base
That sounds good to me. I will send it as a separate patch then. And I can maintain it. No issues. Thanks and Regards, Kotresh H R On Wed, Jun 28, 2017 at 1:02 PM, Niels de Vos wrote: > On Wed, Jun 28, 2017 at 12:51:07PM +0530, Amar Tumballi wrote: > > On Tue, Jun 27, 2017 at 8:46 PM, Niels de Vos wrote: > > > > > On Tue, Jun 27, 2017 at 08:09:25AM -0400, Kaleb S. KEITHLEY wrote: > > > > > > > > xxhash doesn't seem to change much. Last update to the non-test code > was > > > six > > > > months ago. > > > > > > > > bundling giant (for some definition of giant) packages/projects > would be > > > > bad. bundling two (three if you count the test) C files doesn't seem > too > > > bad > > > > when you consider that there are already three or four packages in > fedora > > > > (perl, python, R-digest, ghc (gnu haskell) that have implementations > of > > > > xxhash or murmur but didn't bother to package a C implementation and > use > > > it. > > > > > > I prefer to have as little maintenance components in the Gluster > sources > > > as we can. The maintenance burdon is already very high. The number of > > > changes to xxhash seem limited, but we still need someone to track and > > > pay attention to them. > > > > > > > I agree that someone should maintain it, and we should add it to > > MAINTAINERS file > > (or some other place, where we are tracking the dependencies). > > > > For now, Kotresh will be looking into keeping these changes up-to-date > with > > upstream xxhash project, along with me. > > Kotresh as maintainer/owner, and Aravinda as peer? > > > > > I'd be for packaging it in Fedora rather than bundling it in > gluster. But > > > > then we get to "carry" it in rhgs as we do with userspace-rcu. > > > > > > We should descide what the most maintainable solution is. Having > package > > > maintainers with the explicit task to keep xxhash updated and current > is > > > apealing to me. Merging (even small) projects into the Gluster codebase > > > will add more maintenance need to the project members. Therefor I have > a > > > strong preference to use xxhash (or an other library) that is provided > > > by distributions. The more common the library is, the better it will be > > > maintained without our (Gluster Community's) help. > > > > > > > > While this is desirable, we didn't see any library available for xxhash ( > > http://cyan4973.github.io/xxHash/) in our distro. > > > > I would recommend taking these patches with TODO to use library in future > > when its available, and continue to have xxhash in 'contrib/'. It is not > > new for us to take code from different libraries and use it for our need > > and maintain only that part (eg. libfuse). Lets treat this as similar > setup. > > Yes, if there is no suitable alternative available in the majority of > distributions, this is the only sensible approach. Much of the code in > contrib/ is not maintained at all. We should prevent this from happening > with new code and assigning an owner/maintainer and peer(s) just like > for other components is a must. > > Thanks, > Niels > > > > > > Regards, > > Amar > > > > > > > > > > > > > Niels > > > > > > > > > > On 06/27/2017 04:08 AM, Niels de Vos wrote: > > > > > On Tue, Jun 27, 2017 at 12:25:11PM +0530, Kotresh Hiremath > Ravishankar > > > wrote: > > > > > > Hi, > > > > > > > > > > > > We were looking for faster non-cryptographic hash to be used for > the > > > > > > gfid2path infra [1] > > > > > > The initial testing was done with md5 128bit checksum which was a > > > slow, > > > > > > cryptographic hash > > > > > > and using it makes software not complaint to FIPS [2] > > > > > > > > > > > > On searching online a bit we found out xxhash [3] seems to be > faster > > > from > > > > > > the results of > > > > > > benchmark tests shared and lot of projects use it. So we have > > > decided to us > > > > > > xxHash > > > > > > and added following files to gluster code base with the patch [4] > > > > > > > > > > > > BSD 2-Clause License: > > > > > > contrib/xxhash/xxhash.c > > > > > > contrib/xxhash/xxhash.h > > > > > > > > > > > > GPL v2 License: > > > > > > tests/utils/xxhsum.c > > > > > > > > > > > > NOTE: We have ignored the code guideline check for these files as > > > > > > maintaining it > > > > > > further becomes difficult. > > > > > > > > > > > > Please comment on the same if there are any issues around it. > > > > > > > > > > How performance critical is the hashing for gfid2path? > > > > > > > > > > What is the plan to keep these files maintained? At minimal we > need to > > > > > add these files to MAINTAINERS and the maintainers need to > cherry-pick > > > > > updates and bugfixes from the original project. The few patches a > year > > > > > makes this a recurring task that should not be forgoten. It would > be > > > > > much better to use this as an external library that is provided by > the > > > > > distributions. We already rely on OpenSSL, does this library not >
Re: [Gluster-devel] glusterfsd crash when using quota without io-threads
Hi Kinglong,
I opened https://bugzilla.redhat.com/show_bug.cgi?id=1465861 to track this.
It seems to be an important issue that went undetected.
Thanks for catching this!!
In my attempts to repro this , I found that on each run some random
structures where getting corrupted and running into segfault.
In order to assert that the stack was indeed growing into all the allocated
space and beyond, I set a guard page in the end of the allocated stack
space (so that we hit a segfault before overusing the space).
Below are the code changes.
@@ -443,6 +443,8 @@ synctask_create (struct syncenv *env, size_t stacksize,
synctask_fn_t fn,
struct synctask *newtask = NULL;
xlator_t*this= THIS;
int destroymode = 0;
+int r=0;
+char*v;
VALIDATE_OR_GOTO (env, err);
VALIDATE_OR_GOTO (fn, err);
@@ -498,9 +500,15 @@ synctask_create (struct syncenv *env, size_t
stacksize, synctask_fn_t fn,
gf_common_mt_syncstack);
newtask->ctx.uc_stack.ss_size = env->stacksize;
} else {
-newtask->stack = GF_CALLOC (1, stacksize,
+ newtask->stack = GF_CALLOC (1, stacksize,
gf_common_mt_syncstack);
newtask->ctx.uc_stack.ss_size = stacksize;
+if (stacksize == 16*1024) {
+v = (unsigned long)((char *)(newtask->stack) +
4095) & (~4095);
+r = mprotect(v, 4096, PROT_NONE);
+ gf_msg ("syncop", GF_LOG_ERROR, errno,
+LG_MSG_GETCONTEXT_FAILED, "SKU: using 16k
stack starting at %p, mprotect returned %d, guard page: %p",
newtask->stack, r, v);
+ }
}
(gdb) where
#0 0x7f8a92c51204 in _dl_lookup_symbol_x () from
/lib64/ld-linux-x86-64.so.2
#1 0x7f8a92c561e3 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#2 0x7f8a92c5dd33 in _dl_runtime_resolve_avx () from
/lib64/ld-linux-x86-64.so.2
#3 0x in ?? ()
(gdb) info reg
rdi0x7f8a92946188140233141412232
rbp0x7f8a800b40000x7f8a800b4000
rsp0x7f8a800b40000x7f8a800b4000
r8 0x7f8a92e4ba60140233146677856
(gdb) layout asm
>│0x7f8a92c51204 <_dl_lookup_symbol_x+4> push
%r15 <== push on stack at the guarded page caused segfault
>From the brick log we have,
[syncop.c:515:synctask_create] 0-syncop: SKU: using 16k stack starting at
0x7f8a800b28f0, mprotect returned 0, guard page: 0x7f8a800b3000 [No data
available]
Stack grows downward from 0x7f8a800b68f0 to 0x7f8a800b28f0 and the page
0x7f8a800b3000 - 0x7f8a800b4000 is guarded , which is where the segfault
hit as seen in gdb.
This confirms that the stack space is not sufficient and overflowing,
I am not sure why we don't hit this in the presence of IO threads though,
It may just be that with io threads in graph, we may have some allocated
and unused memory which our stack freely grows into. It may just be a
silent undetected reuse of some memory.
Regards,
Sanoj
On Tue, Jun 13, 2017 at 10:49 AM, Sanoj Unnikrishnan
wrote:
> Hi Kinglong
> I was reading about makecontext/swapcontext as well,
> I did find an article that suggested to use mprotect and force a segfault
> to check if we have a stack space issue here.
> here is the link. http://www.evanjones.ca/software/threading.html.
>
> I don't think i can try this until tomorrow.
> thanks and regards,
> Sanoj
>
> On Tue, Jun 13, 2017 at 5:58 AM, Kinglong Mee
> wrote:
>
>> Hi Sanoj,
>>
>> What's your opinion about this problem?
>>
>> thanks,
>> Kinglong Mee
>>
>> On 6/9/2017 17:20, Kinglong Mee wrote:
>> > Hi Sanoj,
>> >
>> > On 6/9/2017 15:48, Sanoj Unnikrishnan wrote:
>> >> I have not used valgrind before, so I may be wrong here.
>> >>
>> >> I think the valgrind_stack_deregister should have been after
>> GF_FREE_STACK.
>> >> That may explain the instance of invalid_write during stack_destroy
>> calls in after.log.
>> >
>> > No. I move it, but the instance of invalid_write also exist.
>> >
>> >>
>> >> There seems to be numerous issues reported in before.log (I am
>> assuming, you did not have the valgrind_stack_register call in it),
>> >
>> > Yes, the before.log is test without any code change(but without
>> io-threads).
>> >
>> >> From http://valgrind.org/docs/manual/manual-core.html <
>> http://valgrind.org/docs/manual/manual-core.html>, looks like valgrind
>> detects client switching stack only If a memory of > 2MB change in Stack
>> pointer register.
>> >
>> > I test with a larger max-stackframe as,
>> > valgrind --leak-check=full --max-stackframe=242293216
>> >
>> >> Is it possible that since marker is only using 16k, the stack pointer
>> could have been in less than 2MB offset from current Stack Pointer?
>> >
>> > Maybe.
>> > But with io-threads (with adding valgrin
Re: [Gluster-devel] glusterfsd crash when using quota without io-threads
- Original Message -
> From: "Sanoj Unnikrishnan"
> To: "Kinglong Mee"
> Cc: "Gluster Devel"
> Sent: Wednesday, June 28, 2017 5:18:56 PM
> Subject: Re: [Gluster-devel] glusterfsd crash when using quota without
> io-threads
>
> Hi Kinglong,
>
> I opened https://bugzilla.redhat.com/show_bug.cgi?id=1465861 to track this.
> It seems to be an important issue that went undetected.
> Thanks for catching this!!
>
> In my attempts to repro this , I found that on each run some random
> structures where getting corrupted and running into segfault.
> In order to assert that the stack was indeed growing into all the allocated
> space and beyond, I set a guard page in the end of the allocated stack space
> (so that we hit a segfault before overusing the space).
> Below are the code changes.
>
> @@ -443,6 +443,8 @@ synctask_create (struct syncenv *env, size_t stacksize,
> synctask_fn_t fn,
> struct synctask *newtask = NULL;
> xlator_t *this = THIS;
> int destroymode = 0;
> + int r=0;
> + char *v;
>
> VALIDATE_OR_GOTO (env, err);
> VALIDATE_OR_GOTO (fn, err);
> @@ -498,9 +500,15 @@ synctask_create (struct syncenv *env, size_t stacksize,
> synctask_fn_t fn,
> gf_common_mt_syncstack);
> newtask->ctx.uc_stack.ss_size = env->stacksize;
> } else {
> - newtask->stack = GF_CALLOC (1, stacksize,
> + newtask->stack = GF_CALLOC (1, stacksize,
> gf_common_mt_syncstack);
> newtask->ctx.uc_stack.ss_size = stacksize;
> + if (stacksize == 16*1024) {
> + v = (unsigned long)((char *)(newtask->stack) + 4095) & (~4095);
> + r = mprotect(v, 4096, PROT_NONE);
> + gf_msg ("syncop", GF_LOG_ERROR, errno,
> + LG_MSG_GETCONTEXT_FAILED, "SKU: using 16k stack starting at %p, mprotect
> returned %d, guard page: %p", newtask->stack, r, v);
> + }
> }
>
> (gdb) where
> #0 0x7f8a92c51204 in _dl_lookup_symbol_x () from
> /lib64/ld-linux-x86-64.so.2
> #1 0x7f8a92c561e3 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
> #2 0x7f8a92c5dd33 in _dl_runtime_resolve_avx () from
> /lib64/ld-linux-x86-64.so.2
> #3 0x in ?? ()
>
>
> (gdb) info reg
>
> rdi 0x7f8a92946188 140233141412232
> rbp 0x7f8a800b4000 0x7f8a800b4000
> rsp 0x7f8a800b4000 0x7f8a800b4000
> r8 0x7f8a92e4ba60 140233146677856
>
> (gdb) layout asm
>
> >│0x7f8a92c51204 <_dl_lookup_symbol_x+4> push %r15 <== push on stack at the
> >guarded page caused segfault
>
> From the brick log we have,
>
> [syncop.c:515:synctask_create] 0-syncop: SKU: using 16k stack starting at
> 0x7f8a800b28f0, mprotect returned 0, guard page: 0x7f8a800b3000 [No data
> available]
>
> Stack grows downward from 0x7f8a800b68f0 to 0x7f8a800b28f0 and the page
> 0x7f8a800b3000 - 0x7f8a800b4000 is guarded , which is where the segfault hit
> as seen in gdb.
>
> This confirms that the stack space is not sufficient and overflowing,
> I am not sure why we don't hit this in the presence of IO threads though,
Is it a case of too many frames in a single C stack? If that is the case, with
io-threads in the graph STACK_WIND/UNWINDS won't result in the growth of a
single C stack. In the past we've seen these kind of crashes (for eg., while
updating sizes of all ancestors after a write by marker etc).
> It
> may just be that with io threads in graph, we may have some allocated and
> unused memory which our stack freely grows into. It may just be a silent
> undetected reuse of some memory.
>
> Regards,
> Sanoj
>
>
>
> On Tue, Jun 13, 2017 at 10:49 AM, Sanoj Unnikrishnan < sunni...@redhat.com >
> wrote:
>
>
>
> Hi Kinglong
> I was reading about makecontext/swapcontext as well,
> I did find an article that suggested to use mprotect and force a segfault to
> check if we have a stack space issue here.
> here is the link. http://www.evanjones.ca/software/threading.html .
>
> I don't think i can try this until tomorrow.
> thanks and regards,
> Sanoj
>
> On Tue, Jun 13, 2017 at 5:58 AM, Kinglong Mee < kinglong...@gmail.com >
> wrote:
>
>
> Hi Sanoj,
>
> What's your opinion about this problem?
>
> thanks,
> Kinglong Mee
>
> On 6/9/2017 17:20, Kinglong Mee wrote:
> > Hi Sanoj,
> >
> > On 6/9/2017 15:48, Sanoj Unnikrishnan wrote:
> >> I have not used valgrind before, so I may be wrong here.
> >>
> >> I think the valgrind_stack_deregister should have been after
> >> GF_FREE_STACK.
> >> That may explain the instance of invalid_write during stack_destroy calls
> >> in after.log.
> >
> > No. I move it, but the instance of invalid_write also exist.
> >
> >>
> >> There seems to be numerous issues reported in before.log (I am assuming,
> >> you did not have the valgrind_stack_register call in it),
> >
> > Yes, the before.log is test without any code change(but without
> > io-threads).
> >
> >> From http://valgrind.org/docs/manual/manual-core.html <
> >> http://valgrind.org/docs/manual/manual-core.html >, looks like valgrind
> >> detects client switching stack only If a memory of > 2MB change in Stack
> >> pointer register.
> >
> > I test with a larger max-stackframe as,
> >
Re: [Gluster-devel] glusterfsd crash when using quota without io-threads
Hi Sanoj,
Do you testing with changing the 16K to 2M (default stack size)?
If also overflowing, there must be an important issue exist;
not overflowing, only the issue of 16K is small for marker without io-threads?
thanks,
Kinglong Mee
On 6/28/2017 19:48, Sanoj Unnikrishnan wrote:
> Hi Kinglong,
>
> I opened https://bugzilla.redhat.com/show_bug.cgi?id=1465861 to track this.
> It seems to be an important issue that went undetected.
> Thanks for catching this!!
>
> In my attempts to repro this , I found that on each run some random
> structures where getting corrupted and running into segfault.
> In order to assert that the stack was indeed growing into all the allocated
> space and beyond, I set a guard page in the end of the allocated stack space
> (so that we hit a segfault before overusing the space).
> Below are the code changes.
>
> @@ -443,6 +443,8 @@ synctask_create (struct syncenv *env, size_t stacksize,
> synctask_fn_t fn,
> struct synctask *newtask = NULL;
> xlator_t*this= THIS;
> int destroymode = 0;
> +int r=0;
> +char*v;
>
> VALIDATE_OR_GOTO (env, err);
> VALIDATE_OR_GOTO (fn, err);
> @@ -498,9 +500,15 @@ synctask_create (struct syncenv *env, size_t stacksize,
> synctask_fn_t fn,
> gf_common_mt_syncstack);
> newtask->ctx.uc_stack.ss_size = env->stacksize;
> } else {
> -newtask->stack = GF_CALLOC (1, stacksize,
> + newtask->stack = GF_CALLOC (1, stacksize,
> gf_common_mt_syncstack);
> newtask->ctx.uc_stack.ss_size = stacksize;
> +if (stacksize == 16*1024) {
> +v = (unsigned long)((char *)(newtask->stack) + 4095)
> & (~4095);
> +r = mprotect(v, 4096, PROT_NONE);
> + gf_msg ("syncop", GF_LOG_ERROR, errno,
> +LG_MSG_GETCONTEXT_FAILED, "SKU: using 16k
> stack starting at %p, mprotect returned %d, guard page: %p", newtask->stack,
> r, v);
> + }
> }
>
> (gdb) where
> #0 0x7f8a92c51204 in _dl_lookup_symbol_x () from
> /lib64/ld-linux-x86-64.so.2
> #1 0x7f8a92c561e3 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
> #2 0x7f8a92c5dd33 in _dl_runtime_resolve_avx () from
> /lib64/ld-linux-x86-64.so.2
> #3 0x in ?? ()
>
>
> (gdb) info reg
>
> rdi0x7f8a92946188140233141412232
> rbp0x7f8a800b40000x7f8a800b4000
> rsp0x7f8a800b40000x7f8a800b4000
> r8 0x7f8a92e4ba60140233146677856
>
> (gdb) layout asm
>
> >│0x7f8a92c51204 <_dl_lookup_symbol_x+4> push %r15
> <== push on stack at the guarded page caused segfault
>
> From the brick log we have,
>
> [syncop.c:515:synctask_create] 0-syncop: SKU: using 16k stack starting at
> 0x7f8a800b28f0, mprotect returned 0, guard page: 0x7f8a800b3000 [No data
> available]
>
> Stack grows downward from 0x7f8a800b68f0 to 0x7f8a800b28f0 and the page
> 0x7f8a800b3000 - 0x7f8a800b4000 is guarded , which is where the segfault hit
> as seen in gdb.
>
> This confirms that the stack space is not sufficient and overflowing,
> I am not sure why we don't hit this in the presence of IO threads though, It
> may just be that with io threads in graph, we may have some allocated and
> unused memory which our stack freely grows into. It may just be a silent
> undetected reuse of some memory.
>
> Regards,
> Sanoj
>
>
>
> On Tue, Jun 13, 2017 at 10:49 AM, Sanoj Unnikrishnan > wrote:
>
> Hi Kinglong
> I was reading about makecontext/swapcontext as well,
> I did find an article that suggested to use mprotect and force a segfault
> to check if we have a stack space issue here.
> here is the link. http://www.evanjones.ca/software/threading.html
> .
>
> I don't think i can try this until tomorrow.
> thanks and regards,
> Sanoj
>
> On Tue, Jun 13, 2017 at 5:58 AM, Kinglong Mee > wrote:
>
> Hi Sanoj,
>
> What's your opinion about this problem?
>
> thanks,
> Kinglong Mee
>
> On 6/9/2017 17:20, Kinglong Mee wrote:
> > Hi Sanoj,
> >
> > On 6/9/2017 15:48, Sanoj Unnikrishnan wrote:
> >> I have not used valgrind before, so I may be wrong here.
> >>
> >> I think the valgrind_stack_deregister should have been after
> GF_FREE_STACK.
> >> That may explain the instance of invalid_write during
> stack_destroy calls in after.log.
> >
> > No. I move it, but the instance of invalid_write also exist.
> >
> >>
> >> There seems to be numerous issues re
[Gluster-devel] Coverity covscan for 2017-06-28-91db0d47 (master branch)
GlusterFS Coverity covscan results are available from http://download.gluster.org/pub/gluster/glusterfs/static-analysis/master/glusterfs-coverity/2017-06-28-91db0d47 ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] High memory cost of glusterfsd when enable quota
I use glusterfs quota on Centos7 with glusterfs-3.10.2-1.el7.x86_64, system memory is 2GB. My test environment is two glustefs nodes, with one 10GB disk (ext4) each. When enable quota with many files in the glusterfs, the glusterfsd in one node costs high memory towards 40%+ of 2GB, or both nodes costs high memory sometimes. Node one, PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 30017 root 20 0 1487412 115528 4648 S 0.0 6.2 3:11.81 glusterfsd The other node, PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 102510 root 20 0 1439044 700264 4668 S 0.0 37.5 2:31.37 glusterfsd The reproduce is, # gluster volume start fsname # mount -t glusterfs nodename:fsname /mnt/test/ # cp -rf linux-source /mnt/test/ < it's important adds many files before enable quota. # gluster volume quota fsname enable Sometime, needs disable quota and re-enable again, the high memory cost will appear. But, I can't find the high memory cost when debugging by valgrind, the memory cost will increase slowly to 10%. Any comments are welcome. thanks, Kinglong Mee ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Announcing Gluster release 3.11.1 (Short Term Maintenance)
The Gluster community is pleased to announce the release of Gluster 3.11.1 (packages available at [1]). Major changes and features (complete release notes can be found @ [2]) - Improved disperse (EC) volume performance - Group settings for enabling negative lookup caching are provided - Gluster fuse now implements "-oauto_unmount" feature We still carry a major issue that is reported in the release-notes as follows, - Expanding a gluster volume that is sharded may cause file corruption Sharded volumes are typically used for VM images, if such volumes are expanded or possibly contracted (i.e add/remove bricks and rebalance) there are reports of VM images getting corrupted. Status of this bug can be tracked here, #1465123 Thanks, Gluster community [1] Packages: https://download.gluster.org/pub/gluster/glusterfs/3.11/3.11.1/ [2] Complete release notes: https://github.com/gluster/glusterfs/blob/release-3.11/doc/release-notes/3.11.1.md ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Release 3.11.1: Tagged, packaged, released!
Hi, 3.11.1 has been tagged and packages are built, thanks to all for your contributions and help. With this the release tracker bug for 3.11.1 is closed and a 3.11.2 tracker is open [1]. Future blocker bugs for 3.11.2 need to be marked as a blocker against this. Blockers can be broadly classified as, - Regression due to code introduced in 3.11.2 - Corruptions - Crashes If you believe a bug is a blocker, but still does not meet the criteria above, err on the safe option and call it a blocker, whose merit can then be discussed on the lists and appropriate action for the release taken. Thanks, Kaushal & Shyam [1] 3.11.2 tracker: https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-3.11.2 "Releases are made better together" ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-Maintainers] Maintainers 2.0 Proposal
All, Thanks for participating actively in the discussions. With all your co-operation we now have a update on maintainers 2.0 proposal. Vijay Bellur sent a patch last week [1] capturing all the discussions. Please go through the patch and see if you have any more concerns. There are many new names in there, so just review it so you can Ack it. Niels (ndevos) added all the people with their name on maintainers file as reviewers for the patch. Please take some time today and give +1 to it to acknowledge you are aware of the responsibilities. After 20 or more +1 on the patch, we will merge the patch, and accordingly raise a ticket to update the access to merge rights etc. Also, if your name is added in maintainers list (even as peer for component), please become member of Maintainers mailing list [2] This list is an open list (all archives available for anyone to read) so make sure you subscribe and become members. Make sure you update your calendars with maintainer meeting timings, so you can attend it. [1] - https://review.gluster.org/17583 [2] - http://lists.gluster.org/mailman/listinfo/maintainers Main maintainers 2.0 proposal link: https://hackmd.io/s/SkwiZd4qe Write back if you have any more concerns. Regards, Amar On Tue, Apr 18, 2017 at 2:27 PM, Michael Scherer wrote: > Le mardi 18 avril 2017 à 10:25 +0200, Niels de Vos a écrit : > > On Mon, Apr 17, 2017 at 04:53:55PM -0700, Amye Scavarda wrote: > > > On Fri, Apr 14, 2017 at 2:40 AM, Michael Scherer > > > wrote: > > > > > > > Le jeudi 13 avril 2017 à 18:01 -0700, Amye Scavarda a écrit : > > > > > In light of community conversations, I've put some revisions on the > > > > > Maintainers changes, outlined in the hackmd pad: > > > > > https://hackmd.io/s/SkwiZd4qe > > > > > > > > > > Feedback welcomed! > > > > > > > > > > Note that the goals of this are to expand out our reach as a > project > > > > > (Gluster.org) and make it easy to define who's a maintainer for > what > > > > > feature. > > > > > I'll highlight the goals in the document here: > > > > > > > > > > * Refine how we declare component owners in Gluster > > > > > * Create a deeper sense of ownership throughout the open source > project > > > > > * Welcome more contibutors at a project impacting level > > > > > > > > > > We've clarified what the levels of 'owners' and 'peers' are in > terms of > > > > > responsibility, and we'll look to implement this in the 3.12 cycle. > > > > > Thanks! > > > > > > > > So, I realize that the concept of component is not defined in the > > > > document. I assume everybody have a shared understanding about what > it > > > > is, but maybe not, so wouldn't it make sense to define it more > clearly ? > > > > > > > > Is this planned to be done later as part of "We will be working on > > > > carving out new components for things that make logical sense." ? > > > > > > > > As for example, with regard to my previous comment, would > > > > "infrastructure" be a component, would "documentation" be a > component ? > > > > > > > > My understanding is that there's a working spreadsheet being refined > to > > > sort out what's an area that needs a maintainer defined, and what's > > > something that maybe doesn't need a named maintainer. Documentation is > a > > > tricky place to get to, because that's something that you do just > naturally > > > so that future-you doesn't hate current-you. > > > > I agree that documentation should be part of the standard development > > workflow. Unfortunately, this is not something that gets done without > > reminding everyone about it. We still need maintainers/owners to bug > > developers for documentation of new features, monitor the pull-request > > queue and decide if the documentation is written in an acceptible way. > > There is also the overall issue iof documentation consistency. For > example, style, glossary, etc, all kind of stuff that shouldn't be per > component but aligned overall. > > > The maintenance of the gluster.readthedocs.io site might be a > > infrastructure task? > > Wouldn't it be more logical to have it managed by the people that did > champion RTD ? I am unable to find the discussions about it, but I am > quite sure I had some concerns regarding RTD and wouldn't volunteer to > maintain something where I had objections (such as "being unable to fix > anything" is quite high on my usual objection list for taking > responsibility of a piece of infra) > -- > Michael Scherer > Sysadmin, Community Infrastructure and Platform, OSAS > > > > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-devel > -- Amar Tumballi (amarts) ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] [Questions] why shard feature doesn't support quota
Hi all I did a small test on gluster 3.8.4, and found shard feature does not support quota. Here is the step: $ gluster volume set s1 features.shard on $ gluster volume quota s1 enable $ gluster volume quota s1 limit-usage /ttt 1GB $ gluster volume quota s1 list $ cd /mnt/s1/ttt; fallocate -l 2GB testfile Then, i found testfile created sucessully. So, It seems shard doesn't support quota. It's a technical issue or there is a ongoing plan? -- Thanks -Xie ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Updates in Fstat
Hi Gluster Devs, In the past few days, I have added some new features to fstat: Fstat now allows the ability to associate bugs with failures Fstat now fetches aborted runs. Feel free to open a feature request or file a bug on GitHub if you face any issues. Thanks Smit Thakkar ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
