Re: [Gluster-devel] [Gluster-Maintainers] Proposal: move glusterfs development to github workflow, completely

[FNU Raghavendra Manjunath]

+1 to the idea.

On Mon, Aug 26, 2019 at 2:41 PM Niels de Vos  wrote:

> On Mon, Aug 26, 2019 at 08:08:36AM -0700, Joe Julian wrote:
> > You can also see diffs between force pushes now.
>
> That is great! It is the feature that I was looking for. I have not
> noticed it yet, will pay attention to it while working on other
> projects.
>
> Thanks,
> Niels
> ___
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/836554017
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/486278655
>
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
___

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/836554017

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/486278655

Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] fallocate behavior in glusterfs

[FNU Raghavendra Manjunath]

I have sent a rfc patch [1] for review.

https://review.gluster.org/#/c/glusterfs/+/23011/

On Thu, Jul 4, 2019 at 1:13 AM Pranith Kumar Karampuri 
wrote:

>
>
> On Wed, Jul 3, 2019 at 10:59 PM FNU Raghavendra Manjunath <
> rab...@redhat.com> wrote:
>
>>
>>
>> On Wed, Jul 3, 2019 at 3:28 AM Pranith Kumar Karampuri <
>> pkara...@redhat.com> wrote:
>>
>>>
>>>
>>> On Wed, Jul 3, 2019 at 10:14 AM Ravishankar N 
>>> wrote:
>>>
>>>>
>>>> On 02/07/19 8:52 PM, FNU Raghavendra Manjunath wrote:
>>>>
>>>>
>>>> Hi All,
>>>>
>>>> In glusterfs, there is an issue regarding the fallocate behavior. In
>>>> short, if someone does fallocate from the mount point with some size that
>>>> is greater than the available size in the backend filesystem where the file
>>>> is present, then fallocate can fail with a subset of the required number of
>>>> blocks allocated and then failing in the backend filesystem with ENOSPC
>>>> error.
>>>>
>>>> The behavior of fallocate in itself is simlar to how it would have been
>>>> on a disk filesystem (atleast xfs where it was checked). i.e. allocates
>>>> subset of the required number of blocks and then fail with ENOSPC. And the
>>>> file in itself would show the number of blocks in stat to be whatever was
>>>> allocated as part of fallocate. Please refer [1] where the issue is
>>>> explained.
>>>>
>>>> Now, there is one small difference between how the behavior is between
>>>> glusterfs and xfs.
>>>> In xfs after fallocate fails, doing 'stat' on the file shows the number
>>>> of blocks that have been allocated. Whereas in glusterfs, the number of
>>>> blocks is shown as zero which makes tools like "du" show zero consumption.
>>>> This difference in behavior in glusterfs is because of libglusterfs on how
>>>> it handles sparse files etc for calculating number of blocks (mentioned in
>>>> [1])
>>>>
>>>> At this point I can think of 3 things on how to handle this.
>>>>
>>>> 1) Except for how many blocks are shown in the stat output for the file
>>>> from the mount point (on which fallocate was done), the remaining behavior
>>>> of attempting to allocate the requested size and failing when the
>>>> filesystem becomes full is similar to that of XFS.
>>>>
>>>> Hence, what is required is to come up with a solution on how
>>>> libglusterfs calculate blocks for sparse files etc (without breaking any of
>>>> the existing components and features). This makes the behavior similar to
>>>> that of backend filesystem. This might require its own time to fix
>>>> libglusterfs logic without impacting anything else.
>>>>
>>>> I think we should just revert the commit
>>>> b1a5fa55695f497952264e35a9c8eb2bbf1ec4c3 (BZ 817343) and see if it really
>>>> breaks anything (or check whatever it breaks is something that we can live
>>>> with). XFS speculative preallocation is not permanent and the extra space
>>>> is freed up eventually. It can be sped up via procfs tunable:
>>>> http://xfs.org/index.php/XFS_FAQ#Q:_How_can_I_speed_up_or_avoid_delayed_removal_of_speculative_preallocation.3F.
>>>> We could also tune the allocsize option to a low value like 4k so that
>>>> glusterfs quota is not affected.
>>>>
>>>> FWIW, ENOSPC is not the only fallocate problem in gluster because of
>>>> 'iatt->ia_block' tweaking. It also breaks the --keep-size option (i.e. the
>>>> FALLOC_FL_KEEP_SIZE flag in fallocate(2)) and reports incorrect du size.
>>>>
>>> Regards,
>>>> Ravi
>>>>
>>>>
>>>> OR
>>>>
>>>> 2) Once the fallocate fails in the backend filesystem, make posix
>>>> xlator in the brick truncate the file to the previous size of the file
>>>> before attempting fallocate. A patch [2] has been sent for this. But there
>>>> is an issue with this when there are parallel writes and fallocate
>>>> operations happening on the same file. It can lead to a data loss.
>>>>
>>>> a) statpre is obtained ===> before fallocate is attempted, get the stat
>>>> hence the size of the file b) A parrallel Write fop on the same file that
>>>> extends the file is successful c) Fallocate fails d) ftruncate truncat

Re: [Gluster-devel] fallocate behavior in glusterfs

[FNU Raghavendra Manjunath]

On Wed, Jul 3, 2019 at 3:28 AM Pranith Kumar Karampuri 
wrote:

>
>
> On Wed, Jul 3, 2019 at 10:14 AM Ravishankar N 
> wrote:
>
>>
>> On 02/07/19 8:52 PM, FNU Raghavendra Manjunath wrote:
>>
>>
>> Hi All,
>>
>> In glusterfs, there is an issue regarding the fallocate behavior. In
>> short, if someone does fallocate from the mount point with some size that
>> is greater than the available size in the backend filesystem where the file
>> is present, then fallocate can fail with a subset of the required number of
>> blocks allocated and then failing in the backend filesystem with ENOSPC
>> error.
>>
>> The behavior of fallocate in itself is simlar to how it would have been
>> on a disk filesystem (atleast xfs where it was checked). i.e. allocates
>> subset of the required number of blocks and then fail with ENOSPC. And the
>> file in itself would show the number of blocks in stat to be whatever was
>> allocated as part of fallocate. Please refer [1] where the issue is
>> explained.
>>
>> Now, there is one small difference between how the behavior is between
>> glusterfs and xfs.
>> In xfs after fallocate fails, doing 'stat' on the file shows the number
>> of blocks that have been allocated. Whereas in glusterfs, the number of
>> blocks is shown as zero which makes tools like "du" show zero consumption.
>> This difference in behavior in glusterfs is because of libglusterfs on how
>> it handles sparse files etc for calculating number of blocks (mentioned in
>> [1])
>>
>> At this point I can think of 3 things on how to handle this.
>>
>> 1) Except for how many blocks are shown in the stat output for the file
>> from the mount point (on which fallocate was done), the remaining behavior
>> of attempting to allocate the requested size and failing when the
>> filesystem becomes full is similar to that of XFS.
>>
>> Hence, what is required is to come up with a solution on how libglusterfs
>> calculate blocks for sparse files etc (without breaking any of the existing
>> components and features). This makes the behavior similar to that of
>> backend filesystem. This might require its own time to fix libglusterfs
>> logic without impacting anything else.
>>
>> I think we should just revert the commit
>> b1a5fa55695f497952264e35a9c8eb2bbf1ec4c3 (BZ 817343) and see if it really
>> breaks anything (or check whatever it breaks is something that we can live
>> with). XFS speculative preallocation is not permanent and the extra space
>> is freed up eventually. It can be sped up via procfs tunable:
>> http://xfs.org/index.php/XFS_FAQ#Q:_How_can_I_speed_up_or_avoid_delayed_removal_of_speculative_preallocation.3F.
>> We could also tune the allocsize option to a low value like 4k so that
>> glusterfs quota is not affected.
>>
>> FWIW, ENOSPC is not the only fallocate problem in gluster because of
>> 'iatt->ia_block' tweaking. It also breaks the --keep-size option (i.e. the
>> FALLOC_FL_KEEP_SIZE flag in fallocate(2)) and reports incorrect du size.
>>
> Regards,
>> Ravi
>>
>>
>> OR
>>
>> 2) Once the fallocate fails in the backend filesystem, make posix xlator
>> in the brick truncate the file to the previous size of the file before
>> attempting fallocate. A patch [2] has been sent for this. But there is an
>> issue with this when there are parallel writes and fallocate operations
>> happening on the same file. It can lead to a data loss.
>>
>> a) statpre is obtained ===> before fallocate is attempted, get the stat
>> hence the size of the file b) A parrallel Write fop on the same file that
>> extends the file is successful c) Fallocate fails d) ftruncate truncates it
>> to size given by statpre (i.e. the previous stat and the size obtained in
>> step a)
>>
>> OR
>>
>> 3) Make posix check for available disk size before doing fallocate. i.e.
>> in fallocate once posix gets the number of bytes to be allocated for the
>> file from a particular offset, it checks whether so many bytes are
>> available or not in the disk. If not, fail the fallocate fop with ENOSPC
>> (without attempting it on the backend filesystem).
>>
>> There still is a probability of a parallel write happening while this
>> fallocate is happening and by the time falllocate system call is attempted
>> on the disk, the available space might have been less than what was
>> calculated before fallocate.
>> i.e. following things can happen
>>
>>  a) statfs ===> get the available space of the backend filesystem

[Gluster-devel] fallocate behavior in glusterfs

[FNU Raghavendra Manjunath]

Hi All,

In glusterfs, there is an issue regarding the fallocate behavior. In short,
if someone does fallocate from the mount point with some size that is
greater than the available size in the backend filesystem where the file is
present, then fallocate can fail with a subset of the required number of
blocks allocated and then failing in the backend filesystem with ENOSPC
error.

The behavior of fallocate in itself is simlar to how it would have been on
a disk filesystem (atleast xfs where it was checked). i.e. allocates subset
of the required number of blocks and then fail with ENOSPC. And the file in
itself would show the number of blocks in stat to be whatever was allocated
as part of fallocate. Please refer [1] where the issue is explained.

Now, there is one small difference between how the behavior is between
glusterfs and xfs.
In xfs after fallocate fails, doing 'stat' on the file shows the number of
blocks that have been allocated. Whereas in glusterfs, the number of blocks
is shown as zero which makes tools like "du" show zero consumption. This
difference in behavior in glusterfs is because of libglusterfs on how it
handles sparse files etc for calculating number of blocks (mentioned in [1])

At this point I can think of 3 things on how to handle this.

1) Except for how many blocks are shown in the stat output for the file
from the mount point (on which fallocate was done), the remaining behavior
of attempting to allocate the requested size and failing when the
filesystem becomes full is similar to that of XFS.

Hence, what is required is to come up with a solution on how libglusterfs
calculate blocks for sparse files etc (without breaking any of the existing
components and features). This makes the behavior similar to that of
backend filesystem. This might require its own time to fix libglusterfs
logic without impacting anything else.

OR

2) Once the fallocate fails in the backend filesystem, make posix xlator in
the brick truncate the file to the previous size of the file before
attempting fallocate. A patch [2] has been sent for this. But there is an
issue with this when there are parallel writes and fallocate operations
happening on the same file. It can lead to a data loss.

a) statpre is obtained ===> before fallocate is attempted, get the stat
hence the size of the file b) A parrallel Write fop on the same file that
extends the file is successful c) Fallocate fails d) ftruncate truncates it
to size given by statpre (i.e. the previous stat and the size obtained in
step a)

OR

3) Make posix check for available disk size before doing fallocate. i.e. in
fallocate once posix gets the number of bytes to be allocated for the file
from a particular offset, it checks whether so many bytes are available or
not in the disk. If not, fail the fallocate fop with ENOSPC (without
attempting it on the backend filesystem).

There still is a probability of a parallel write happening while this
fallocate is happening and by the time falllocate system call is attempted
on the disk, the available space might have been less than what was
calculated before fallocate.
i.e. following things can happen

 a) statfs ===> get the available space of the backend filesystem
 b) a parallel write succeeds and extends the file
 c) fallocate is attempted assuming there is sufficient space in the backend

While the above situation can arise, I think we are still fine. Because
fallocate is attempted from the offset received in the fop. So,
irrespective of whether write extended the file or not, the fallocate
itself will be attempted for so many bytes from the offset which we found
to be available by getting statfs information.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1724754#c3
[2] https://review.gluster.org/#/c/glusterfs/+/22969/

Please provide feedback.

Regards,
Raghavendra
___

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/836554017

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/486278655

Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Build failed in Jenkins: regression-test-with-multiplex #1359

[FNU Raghavendra Manjunath]

Yes. I have sent this patch [1] for review. It is now not failing in
regression tests. (i.e. uss.t is not failing)

[1] https://review.gluster.org/#/c/glusterfs/+/22728/

Regards,
Raghavendra

On Sat, Jun 1, 2019 at 7:25 AM Atin Mukherjee  wrote:

> subdir-mount.t has started failing in brick mux regression nightly. This
> needs to be fixed.
>
> Raghavendra - did we manage to get any further clue on uss.t failure?
>
> -- Forwarded message -
> From: 
> Date: Fri, 31 May 2019 at 23:34
> Subject: [Gluster-Maintainers] Build failed in Jenkins:
> regression-test-with-multiplex #1359
> To: , , ,
> , 
>
>
> See <
> https://build.gluster.org/job/regression-test-with-multiplex/1359/display/redirect?page=changes
> >
>
> Changes:
>
> [atin] glusterd: add an op-version check
>
> [atin] glusterd/svc: glusterd_svcs_stop should call individual wrapper
> function
>
> [atin] glusterd/svc: Stop stale process using the glusterd_proc_stop
>
> [Amar Tumballi] lcov: more coverage to shard, old-protocol, sdfs
>
> [Kotresh H R] tests/geo-rep: Add EC volume test case
>
> [Amar Tumballi] glusterfsd/cleanup: Protect graph object under a lock
>
> [Mohammed Rafi KC] glusterd/shd: Optimize the glustershd manager to send
> reconfigure
>
> [Kotresh H R] tests/geo-rep: Add tests to cover glusterd geo-rep
>
> [atin] glusterd: Optimize code to copy dictionary in handshake code path
>
> --
> [...truncated 3.18 MB...]
> ./tests/basic/afr/stale-file-lookup.t  -  9 second
> ./tests/basic/afr/granular-esh/replace-brick.t  -  9 second
> ./tests/basic/afr/granular-esh/add-brick.t  -  9 second
> ./tests/basic/afr/gfid-mismatch.t  -  9 second
> ./tests/performance/open-behind.t  -  8 second
> ./tests/features/ssl-authz.t  -  8 second
> ./tests/features/readdir-ahead.t  -  8 second
> ./tests/bugs/upcall/bug-1458127.t  -  8 second
> ./tests/bugs/transport/bug-873367.t  -  8 second
> ./tests/bugs/replicate/bug-1498570-client-iot-graph-check.t  -  8 second
> ./tests/bugs/replicate/bug-1132102.t  -  8 second
> ./tests/bugs/quota/bug-1250582-volume-reset-should-not-remove-quota-quota-deem-statfs.t
> -  8 second
> ./tests/bugs/quota/bug-1104692.t  -  8 second
> ./tests/bugs/posix/bug-1360679.t  -  8 second
> ./tests/bugs/posix/bug-1122028.t  -  8 second
> ./tests/bugs/nfs/bug-1157223-symlink-mounting.t  -  8 second
> ./tests/bugs/glusterfs/bug-861015-log.t  -  8 second
> ./tests/bugs/glusterd/sync-post-glusterd-restart.t  -  8 second
> ./tests/bugs/glusterd/bug-1696046.t  -  8 second
> ./tests/bugs/fuse/bug-983477.t  -  8 second
> ./tests/bugs/ec/bug-1227869.t  -  8 second
> ./tests/bugs/distribute/bug-1088231.t  -  8 second
> ./tests/bugs/distribute/bug-1086228.t  -  8 second
> ./tests/bugs/cli/bug-1087487.t  -  8 second
> ./tests/bugs/cli/bug-1022905.t  -  8 second
> ./tests/bugs/bug-1258069.t  -  8 second
> ./tests/bugs/bitrot/1209752-volume-status-should-show-bitrot-scrub-info.t
> -  8 second
> ./tests/basic/xlator-pass-through-sanity.t  -  8 second
> ./tests/basic/quota-nfs.t  -  8 second
> ./tests/basic/glusterd/arbiter-volume.t  -  8 second
> ./tests/basic/ctime/ctime-noatime.t  -  8 second
> ./tests/line-coverage/cli-peer-and-volume-operations.t  -  7 second
> ./tests/gfid2path/get-gfid-to-path.t  -  7 second
> ./tests/bugs/upcall/bug-1369430.t  -  7 second
> ./tests/bugs/snapshot/bug-1260848.t  -  7 second
> ./tests/bugs/shard/shard-inode-refcount-test.t  -  7 second
> ./tests/bugs/shard/bug-1258334.t  -  7 second
> ./tests/bugs/replicate/bug-767585-gfid.t  -  7 second
> ./tests/bugs/replicate/bug-1448804-check-quorum-type-values.t  -  7 second
> ./tests/bugs/replicate/bug-1250170-fsync.t  -  7 second
> ./tests/bugs/posix/bug-1175711.t  -  7 second
> ./tests/bugs/nfs/bug-915280.t  -  7 second
> ./tests/bugs/md-cache/setxattr-prepoststat.t  -  7 second
> ./tests/bugs/md-cache/bug-1211863_unlink.t  -  7 second
> ./tests/bugs/glusterfs/bug-848251.t  -  7 second
> ./tests/bugs/distribute/bug-1122443.t  -  7 second
> ./tests/bugs/changelog/bug-1208470.t  -  7 second
> ./tests/bugs/bug-1702299.t  -  7 second
> ./tests/bugs/bug-1371806_2.t  -  7 second
> ./tests/bugs/bitrot/1209818-vol-info-show-scrub-process-properly.t  -  7
> second
> ./tests/bugs/bitrot/1209751-bitrot-scrub-tunable-reset.t  -  7 second
> ./tests/bugs/bitrot/1207029-bitrot-daemon-should-start-on-valid-node.t  -
> 7 second
> ./tests/bitrot/br-stub.t  -  7 second
> ./tests/basic/glusterd/arbiter-volume-probe.t  -  7 second
> ./tests/basic/gfapi/libgfapi-fini-hang.t  -  7 second
> ./tests/basic/fencing/fencing-crash-conistency.t  -  7 second
> ./tests/basic/distribute/file-create.t  -  7 second
> ./tests/basic/afr/tarissue.t  -  7 second
> ./tests/basic/afr/gfid-heal.t  -  7 second
> ./tests/bugs/snapshot/bug-1178079.t  -  6 second
> ./tests/bugs/snapshot/bug-1064768.t  -  6 second
> ./tests/bugs/shard/bug-1342298.t  -  6 second
> ./tests/bugs/shard/bug-1259651.t  -  6 second
> 

Re: [Gluster-devel] making frame->root->unique more effective in debugging hung frames

[FNU Raghavendra Manjunath]

The idea looks OK. One of the things that probably need to be considered
(more of an implementation detail though) is how to generate
frame->root->unique.

Because, for fuse, frame->root->unique is obtained by finh->unique which
IIUC is got from the incoming fop from kernel itself.
For protocol/server IIUC frame->root->unique is got from req->xit of the
rpc request, which itself is obtained from transport->xid of the
rpc_transport_t structure (and from my understanding, the transport->xid is
just incremented by everytime a
new rpc request is created).

Overall the suggestion looks fine though.

Regards,
Raghavendra


On Fri, May 24, 2019 at 2:27 AM Pranith Kumar Karampuri 
wrote:

> Hi,
> At the moment new stack doesn't populate frame->root->unique in
> all cases. This makes it difficult to debug hung frames by examining
> successive state dumps. Fuse and server xlator populate it whenever they
> can, but other xlators won't be able to assign one when they need to create
> a new frame/stack. Is it okay to change create_frame() code to always
> populate it with an increasing number for this purpose?
>  I checked both fuse and server xlator use it only in gf_log() so it
> doesn't seem like there is any other link between frame->root->unique and
> the functionality of fuse, server xlators.
>   Do let me know if I missed anything before sending this change.
>
> --
> Pranith
> ___
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/836554017
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/486278655
>
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
___

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/836554017

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/486278655

Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] ./tests/basic/uss.t is timing out in release-6 branch

[FNU Raghavendra Manjunath]

More analysis:

It looks like in the 1st iteration, the testcase is stuck  at the test
(TEST ! stat $M0/.history/snap6/aaa) from line 385 of uss.t

Before, it was the last test to be executed from uss.t. So the assumption
was that after the completion of that test (i.e. test from line 385),
cleanup function was
either getting blocked or taking more time to do cleanups.

Hence the patch [1] was submitted to reduce the amount of work done by
cleanup function. The patch ensured that, the snapshots, volume etc created
in the test
are deleted before cleanup function is executed. But even with that, we
observed uss.t to fail sometimes (mainly with brick-mux regressions). To
get more infomration
regarding the failure, another patch [2] was sent. From that patch some
more information is received.

1) Everytime uss.t times out, the script (uss.t) is stuck in executing the
particular test from line 385 (TEST ! stat $M0/.history/snap6/aaa)

- This test's purpose is to ensure that, looking for a file that does
not exist in a snapshot should fail.

2) Adding TRACE logs via [2] indicates that:
 -  the stat request sent by the test reaches snapshot daemon and later
the gfapi client instance that the snapshot daemon spawns
to communicate with the snapshot volume.
 - The stat request is served by the md-cache xlator in the gfapi
client instance (and hence successful).

  "[2019-05-16 18:31:18.607521]:++ G_LOG:./tests/basic/uss.t:
TEST: 392 ! stat /mnt/glusterfs/0/.history/snap6/aaa ++
   [2019-05-16 18:31:18.617104] T [MSGID: 0]
[syncop.c:2424:syncop_stat] 0-stack-trace: stack-address: 0x7fc63405dba8,
winding from gfapi to meta-autoload
   [2019-05-16 18:31:18.617119] T [MSGID: 0]
[defaults.c:2841:default_stat] 0-stack-trace: stack-address:
0x7fc63405dba8, winding from meta-autoload to
0e69605de2974f1b887deee5b3f63b52
   [2019-05-16 18:31:18.617130] T [MSGID: 0]
[io-stats.c:2709:io_stats_stat] 0-stack-trace: stack-address:
0x7fc63405dba8, winding from 0e69605de2974f1b887deee5b3f63b52 to
0e69605de2974f1b887deee5b3f63b52-io-threads
   [2019-05-16 18:31:18.617142] D [MSGID: 0]
[io-threads.c:376:iot_schedule]
0-0e69605de2974f1b887deee5b3f63b52-io-threads: STAT scheduled as fast
priority fop
   [2019-05-16 18:31:18.617162] T [MSGID: 0]
[defaults.c:2068:default_stat_resume] 0-stack-trace: stack-address:
0x7fc63405dba8, winding from 0e69605de2974f1b887deee5b3f63b52-io-threads to
0e69605de2974f1b887deee5b3f63b52-md-cache
   [2019-05-16 18:31:18.617176] T [MSGID: 0] [md-cache.c:1359:mdc_stat]
0-stack-trace: stack-address: 0x7fc63405dba8,
0e69605de2974f1b887deee5b3f63b52-md-cache returned 0  => SUCCESSFUL
HERE
   [2019-05-16 18:31:18.617186] T [MSGID: 0]
[defaults.c:1406:default_stat_cbk] 0-stack-trace: stack-address:
0x7fc63405dba8, 0e69605de2974f1b887deee5b3f63b52-io-threads returned 0
   [2019-05-16 18:31:18.617195] T [MSGID: 0]
[io-stats.c:2059:io_stats_stat_cbk] 0-stack-trace: stack-address:
0x7fc63405dba8, 0e69605de2974f1b887deee5b3f63b52 returned 0
  "
- The stat response does not reach snapshot daemon. So snapshot daemon
is not able to send any response back to the gluster client which initiated
this stat request. This leads to client waiting for a response resulting in
timeout as per
  the regression test infra (which sets 200 seconds timeout)

Suspects:
==

* First of all the the stat request from the line 385 (TEST ! stat
$M0/.history/snap6/aaa) should not be successful. Because, the test deletes
the snapshot "snap6", removes the file "aaa" from the mount point,
  again takes the snapshot "snap6" and performs the stat operatoin on the
deleted file "aaa". So the stat should fail.

* The patch [2] has been sent to collect more information about the failure
(with more logs added to snapview-server and also log level being changed
to TRACE in the .t file)

[1] https://review.gluster.org/#/c/glusterfs/+/22649/
[2] https://review.gluster.org/#/c/glusterfs/+/22728/

Regards,
Raghavendra

On Wed, May 1, 2019 at 11:11 AM Sanju Rakonde  wrote:

> Thank you Raghavendra.
>
> On Tue, Apr 30, 2019 at 11:46 PM FNU Raghavendra Manjunath <
> rab...@redhat.com> wrote:
>
>>
>> To make things relatively easy for the cleanup () function in the test
>> framework, I think it would be better to ensure that uss.t itself deletes
>> snapshots and the volume once the tests are done. Patch [1] has been
>> submitted for review.
>>
>> [1] https://review.gluster.org/#/c/glusterfs/+/22649/
>>
>> Regards,
>> Raghavendra
>>
>> On Tue, Apr 30, 2019 at 10:42 AM FNU Raghavendra Manjunath <
>> rab...@redhat.com> wrote:
>>
>>>
>>> The failure looks similar to the issue I had mentioned in [1]
>>>
>>> In short for some reason the cleanup (the cleanup function that

Re: [Gluster-devel] ./tests/bugs/snapshot/bug-1399598-uss-with-ssl.t generating core very often

[FNU Raghavendra Manjunath]

I am working on other uss issue. i.e. the occasional failure of uss.t due
to delays in the brick-mux regression. Rafi? Can you please look into this?

Regards,
Raghavendra

On Thu, May 16, 2019 at 9:48 AM Sanju Rakonde  wrote:

> In most of the regression jobs 
> ./tests/bugs/snapshot/bug-1399598-uss-with-ssl.t
> is dumping core, hence the regression is failing for many patches.
>
> Rafi/Raghavendra, can you please look into this issue?
>
> --
> Thanks,
> Sanju
>
___

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/836554017

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/486278655

Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Meeting Details on footer of the gluster-devel and gluster-user mailing list

[FNU Raghavendra Manjunath]

+ 1 to this.

There is also one more thing. For some reason, the community meeting is not
visible in my calendar (especially NA region). I am not sure if anyone else
also facing this issue.

Regards,
Raghavendra

On Tue, May 7, 2019 at 5:19 AM Ashish Pandey  wrote:

> Hi,
>
> While we send a mail on gluster-devel or gluster-user mailing list,
> following content gets auto generated and placed at the end of mail.
>
> Gluster-users mailing list
> gluster-us...@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
> In the similar way, is it possible to attach meeting schedule and link at the 
> end of every such mails?
> Like this -
>
> Meeting schedule -
>
>
>- APAC friendly hours
>   - Tuesday 14th May 2019, 11:30AM IST
>   - Bridge: https://bluejeans.com/836554017
>   - NA/EMEA
>   - Tuesday 7th May 2019, 01:00 PM EDT
>   - Bridge: https://bluejeans.com/486278655
>
> Or just a link to meeting minutes details??
>  
> https://github.com/gluster/community/tree/master/meetings
>
> This will help developers and users of the community to know when and where 
> meeting happens and how to attend those meetings.
>
> ---
> Ashish
>
>
>
>
>
>
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] inode table destruction

[FNU Raghavendra Manjunath]

Hi All,

There is a good chance that, the inode on which unref came has already been
 zero refed and added to the purge list. This can happen when inode table is
 being destroyed (glfs_fini is something which destroys the inode table).

 Consider a directory 'a' which has a file 'b'. Now as part of inode table
destruction
 zero refing of inodes does not happen from leaf to the root. It happens in
the order
 inodes are present in the list. So, in this example, the dentry of 'b'
would have its
 parent set to the inode of 'a'. So if 'a' gets zero refed first (as part of
 inode table cleanup) and then 'b' has to zero refed, then dentry_unset is
called on
 the dentry of 'b' and it further goes on to call inode_unref on b's parent
which is
 'a'. In this situation, GF_ASSERT would be called as the refcount of 'a'
has been
 already set to zero.

Below is a snippet of the core file generated from such ASSERT call in one
of the regression test runs.

"

*14:39:49* No symbol table info available.*14:39:49* #1
0x7f2a539fc8f8 in abort () from /lib64/libc.so.6*14:39:49* No
symbol table info available.*14:39:49* #2  0x7f2a539f4026 in
__assert_fail_base () from /lib64/libc.so.6*14:39:49* No symbol table
info available.*14:39:49* #3  0x7f2a539f40d2 in __assert_fail ()
from /lib64/libc.so.6*14:39:49* No symbol table info
available.*14:39:49* #4  0x7f2a553e3208 in __inode_unref
(inode=0x7f2a3c05edf8, clear=false) at
/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:483*14:39:49*
index = 0*14:39:49* this = 0x7f2a3c03e840*14:39:49*
 nlookup = 0*14:39:49* __PRETTY_FUNCTION__ =
"__inode_unref"*14:39:49* #5  0x7f2a553e2745 in __dentry_unset
(dentry=0x7f2a3c064e48) at
/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:212*14:39:49*
No locals.*14:39:49* #6  0x7f2a553e308a in __inode_retire
(inode=0x7f2a3c05ebc8) at
/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:442*14:39:49*
dentry = 0x7f2a3c064e48*14:39:49* t =
0x7f2a3c064398*14:39:49* #7  0x7f2a553e392f in
__inode_ref_reduce_by_n (inode=0x7f2a3c05ebc8, nref=0) at
/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:708*14:39:49*
nlookup = 0*14:39:49* __PRETTY_FUNCTION__ =
"__inode_ref_reduce_by_n"*14:39:49* #8  0x7f2a553e61d5 in
inode_table_destroy (inode_table=0x7f2a28007f90) at
/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:1867*14:39:49*
trav = 0x7f2a3c05ebc8*14:39:49* __FUNCTION__ =
"inode_table_destroy"*14:39:49* #9  0x7f2a553e600c in
inode_table_destroy_all (ctx=0x7f2a3c001170) at
/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:1791*14:39:49*
trav_graph = 0x7f2a240041a0*14:39:49* tmp =
0x7f2a3c0013c8*14:39:49* tree = 0x7f2a24022af0*14:39:49*
  inode_table = 0x7f2a28007f90*14:39:49* #10 0x7f2a46a83390 in
pub_glfs_fini (fs=0x7f2a3c000ff0) at
/home/jenkins/root/workspace/centos7-regression/api/src/glfs.c:1346*14:39:49*
ret = 0*14:39:49* countdown = 98*14:39:49*
subvol = 0x7f2a24022af0*14:39:49* ctx =
0x7f2a3c001170*14:39:49* graph = 0x7f2a240041a0*14:39:49*
   call_pool = 0x7f2a3c000df0*14:39:49* fs_init = 1*14:39:49*
   err = 0*14:39:49* old_THIS = 0x7f2a380084c0*14:39:49*
  __FUNCTION__ = "pub_glfs_fini"

"


IIUC the solution for it would be to add a flag in the inode table which
tells whether the
cleanup of inode table has started or not. Do not call GF_ASSRT
(inode->ref) if the inode
table to which the inode being unrefed belongs to, is already getting
cleaned up.

A patch [1] has been submitted for review with the change mentioned above.

[1] https://review.gluster.org/#/c/glusterfs/+/22650/

Regards,
Raghavendra
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] ./tests/basic/uss.t is timing out in release-6 branch

[FNU Raghavendra Manjunath]

To make things relatively easy for the cleanup () function in the test
framework, I think it would be better to ensure that uss.t itself deletes
snapshots and the volume once the tests are done. Patch [1] has been
submitted for review.

[1] https://review.gluster.org/#/c/glusterfs/+/22649/

Regards,
Raghavendra

On Tue, Apr 30, 2019 at 10:42 AM FNU Raghavendra Manjunath <
rab...@redhat.com> wrote:

>
> The failure looks similar to the issue I had mentioned in [1]
>
> In short for some reason the cleanup (the cleanup function that we call in
> our .t files) seems to be taking more time and also not cleaning up
> properly. This leads to problems for the 2nd iteration (where basic things
> such as volume creation or volume start itself fails due to ENODATA or
> ENOENT errors).
>
> The 2nd iteration of the uss.t ran had the following errors.
>
> "[2019-04-29 09:08:15.275773]:++ G_LOG:./tests/basic/uss.t: TEST:
> 39 gluster --mode=script --wignore volume set patchy nfs.disable false
> ++
> [2019-04-29 09:08:15.390550]  : volume set patchy nfs.disable false :
> SUCCESS
> [2019-04-29 09:08:15.404624]:++ G_LOG:./tests/basic/uss.t: TEST:
> 42 gluster --mode=script --wignore volume start patchy ++
> [2019-04-29 09:08:15.468780]  : volume start patchy : FAILED : Failed to
> get extended attribute trusted.glusterfs.volume-id for brick dir
> /d/backends/3/patchy_snap_mnt. Reason : No data available
> "
>
> These are the initial steps to create and start volume. Why
> trusted.glusterfs.volume-id extended attribute is absent is not sure. The
> analysis in [1] had errors of ENOENT (i.e. export directory itself was
> absent).
> I suspect this to be because of some issue with the cleanup mechanism at
> the end of the tests.
>
> [1]
> https://lists.gluster.org/pipermail/gluster-devel/2019-April/056104.html
>
> On Tue, Apr 30, 2019 at 8:37 AM Sanju Rakonde  wrote:
>
>> Hi Raghavendra,
>>
>> ./tests/basic/uss.t is timing out in release-6 branch consistently. One
>> such instance is https://review.gluster.org/#/c/glusterfs/+/22641/. Can
>> you please look into this?
>>
>> --
>> Thanks,
>> Sanju
>>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] ./tests/basic/uss.t is timing out in release-6 branch

[FNU Raghavendra Manjunath]

The failure looks similar to the issue I had mentioned in [1]

In short for some reason the cleanup (the cleanup function that we call in
our .t files) seems to be taking more time and also not cleaning up
properly. This leads to problems for the 2nd iteration (where basic things
such as volume creation or volume start itself fails due to ENODATA or
ENOENT errors).

The 2nd iteration of the uss.t ran had the following errors.

"[2019-04-29 09:08:15.275773]:++ G_LOG:./tests/basic/uss.t: TEST:
39 gluster --mode=script --wignore volume set patchy nfs.disable false
++
[2019-04-29 09:08:15.390550]  : volume set patchy nfs.disable false :
SUCCESS
[2019-04-29 09:08:15.404624]:++ G_LOG:./tests/basic/uss.t: TEST: 42
gluster --mode=script --wignore volume start patchy ++
[2019-04-29 09:08:15.468780]  : volume start patchy : FAILED : Failed to
get extended attribute trusted.glusterfs.volume-id for brick dir
/d/backends/3/patchy_snap_mnt. Reason : No data available
"

These are the initial steps to create and start volume. Why
trusted.glusterfs.volume-id extended attribute is absent is not sure. The
analysis in [1] had errors of ENOENT (i.e. export directory itself was
absent).
I suspect this to be because of some issue with the cleanup mechanism at
the end of the tests.

[1] https://lists.gluster.org/pipermail/gluster-devel/2019-April/056104.html

On Tue, Apr 30, 2019 at 8:37 AM Sanju Rakonde  wrote:

> Hi Raghavendra,
>
> ./tests/basic/uss.t is timing out in release-6 branch consistently. One
> such instance is https://review.gluster.org/#/c/glusterfs/+/22641/. Can
> you please look into this?
>
> --
> Thanks,
> Sanju
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Proposal: Changes in Gluster Community meetings

[FNU Raghavendra Manjunath]

Hi,

This is the agenda for tomorrow's community meeting for NA/EMEA timezone.

https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both




On Thu, Apr 11, 2019 at 4:56 AM Amar Tumballi Suryanarayan <
atumb...@redhat.com> wrote:

> Hi All,
>
> Below is the final details of our community meeting, and I will be sending
> invites to mailing list following this email. You can add Gluster Community
> Calendar so you can get notifications on the meetings.
>
> We are starting the meetings from next week. For the first meeting, we
> need 1 volunteer from users to discuss the use case / what went well, and
> what went bad, etc. preferrably in APAC region.  NA/EMEA region, next week.
>
> Draft Content: https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g
> 
> Gluster Community Meeting
> Previous
> Meeting minutes:
>
>- http://github.com/gluster/community
>
>
> Date/Time:
> Check the community calendar
> 
> Bridge
>
>- APAC friendly hours
>   - Bridge: https://bluejeans.com/836554017
>- NA/EMEA
>   - Bridge: https://bluejeans.com/486278655
>
> --
> Attendance
>
>- Name, Company
>
> Host
>
>- Who will host next meeting?
>   - Host will need to send out the agenda 24hr - 12hrs in advance to
>   mailing list, and also make sure to send the meeting minutes.
>   - Host will need to reach out to one user at least who can talk
>   about their usecase, their experience, and their needs.
>   - Host needs to send meeting minutes as PR to
>   http://github.com/gluster/community
>
> User stories
>
>- Discuss 1 usecase from a user.
>   - How was the architecture derived, what volume type used, options,
>   etc?
>   - What were the major issues faced ? How to improve them?
>   - What worked good?
>   - How can we all collaborate well, so it is win-win for the
>   community and the user? How can we
>
> Community
>
>-
>
>Any release updates?
>-
>
>Blocker issues across the project?
>-
>
>Metrics
>- Number of new bugs since previous meeting. How many are not triaged?
>   - Number of emails, anything unanswered?
>
> Conferences
> / Meetups
>
>- Any conference in next 1 month where gluster-developers are going?
>gluster-users are going? So we can meet and discuss.
>
> Developer
> focus
>
>-
>
>Any design specs to discuss?
>-
>
>Metrics of the week?
>- Coverity
>   - Clang-Scan
>   - Number of patches from new developers.
>   - Did we increase test coverage?
>   - [Atin] Also talk about most frequent test failures in the CI and
>   carve out an AI to get them fixed.
>
> RoundTable
>
>- 
>
> 
>
> Regards,
> Amar
>
> On Mon, Mar 25, 2019 at 8:53 PM Amar Tumballi Suryanarayan <
> atumb...@redhat.com> wrote:
>
>> Thanks for the feedback Darrell,
>>
>> The new proposal is to have one in North America 'morning' time. (10AM
>> PST), And another in ASIA day time, which is evening 7pm/6pm in Australia,
>> 9pm Newzealand, 5pm Tokyo, 4pm Beijing.
>>
>> For example, if we choose Every other Tuesday for meeting, and 1st of the
>> month is Tuesday, we would have North America time for 1st, and on 15th it
>> would be ASIA/Pacific time.
>>
>> Hopefully, this way, we can cover all the timezones, and meeting minutes
>> would be committed to github repo, so that way, it will be easier for
>> everyone to be aware of what is happening.
>>
>> Regards,
>> Amar
>>
>> On Mon, Mar 25, 2019 at 8:40 PM Darrell Budic 
>> wrote:
>>
>>> As a user, I’d like to visit more of these, but the time slot is my 3AM.
>>> Any possibility for a rolling schedule (move meeting +6 hours each week
>>> with rolling attendance from maintainers?) or an occasional regional
>>> meeting 12 hours opposed to the one you’re proposing?
>>>
>>>   -Darrell
>>>
>>> On Mar 25, 2019, at 4:25 AM, Amar Tumballi Suryanarayan <
>>> atumb...@redhat.com> wrote:
>>>
>>> All,
>>>
>>> We currently have 3 meetings which are public:
>>>
>>> 1. Maintainer's Meeting
>>>
>>> - Runs once in 2 weeks (on Mondays), and current attendance is around
>>> 3-5 on an avg, and not much is discussed.
>>> - Without majority attendance, we can't take any decisions too.
>>>
>>> 2. Community meeting
>>>
>>> - 

Re: [Gluster-devel] test failure reports for last 15 days

[FNU Raghavendra Manjunath]

===> (export directory is not
present)
[2019-04-10 15:35:52.117735]:++ G_LOG:./tests/basic/uss.t: TEST: 44
_GFS --attribute-timeout=0 --entry-timeout=0 --volfile-server=
builder205.int.aws.gluster.org --volfile-id=patchy /mnt/glusterfs/0
++


I suspect something wrong with the cleanup sequence which causes the
timeout of the test in the 1st iteration and the export directory issues in
the next iteration causes the failure of uss.t in the 2nd iteration.


Regards,
Raghavendra



On Wed, Apr 10, 2019 at 4:07 PM FNU Raghavendra Manjunath 
wrote:

>
>
> On Wed, Apr 10, 2019 at 9:59 AM Atin Mukherjee 
> wrote:
>
>> And now for last 15 days:
>>
>>
>> https://fstat.gluster.org/summary?start_date=2019-03-25_date=2019-04-10
>>
>> ./tests/bitrot/bug-1373520.t 18  ==> Fixed through
>> https://review.gluster.org/#/c/glusterfs/+/22481/, I don't see this
>> failing in brick mux post 5th April
>>
>
> The above patch has been sent to fix the failure with brick mux enabled.
>
>
>> ./tests/bugs/ec/bug-1236065.t 17  ==> happens only in brick mux,
>> needs analysis.
>> ./tests/basic/uss.t 15  ==> happens in both brick mux and non
>> brick mux runs, test just simply times out. Needs urgent analysis.
>>
>
> Nothing has changed in snapview-server and snapview-client recently.
> Looking into it.
>
> ./tests/basic/ec/ec-fix-openfd.t 13  ==> Fixed through
>> https://review.gluster.org/#/c/22508/ , patch merged today.
>> ./tests/basic/volfile-sanity.t  8  ==> Some race, though this
>> succeeds in second attempt every time.
>>
>> There're plenty more with 5 instances of failure from many tests. We need
>> all maintainers/owners to look through these failures and fix them, we
>> certainly don't want to get into a stage where master is unstable and we
>> have to lock down the merges till all these failures are resolved. So
>> please help.
>>
>> (Please note fstat stats show up the retries as failures too which in a
>> way is right)
>>
>>
>> On Tue, Feb 26, 2019 at 5:27 PM Atin Mukherjee 
>> wrote:
>>
>>> [1] captures the test failures report since last 30 days and we'd need
>>> volunteers/component owners to see why the number of failures are so high
>>> against few tests.
>>>
>>> [1]
>>> https://fstat.gluster.org/summary?start_date=2019-01-26_date=2019-02-25=all
>>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] test failure reports for last 15 days

[FNU Raghavendra Manjunath]

On Wed, Apr 10, 2019 at 9:59 AM Atin Mukherjee  wrote:

> And now for last 15 days:
>
> https://fstat.gluster.org/summary?start_date=2019-03-25_date=2019-04-10
>
> ./tests/bitrot/bug-1373520.t 18  ==> Fixed through
> https://review.gluster.org/#/c/glusterfs/+/22481/, I don't see this
> failing in brick mux post 5th April
>

The above patch has been sent to fix the failure with brick mux enabled.


> ./tests/bugs/ec/bug-1236065.t 17  ==> happens only in brick mux, needs
> analysis.
> ./tests/basic/uss.t 15  ==> happens in both brick mux and non
> brick mux runs, test just simply times out. Needs urgent analysis.
>

Nothing has changed in snapview-server and snapview-client recently.
Looking into it.

./tests/basic/ec/ec-fix-openfd.t 13  ==> Fixed through
> https://review.gluster.org/#/c/22508/ , patch merged today.
> ./tests/basic/volfile-sanity.t  8  ==> Some race, though this succeeds
> in second attempt every time.
>
> There're plenty more with 5 instances of failure from many tests. We need
> all maintainers/owners to look through these failures and fix them, we
> certainly don't want to get into a stage where master is unstable and we
> have to lock down the merges till all these failures are resolved. So
> please help.
>
> (Please note fstat stats show up the retries as failures too which in a
> way is right)
>
>
> On Tue, Feb 26, 2019 at 5:27 PM Atin Mukherjee 
> wrote:
>
>> [1] captures the test failures report since last 30 days and we'd need
>> volunteers/component owners to see why the number of failures are so high
>> against few tests.
>>
>> [1]
>> https://fstat.gluster.org/summary?start_date=2019-01-26_date=2019-02-25=all
>>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Release 5: Branched and further dates

[FNU Raghavendra Manjunath]

On Thu, Oct 4, 2018 at 2:47 PM Kotresh Hiremath Ravishankar <
khire...@redhat.com> wrote:

>
>
> On Thu, Oct 4, 2018 at 9:03 PM Shyam Ranganathan 
> wrote:
>
>> On 09/13/2018 11:10 AM, Shyam Ranganathan wrote:
>> > RC1 would be around 24th of Sep. with final release tagging around 1st
>> > of Oct.
>>
>> RC1 now stands to be tagged tomorrow, and patches that are being
>> targeted for a back port include,
>>
>> 1) https://review.gluster.org/c/glusterfs/+/21314 (snapshot volfile in
>> mux cases)
>>
>> @RaBhat working on this.
>>
>>

The following patch addresses the issue in release-5 branch.

https://review.gluster.org/#/c/glusterfs/+/21347/


Regards,
Raghavendra

> 2) Py3 corrections in master
>>
>> @Kotresh are all changes made to master backported to release-5 (may not
>> be merged, but looking at if they are backported and ready for merge)?
>>
>
> All changes made to master are backported to release-5. But py3 support is
> still not complete.
>
>>
>> 3) Release notes review and updates with GD2 content pending
>>
>> @Kaushal/GD2 team can we get the updates as required?
>> https://review.gluster.org/c/glusterfs/+/21303
>>
>> 4) This bug [2] was filed when we released 4.0.
>>
>> The issue has not bitten us in 4.0 or in 4.1 (yet!) (i.e the options
>> missing and hence post-upgrade clients failing the mount). This is
>> possibly the last chance to fix it.
>>
>> Glusterd and protocol maintainers, can you chime in, if this bug needs
>> to be and can be fixed? (thanks to @anoopcs for pointing it out to me)
>>
>> The tracker bug [1] does not have any other blockers against it, hence
>> assuming we are not tracking/waiting on anything other than the set above.
>>
>> Thanks,
>> Shyam
>>
>> [1] Tracker: https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-5.0
>> [2] Potential upgrade bug:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1540659
>>
>
>
> --
> Thanks and Regards,
> Kotresh H R
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Memory overwrites due to processing vol files???

[FNU Raghavendra Manjunath]

On Fri, Sep 28, 2018 at 4:01 PM Shyam Ranganathan 
wrote:

> We tested with ASAN and without the fix at [1], and it consistently
> crashes at the mdcache xlator when brick mux is enabled.
> On 09/28/2018 03:50 PM, FNU Raghavendra Manjunath wrote:
> >
> > I was looking into the issue and  this is what I could find while
> > working with shyam.
> >
> > There are 2 things here.
> >
> > 1) The multiplexed brick process for the snapshot(s) getting the client
> > volfile (I suspect, it happened
> >  when restore operation was performed).
> > 2) Memory corruption happening while the multiplexed brick process is
> > building the graph (for the client
> >  volfile it got above)
> >
> > I have been able to reproduce the issue in my local computer once, when
> > I ran the testcase tests/bugs/snapshot/bug-1275616.t
> >
> > Upon comparison, we found that the backtrace of the core I got and the
> > core generated in the regression runs was similar.
> > In fact, the victim information shyam mentioned before, is also similar
> > in the core that I was able to get.
> >
> > On top of that, when the brick process was run with valgrind, it
> > reported following memory corruption
> >
> > ==31257== Conditional jump or move depends on uninitialised value(s)
> > ==31257==at 0x1A7D0564: mdc_xattr_list_populate (md-cache.c:3127)
> > ==31257==by 0x1A7D1903: mdc_init (md-cache.c:3486)
> > ==31257==by 0x4E62D41: __xlator_init (xlator.c:684)
> > ==31257==by 0x4E62E67: xlator_init (xlator.c:709)
> > ==31257==by 0x4EB2BEB: glusterfs_graph_init (graph.c:359)
> > ==31257==by 0x4EB37F8: glusterfs_graph_activate (graph.c:722)
> > ==31257==by 0x40AEC3: glusterfs_process_volfp (glusterfsd.c:2528)
> > ==31257==by 0x410868: mgmt_getspec_cbk (glusterfsd-mgmt.c:2076)
> > ==31257==by 0x518408D: rpc_clnt_handle_reply (rpc-clnt.c:755)
> > ==31257==by 0x51845C1: rpc_clnt_notify (rpc-clnt.c:923)
> > ==31257==by 0x518084E: rpc_transport_notify (rpc-transport.c:525)
> > ==31257==by 0x123273DF: socket_event_poll_in (socket.c:2504)
> > ==31257==  Uninitialised value was created by a heap allocation
> > ==31257==at 0x4C2DB9D: malloc (vg_replace_malloc.c:299)
> > ==31257==by 0x4E9F58E: __gf_malloc (mem-pool.c:136)
> > ==31257==by 0x1A7D052A: mdc_xattr_list_populate (md-cache.c:3123)
> > ==31257==by 0x1A7D1903: mdc_init (md-cache.c:3486)
> > ==31257==by 0x4E62D41: __xlator_init (xlator.c:684)
> > ==31257==by 0x4E62E67: xlator_init (xlator.c:709)
> > ==31257==by 0x4EB2BEB: glusterfs_graph_init (graph.c:359)
> > ==31257==by 0x4EB37F8: glusterfs_graph_activate (graph.c:722)
> > ==31257==by 0x40AEC3: glusterfs_process_volfp (glusterfsd.c:2528)
> > ==31257==by 0x410868: mgmt_getspec_cbk (glusterfsd-mgmt.c:2076)
> > ==31257==by 0x518408D: rpc_clnt_handle_reply (rpc-clnt.c:755)
> > ==31257==by 0x51845C1: rpc_clnt_notify (rpc-clnt.c:923)
> >
> > Based on the above observations, I think the below patch  by Shyam
> > should fix the crash.
>
> [1]
>
> > https://review.gluster.org/#/c/glusterfs/+/21299/
> >
> > But, I am still trying understand, why a brick process should get a
> > client volfile (i.e. the 1st issue mentioned above).
> >
>

It was glusterd which was giving the client volfile instead of the brick
volfile.

The following patch has been submitted for review to address the cause of
this problem.

https://review.gluster.org/#/c/glusterfs/+/21314/

Regards,
Raghavendra


> > Regards,
> > Raghavendra
> >
> > On Wed, Sep 26, 2018 at 9:00 PM Shyam Ranganathan  > <mailto:srang...@redhat.com>> wrote:
> >
> > On 09/26/2018 10:21 AM, Shyam Ranganathan wrote:
> > > 2. Testing dashboard to maintain release health (new, thanks Nigel)
> > >   - Dashboard at [2]
> > >   - We already have 3 failures here as follows, needs attention
> from
> > > appropriate *maintainers*,
> > > (a)
> > >
> >
> https://build.gluster.org/job/regression-test-with-multiplex/871/consoleText
> > >   - Failed with core:
> > ./tests/basic/afr/gfid-mismatch-resolution-with-cli.t
> > > (b)
> > >
> >
> https://build.gluster.org/job/regression-test-with-multiplex/873/consoleText
> > >   - Failed with core: ./tests/bugs/snapshot/bug-1275616.t
> > >   - Also test ./tests/bugs/glusterd/validating-server-quorum.t
> > had to be
> > > retried
> >
> > I was lookin

Re: [Gluster-devel] [Gluster-Maintainers] Memory overwrites due to processing vol files???

[FNU Raghavendra Manjunath]

I was looking into the issue and  this is what I could find while working
with shyam.

There are 2 things here.

1) The multiplexed brick process for the snapshot(s) getting the client
volfile (I suspect, it happened
 when restore operation was performed).
2) Memory corruption happening while the multiplexed brick process is
building the graph (for the client
 volfile it got above)

I have been able to reproduce the issue in my local computer once, when I
ran the testcase tests/bugs/snapshot/bug-1275616.t

Upon comparison, we found that the backtrace of the core I got and the core
generated in the regression runs was similar.
In fact, the victim information shyam mentioned before, is also similar in
the core that I was able to get.

On top of that, when the brick process was run with valgrind, it reported
following memory corruption

==31257== Conditional jump or move depends on uninitialised value(s)
==31257==at 0x1A7D0564: mdc_xattr_list_populate (md-cache.c:3127)
==31257==by 0x1A7D1903: mdc_init (md-cache.c:3486)
==31257==by 0x4E62D41: __xlator_init (xlator.c:684)
==31257==by 0x4E62E67: xlator_init (xlator.c:709)
==31257==by 0x4EB2BEB: glusterfs_graph_init (graph.c:359)
==31257==by 0x4EB37F8: glusterfs_graph_activate (graph.c:722)
==31257==by 0x40AEC3: glusterfs_process_volfp (glusterfsd.c:2528)
==31257==by 0x410868: mgmt_getspec_cbk (glusterfsd-mgmt.c:2076)
==31257==by 0x518408D: rpc_clnt_handle_reply (rpc-clnt.c:755)
==31257==by 0x51845C1: rpc_clnt_notify (rpc-clnt.c:923)
==31257==by 0x518084E: rpc_transport_notify (rpc-transport.c:525)
==31257==by 0x123273DF: socket_event_poll_in (socket.c:2504)
==31257==  Uninitialised value was created by a heap allocation
==31257==at 0x4C2DB9D: malloc (vg_replace_malloc.c:299)
==31257==by 0x4E9F58E: __gf_malloc (mem-pool.c:136)
==31257==by 0x1A7D052A: mdc_xattr_list_populate (md-cache.c:3123)
==31257==by 0x1A7D1903: mdc_init (md-cache.c:3486)
==31257==by 0x4E62D41: __xlator_init (xlator.c:684)
==31257==by 0x4E62E67: xlator_init (xlator.c:709)
==31257==by 0x4EB2BEB: glusterfs_graph_init (graph.c:359)
==31257==by 0x4EB37F8: glusterfs_graph_activate (graph.c:722)
==31257==by 0x40AEC3: glusterfs_process_volfp (glusterfsd.c:2528)
==31257==by 0x410868: mgmt_getspec_cbk (glusterfsd-mgmt.c:2076)
==31257==by 0x518408D: rpc_clnt_handle_reply (rpc-clnt.c:755)
==31257==by 0x51845C1: rpc_clnt_notify (rpc-clnt.c:923)

Based on the above observations, I think the below patch  by Shyam should
fix the crash.
https://review.gluster.org/#/c/glusterfs/+/21299/

But, I am still trying understand, why a brick process should get a client
volfile (i.e. the 1st issue mentioned above).

Regards,
Raghavendra

On Wed, Sep 26, 2018 at 9:00 PM Shyam Ranganathan 
wrote:

> On 09/26/2018 10:21 AM, Shyam Ranganathan wrote:
> > 2. Testing dashboard to maintain release health (new, thanks Nigel)
> >   - Dashboard at [2]
> >   - We already have 3 failures here as follows, needs attention from
> > appropriate *maintainers*,
> > (a)
> >
> https://build.gluster.org/job/regression-test-with-multiplex/871/consoleText
> >   - Failed with core:
> ./tests/basic/afr/gfid-mismatch-resolution-with-cli.t
> > (b)
> >
> https://build.gluster.org/job/regression-test-with-multiplex/873/consoleText
> >   - Failed with core: ./tests/bugs/snapshot/bug-1275616.t
> >   - Also test ./tests/bugs/glusterd/validating-server-quorum.t had
> to be
> > retried
>
> I was looking at the cores from the above 2 instances, the one in job
> 873 is been a typical pattern, where malloc fails as there is internal
> header corruption in the free bins.
>
> When examining the victim that would have been allocated, it is often
> carrying incorrect size and other magic information. If the data in
> victim is investigated it looks like a volfile.
>
> With the crash in 871, I thought there maybe a point where this is
> detected earlier, but not able to make headway in the same.
>
> So, what could be corrupting this memory and is it when the graph is
> being processed? Can we run this with ASAN or such (I have not tried,
> but need pointers if anyone has run tests with ASAN).
>
> Here is the (brief) stack analysis of the core in 873:
> NOTE: we need to start avoiding flushing the logs when we are dumping
> core, as that leads to more memory allocations and causes a sort of
> double fault in such cases.
>
> Core was generated by `/build/install/sbin/glusterfsd -s
> builder101.cloud.gluster.org --volfile-id /sn'.
> Program terminated with signal 6, Aborted.
> #0  0x7f23cf590277 in __GI_raise (sig=sig@entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> 56return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
> (gdb) bt
> #0  0x7f23cf590277 in __GI_raise (sig=sig@entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x7f23cf591968 in __GI_abort () at abort.c:90
> #2  0x7f23cf5d2d37 in 

Re: [Gluster-devel] Need inputs on patch #17985

[FNU Raghavendra Manjunath]

>From snapview client perspective one important thing to note. For building
the context for the entry point (by default ".snaps") a explicit lookup has
to be done on it. The dentry for ".snaps" is not returned when readdir is
done on its parent directory (Not even when ls -a is done). So for building
the context of .snaps (in the context snapview client saves the information
about whether it is a real inode or virtual inode) we need a lookup.

>From snapview server perspective as well a lookup might be needed. In
snapview server a glfs handle is established between the snapview server
and the snapshot brick. So a inode in snapview server process contains the
glfs handle for the object being accessed from snapshot.  In snapview
server readdirp does not build the inode context (which contains the glfs
handle etc) because glfs handle is returned only in lookup.

Regards,
Raghavendra


On Tue, Aug 29, 2017 at 12:53 AM, Raghavendra Gowdappa 
wrote:

>
>
> - Original Message -
> > From: "Raghavendra G" 
> > To: "Nithya Balachandran" 
> > Cc: "Raghavendra Gowdappa" , anoo...@redhat.com,
> "Gluster Devel" ,
> > raghaven...@redhat.com
> > Sent: Tuesday, August 29, 2017 8:52:28 AM
> > Subject: Re: [Gluster-devel] Need inputs on patch #17985
> >
> > On Thu, Aug 24, 2017 at 2:53 PM, Nithya Balachandran <
> nbala...@redhat.com>
> > wrote:
> >
> > > It has been a while but iirc snapview client (loaded abt dht/tier etc)
> had
> > > some issues when we ran tiering tests. Rafi might have more info on
> this -
> > > basically it was expecting to find the inode_ctx populated but it was
> not.
> > >
> >
> > Thanks Nithya. @Rafi, @Raghavendra Bhat, is it possible to take the
> > ownership of,
> >
> > * Identifying whether the patch in question causes the issue?
>
> gf_svc_readdirp_cbk is setting relevant state in inode [1]. I quickly
> checked whether its the same state stored by gf_svc_lookup_cbk and it looks
> like the same state. So, I guess readdirp is handled correctly by
> snapview-client and an explicit lookup is not required. But, will wait for
> inputs from rabhat and rafi.
>
> [1] https://github.com/gluster/glusterfs/blob/master/xlators/
> features/snapview-client/src/snapview-client.c#L1962
>
> > * Send a fix or at least evaluate whether a fix is possible.
> >
> > @Others,
> >
> > With the motivation of getting some traction on this, Is it ok if we:
> > * Set a deadline of around 15 days to complete the review (or testing
> with
> > the patch in question) of respective components and to come up with
> issues
> > (if any).
> > * Post the deadline, if there are no open issues, go ahead and merge the
> > patch?
> >
> > If time is not enough, let us know and we can come up with a reasonable
> > time.
> >
> > regards,
> > Raghavendra
> >
> >
> > > On 24 August 2017 at 10:13, Raghavendra G 
> > > wrote:
> > >
> > >> Note that we need to consider xlators on brick stack too. I've added
> > >> maintainers/peers of xlators on brick stack. Please explicitly
> ack/nack
> > >> whether this patch affects your component.
> > >>
> > >> For reference, following are the xlators loaded in brick stack
> > >>
> > >> storage/posix
> > >> features/trash
> > >> features/changetimerecorder
> > >> features/changelog
> > >> features/bitrot-stub
> > >> features/access-control
> > >> features/locks
> > >> features/worm
> > >> features/read-only
> > >> features/leases
> > >> features/upcall
> > >> performance/io-threads
> > >> features/selinux
> > >> features/marker
> > >> features/barrier
> > >> features/index
> > >> features/quota
> > >> debug/io-stats
> > >> performance/decompounder
> > >> protocol/server
> > >>
> > >>
> > >> For those not following this thread, the question we need to answer
> is,
> > >> "whether the xlator you are associated with works fine if a non-lookup
> > >> fop (like open, setattr, stat etc) hits it without a lookup ever being
> > >> done
> > >> on that inode"
> > >>
> > >> regards,
> > >> Raghavendra
> > >>
> > >> On Wed, Aug 23, 2017 at 11:56 AM, Raghavendra Gowdappa <
> > >> rgowd...@redhat.com> wrote:
> > >>
> > >>> Thanks Pranith and Ashish for your inputs.
> > >>>
> > >>> - Original Message -
> > >>> > From: "Pranith Kumar Karampuri" 
> > >>> > To: "Ashish Pandey" 
> > >>> > Cc: "Raghavendra Gowdappa" , "Xavier
> Hernandez" <
> > >>> xhernan...@datalab.es>, "Gluster Devel"
> > >>> > 
> > >>> > Sent: Wednesday, August 23, 2017 11:55:19 AM
> > >>> > Subject: Re: Need inputs on patch #17985
> > >>> >
> > >>> > Raghavendra,
> > >>> > As Ashish mentioned, there aren't any known problems if upper
> > >>> xlators
> > >>> > don't send lookups in EC at the moment.
> > >>> >
> > >>> > On Wed, Aug 23, 2017 at 9:07 AM, Ashish Pandey <
> aspan...@redhat.com>
> > >>> wrote:
> > 

Re: [Gluster-devel] [Gluster-Maintainers] Gluster Test Thursday - Release 3.9

[FNU Raghavendra Manjunath]

Tested Bitrot related aspects. Created data, enabled bitrot and created
more data. The files were signed by the bitrot daemon. Simulated the
corruption by editing a file directly in the backend.
Triggered scrubbing (on demand). Found that the corrupted files were marked
bad by the scrubber.

Also ran general tests such as compiling gluster code base on the mount
point, dbench. The tests passed properly.

Still running some more tests. Will keep updated.

Regards,
Raghavendra


On Fri, Nov 4, 2016 at 12:43 AM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On Thu, Nov 3, 2016 at 4:42 PM, Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>>
>>
>> On Thu, Nov 3, 2016 at 9:55 AM, Pranith Kumar Karampuri <
>> pkara...@redhat.com> wrote:
>>
>>>
>>>
>>> On Wed, Nov 2, 2016 at 7:00 PM, Krutika Dhananjay 
>>> wrote:
>>>
 Just finished testing VM storage use-case.

 *Volume configuration used:*

 [root@srv-1 ~]# gluster volume info

 Volume Name: rep
 Type: Replicate
 Volume ID: 2c603783-c1da-49b7-8100-0238c777b731
 Status: Started
 Snapshot Count: 0
 Number of Bricks: 1 x 3 = 3
 Transport-type: tcp
 Bricks:
 Brick1: srv-1:/bricks/rep1
 Brick2: srv-2:/bricks/rep2
 Brick3: srv-3:/bricks/rep4
 Options Reconfigured:
 nfs.disable: on
 performance.readdir-ahead: on
 transport.address-family: inet
 performance.quick-read: off
 performance.read-ahead: off
 performance.io-cache: off
 performance.stat-prefetch: off
 cluster.eager-lock: enable
 network.remote-dio: enable
 cluster.quorum-type: auto
 cluster.server-quorum-type: server
 features.shard: on
 cluster.granular-entry-heal: on
 cluster.locking-scheme: granular
 network.ping-timeout: 30
 server.allow-insecure: on
 storage.owner-uid: 107
 storage.owner-gid: 107
 cluster.data-self-heal-algorithm: full

 Used FUSE to mount the volume locally on each of the 3 nodes (no
 external clients).
 shard-block-size - 4MB.

 *TESTS AND RESULTS:*

 *What works:*

 * Created 3 vm images, one per hypervisor. Installed fedora 24 on all
 of them.
   Used virt-manager for ease of setting up the environment.
 Installation went fine. All green.

 * Rebooted the vms. Worked fine.

 * Killed brick-1. Ran dd on the three vms to create a 'src' file.
 Captured their md5sum value. Verified that
 the gfid indices and name indices are created under
 .glusterfs/indices/xattrop and .glusterfs/indices/entry-changes
 respectively as they should. Brought the brick back up. Waited until heal
 completed. Captured md5sum again. They matched.

 * Killed brick-2. Copied 'src' file from the step above into new file
 using dd. Captured md5sum on the newly created file.
 Checksum matched. Waited for heal to finish. Captured md5sum again.
 Everything matched.

 * Repeated the test above with brick-3 being killed and brought back up
 after a while. Worked fine.

 At the end I also captured md5sums from the backend of the shards on
 the three replicas. They all were found to be
 in sync. So far so good.

 *What did NOT work:*

 * Started dd again on all 3 vms to copy the existing files to new
 files. While dd was running, I ran replace-brick to replace the third brick
 with a new brick on the same node with a different path. This caused dd on
 all three vms to simultaneously fail with "Input/Output error". I tried to
 read off the files, even that failed. Rebooted the vms. By this time,
 /.shard is in
 split-brain as per heal-info. And the vms seem to have suffered
 corruption and are in an irrecoverable state.

 I checked the logs. The pattern is very much similar to the one in the
 add-brick bug Lindsay reported here - https://bugzilla.redhat.com/sh
 ow_bug.cgi?id=1387878. Seems like something is going wrong each time
 there is a graph switch.

 @Aravinda and Pranith:

 I will need some time to debug this, if 3.9 release can wait until it
 is RC'd and fixed.
 Otherwise we will need to caution the users to not do replace-brick,
 add-brick etc (or any form of graph switch for that matter) *might* cause
 vm corruption, irrespective of whether the users are using FUSE or gfapi,
 in 3.9.0.

 Let me know what your decision is.

>>>
>>> Since this bug is not a regression let us document this as a known
>>> issue. Let us do our best to get the fix in next release.
>>>
>>> I am almost done with testing afr and ec.
>>>
>>> For afr, leaks etc were not there in the tests I did.
>>> But I am seeing performance drop for crawling related tests.
>>>
>>> This is with 3.9.0rc2
>>> running directory_crawl_create ... done (252.91 secs)
>>> running directory_crawl ... done (104.83 secs)
>>> running 

Re: [Gluster-devel] Checklist for Bitrot component for upstream release

[FNU Raghavendra Manjunath]

I would recommend the following tests.

1) The tests in our regression tests
2) Creating  data set of many files (of different sizes) and then enabling
bit-rot on the volume.
3) scrubber throttling
4) Tests such as open + write + close and again open + write + close (i.e.
before the bit-rot daemon can sign the file the file is modified again)
5) Exceed the lru size of the inode table. (So that inodes are forgotten).
The next lookup of the file should properly populate all the necessary
details into the inode context.
6) Simulating the corruption by directly changing a file in the backend and
checking if its marked as bad or not. (Also access to such bad files should
give EIO)
7) Change scrubbing intervals and check if it kicks in correctly.

Kotresh,

Can you please add any other tests that should be considered?

Regards,
Raghavendra


On Fri, Sep 2, 2016 at 2:25 PM, Pranith Kumar Karampuri  wrote:

>
>
> On Fri, Sep 2, 2016 at 11:39 PM, Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>> hi,
>> Did you get a chance to decide on the tests that need to be done
>> before doing a release for Bitrot component? Could you let me know who will
>> be providing with the list?
>>
>> I can update it at https://public.pad.fsfe.org/p/
>> gluster-component-release-checklist
>>
>> --
>> Pranith
>>
> Sorry, forgot to add 'Aravinda & Pranith'
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Fuse client hangs on doing multithreading IO tests

[FNU Raghavendra Manjunath]

Hi,

Any idea how big were the files that were being read?

Can you please attach the logs from all the gluster server and client
nodes? (the logs can be found in /var/log/glusterfs)

Also please provide the /var/log/messages from all the server and client
nodes.

Regards,
Raghavendra


On Fri, Jun 24, 2016 at 10:32 AM, 冷波  wrote:

> Hi,
>
>
> We found a problem when doing traffic tests. We created a replicated
> volume with two storage nodes (CentOS 6.5). There was one FUSE client
> (CentOS 6.7) which did multi-threading reads and writes. Most of IOs are
> reads for big files. All machines used 10Gbe NICs. And the typical read
> throught was 4-6Gbps (0.5-1.5GB/s).
>
>
> After the test ran several minutes, the test program hung. The throughput
> suddenly dropped to zero. Then there was no traffic any more. If we ran df,
> df would hang, too. But we could still read or write the volume from other
> clients.
>
>
> We tried several GlusterFS version from 3.7.5 to 3.8.0. Each version had
> this problem. We also tried to restore default GlusterFS options, but the
> problem still existed.
>
>
> The GlusterFS version was 3.7.11 for the following stacks.
>
>
> This was the stack of dd when hanging:
>
> [] wait_answer_interruptible+0x81/0xc0 [fuse]
>
> [] __fuse_request_send+0x1db/0x2b0 [fuse]
>
> [] fuse_request_send+0x12/0x20 [fuse]
>
> [] fuse_statfs+0xda/0x150 [fuse]
>
> [] statfs_by_dentry+0x74/0xa0
>
> [] vfs_statfs+0x1b/0xb0
>
> [] user_statfs+0x47/0xb0
>
> [] sys_statfs+0x2a/0x50
>
> [] system_call_fastpath+0x16/0x1b
>
> [] 0x
>
>
> This was the stack of gluster:
>
> [] futex_wait_queue_me+0xba/0xf0
>
> [] futex_wait+0x1c0/0x310
>
> [] do_futex+0x121/0xae0
>
> [] sys_futex+0x7b/0x170
>
> [] system_call_fastpath+0x16/0x1b
>
> [] 0x
>
>
> This was the stack of the test program:
>
> [] hrtimer_nanosleep+0xc4/0x180
>
> [] sys_nanosleep+0x6e/0x80
>
> [] system_call_fastpath+0x16/0x1b
>
> [] 0x
>
>
> Any clue?
>
> Thanks,
> Paul
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Release Management Process change - proposal

[FNU Raghavendra Manjunath]

+1.



On Tue, May 10, 2016 at 2:58 AM, Kaushal M  wrote:

> On Tue, May 10, 2016 at 12:01 AM, Vijay Bellur  wrote:
> > Hi All,
> >
> > We are blocked on 3.7.12 owing to this proposal. Appreciate any
> > feedback on this!
> >
> > Thanks,
> > Vijay
> >
> > On Thu, Apr 28, 2016 at 11:58 PM, Vijay Bellur 
> wrote:
> >> Hi All,
> >>
> >> We have encountered a spate of regressions in recent 3.7.x releases. The
> >> 3.7.x maintainers are facing additional burdens to ensure functional,
> >> performance and upgrade correctness. I feel component maintainers
> should own
> >> these aspects of stability as we own the components and understand our
> >> components better than anybody else. In order to have more active
> >> participation from maintainers for every release going forward, I
> propose
> >> this process:
> >>
> >> 1. All component maintainers will need to provide an explicit ack about
> the
> >> content and quality of their respective components before a release is
> >> tagged.
> >>
> >> 2. A release will not be tagged if any component is not acked by a
> >> maintainer.
> >>
> >> 3. Release managers will co-ordinate getting acks from maintainers and
> >> perform necessary housekeeping (closing bugs etc.).
> >>
> >> This is not entirely new and a part of this process has been outlined
> in the
> >> Guidelines for Maintainers [1] document. I am inclined to enforce this
> >> process with more vigor to ensure that we do better on quality &
> stability.
> >>
> >> Thoughts, questions and feedback about the process are very welcome!
> >>
>
> +1 from me. Spreading out the verification duties will help us do
> better releases.
>
> >> Thanks,
> >> Vijay
> >>
> >> [1]
> >>
> http://www.gluster.org/community/documentation/index.php/Guidelines_For_Maintainers
> >>
> >>
> > ___
> > maintainers mailing list
> > maintain...@gluster.org
> > http://www.gluster.org/mailman/listinfo/maintainers
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Testcase broken due to posix iatt commit

[FNU Raghavendra Manjunath]

Root inode gets linked during mount itselsf.

We get lookups on root. But since root inode is always present in the inode
table, after the mount if there is a fop on an entry whose parent is root,
then that fop will be able to find the parent in the inode table as part of
resolve. So it need not explicitly send a hard resolution for resolving
parent inode.

Du? Did I miss anything?

Regards,
Raghavendra
On Mar 28, 2016 11:02 AM, "Vijaikumar Mallikarjuna" <vmall...@redhat.com>
wrote:

> Hi Du/Johny,
>
> Don't we get a lookup on root inode? Please confirm.
>
> If we don't get lookup on root inode, then we need to create inode-ctx in
> a posix_acl_ctx_get() function.
>
> Thanks,
> Vijay
> On 28-Mar-2016 7:37 PM, "FNU Raghavendra Manjunath" <rab...@redhat.com>
> wrote:
>
>> CCing Vijay Kumar who made the acl related changes in that patch.
>>
>> Vijay? Can you please look into it?
>>
>> Regards,
>> Raghavendra
>>
>> On Mon, Mar 28, 2016 at 9:57 AM, Avra Sengupta <aseng...@redhat.com>
>> wrote:
>>
>>> Hi Raghavendra,
>>>
>>> As part of the patch (http://review.gluster.org/#/c/13730/16), the
>>> inode_ctx is not created in posix_acl_ctx_get(). Because of this the
>>> testcase in http://review.gluster.org/#/c/13623/ breaks. It fails with
>>> the following logs:
>>>
>>> [2016-03-28 13:43:39.216168] D [MSGID: 0]
>>> [io-threads.c:346:iot_schedule] 0-patchy-io-threads: CREATE scheduled as
>>> normal fop
>>> [2016-03-28 13:43:39.216495] E [posix-acl.c:199:acl_permits] (-->
>>> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x1ba)[0x7f6fea72780a]
>>> (-->
>>> /usr/local/lib/glusterfs/3.8dev/xlator/features/access-control.so(+0x49c4)[0x7f6fde5499c4]
>>> (-->
>>> /usr/local/lib/glusterfs/3.8dev/xlator/features/access-control.so(+0xa855)[0x7f6fde54f855]
>>> (-->
>>> /usr/local/lib/glusterfs/3.8dev/xlator/features/locks.so(+0xd37e)[0x7f6fde33837e]
>>> (-->
>>> /usr/local/lib/glusterfs/3.8dev/xlator/features/upcall.so(+0x640f)[0x7f6fde12040f]
>>> ) 0-patchy-access-control: inode ctx is NULL for
>>> ----0001
>>> [2016-03-28 13:43:39.216544] I [MSGID: 115071]
>>> [server-rpc-fops.c:1612:server_create_cbk] 0-patchy-server: 8: CREATE
>>> /file1 (----0001/file1) ==> (Permission denied)
>>> [Permission denied]
>>>
>>> Is it because we missed the inode_ctx creation that was being done by
>>> posix_acl_ctx_get() previously? Can you please shed some light on this
>>>
>>> Regards,
>>> Avra
>>>
>>>
>>>
>>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Testcase broken due to posix iatt commit

[FNU Raghavendra Manjunath]

CCing Vijay Kumar who made the acl related changes in that patch.

Vijay? Can you please look into it?

Regards,
Raghavendra

On Mon, Mar 28, 2016 at 9:57 AM, Avra Sengupta  wrote:

> Hi Raghavendra,
>
> As part of the patch (http://review.gluster.org/#/c/13730/16), the
> inode_ctx is not created in posix_acl_ctx_get(). Because of this the
> testcase in http://review.gluster.org/#/c/13623/ breaks. It fails with
> the following logs:
>
> [2016-03-28 13:43:39.216168] D [MSGID: 0] [io-threads.c:346:iot_schedule]
> 0-patchy-io-threads: CREATE scheduled as normal fop
> [2016-03-28 13:43:39.216495] E [posix-acl.c:199:acl_permits] (-->
> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x1ba)[0x7f6fea72780a]
> (-->
> /usr/local/lib/glusterfs/3.8dev/xlator/features/access-control.so(+0x49c4)[0x7f6fde5499c4]
> (-->
> /usr/local/lib/glusterfs/3.8dev/xlator/features/access-control.so(+0xa855)[0x7f6fde54f855]
> (-->
> /usr/local/lib/glusterfs/3.8dev/xlator/features/locks.so(+0xd37e)[0x7f6fde33837e]
> (-->
> /usr/local/lib/glusterfs/3.8dev/xlator/features/upcall.so(+0x640f)[0x7f6fde12040f]
> ) 0-patchy-access-control: inode ctx is NULL for
> ----0001
> [2016-03-28 13:43:39.216544] I [MSGID: 115071]
> [server-rpc-fops.c:1612:server_create_cbk] 0-patchy-server: 8: CREATE
> /file1 (----0001/file1) ==> (Permission denied)
> [Permission denied]
>
> Is it because we missed the inode_ctx creation that was being done by
> posix_acl_ctx_get() previously? Can you please shed some light on this
>
> Regards,
> Avra
>
>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Report ESTALE as ENOENT

[FNU Raghavendra Manjunath]

I can understand the concern. But I think instead of generally converting
all the ESTALE errors ENOENT, probably we should try to analyze the errors
that are generated by lower layers (like posix).

Even fuse kernel module some times returns ESTALE. (Well, I can see it
returning ESTALE errors in some cases in the code. Someone please correct
me if thats not the case).  And aso I am not sure if converting all the
ESTALE errors to ENOENT is ok or not.

For fd based operations, I am not sure if ENOENT can be sent or not (as
though the file is unlinked, it can be accessed if there were open fds on
it before unlinking the file).

I feel, we have to look into some parts to check if they generating ESTALE
is a proper error or not. Also, if there is any bug in below layers fixing
which can avoid ESTALE errors, then I feel that would be the better option.

My 2 cents.

Regards,
Raghavendra


On Mon, Mar 28, 2016 at 1:39 AM, Prashanth Pai <p...@redhat.com> wrote:

> TL;DR: +1 to report ESTALE as ENOENT
>
> While ESTALE is an acceptable errno for NFS clients, it's not so much for
> FUSE clients. Many applications that talk to a FUSE mount do not handle
> ESTALE and expect the behavior to be analogous to that of local
> filesystems such as XFS. While it's okay for brick to send ESTALE to
> glusterfs client stack, one has to be very careful about errno returned by
> FUSE back to applications.
>
> For example, syscalls such as fgetxattr are not expected (at least from
> manpage) to throw ESTALE but with glusterfs, it does[1]. Further, POSIX
> guarantees that once an application has a valid fd, operations like
> fgetxattr() on the fd should succeed even after another application(client)
> issues an unlink()
>
> [1]:http://paste.openstack.org/show/335506/
>
> Regards,
>  -Prashanth Pai
>
> - Original Message -
> > From: "FNU Raghavendra Manjunath" <rab...@redhat.com>
> > To: "Soumya Koduri" <skod...@redhat.com>
> > Cc: "Ira Cooper" <icoo...@redhat.com>, "Gluster Devel" <
> gluster-devel@gluster.org>
> > Sent: Thursday, March 24, 2016 8:11:19 PM
> > Subject: Re: [Gluster-devel] Report ESTALE as ENOENT
> >
> >
> > I would still prefer not converting all the ESTALE to ENOENT. I think we
> need
> > to understand this specific case of parallel rm -rfs getting ESTALE
> errors
> > and handle it accordingly.
> >
> > Regarding, gfapi not honoring the ESTALE errors, I think it would be
> better
> > to do revalidates upon getting ESTALE.
> >
> > Just my 2 cents.
> >
> > Regards,
> > Raghavendra
> >
> >
> > On Thu, Mar 24, 2016 at 10:31 AM, Soumya Koduri < skod...@redhat.com >
> wrote:
> >
> >
> > Thanks for the information.
> >
> > On 03/24/2016 07:34 PM, FNU Raghavendra Manjunath wrote:
> >
> >
> >
> > Yes. I think the caching example mentioned by Shyam is a good example of
> > ESTALE error. Also User Serviceable Snapshots (USS) relies heavily on
> > ESTALE errors. Because the files/directories from the snapshots are
> > assigned a virtual gfid on the fly when being looked up. If those inodes
> > are purged out of the inode table due to lru list becoming full, then a
> > access to that gfid from the client, will make snapview-server send
> > ESTALE and either fuse (I think our fuse xlator does a revalidate upon
> > getting ESTALE) or NFS client can revalidate via path based resolution.
> >
> > So wouldn't it be wrong not to send ESTALE to NFS-clients and map it to
> > ENOENT, as was intended in the original mail.
> >
> > NFSv3 rfc [1] mentions that NFS3ERR_STALE is a valid error for REMOVE
> fop.
> >
> > Also (at least in gfapi) the resolve code path doesn't seem to be
> honoring
> > ESTALE errors - glfs_resolve_component(..), glfs_refresh_inode_safe(..)
> > etc.. Do we need to fix them?
> >
> >
> > Thanks,
> > Soumya
> >
> > [1] https://www.ietf.org/rfc/rfc1813.txt (section# 3.3.12)
> >
> >
> >
> >
> > Regards,
> > Raghavendra
> >
> >
> > On Thu, Mar 24, 2016 at 9:51 AM, Shyam < srang...@redhat.com
> > > wrote:
> >
> > On 03/23/2016 12:07 PM, Ravishankar N wrote:
> >
> > On 03/23/2016 09:16 PM, Soumya Koduri wrote:
> >
> > If it occurs only when the file/dir is not actually present
> > at the
> > back-end, shouldn't we fix the server to send ENOENT then?
> >
> > I never fully understood it here is the answer:
> > http://review.gluster.org/#/c/6318/
> >
> >
> > The intention of ESTALE is to state that the inode#/

Re: [Gluster-devel] Report ESTALE as ENOENT

[FNU Raghavendra Manjunath]

I would still prefer not converting all the ESTALE to ENOENT. I think we
need to understand this specific case of parallel rm -rfs getting ESTALE
errors and handle it accordingly.

Regarding, gfapi not honoring the ESTALE errors, I think it would be better
to do revalidates upon getting ESTALE.

Just my 2 cents.

Regards,
Raghavendra


On Thu, Mar 24, 2016 at 10:31 AM, Soumya Koduri <skod...@redhat.com> wrote:

> Thanks for the information.
>
> On 03/24/2016 07:34 PM, FNU Raghavendra Manjunath wrote:
>
>>
>> Yes. I think the caching example mentioned by Shyam is a good example of
>> ESTALE error. Also User Serviceable Snapshots (USS) relies heavily on
>> ESTALE errors. Because the files/directories from the snapshots are
>> assigned a virtual gfid on the fly when being looked up. If those inodes
>> are purged out of the inode table due to lru list becoming full, then a
>> access to that gfid from the client, will make snapview-server send
>> ESTALE and either fuse (I think our fuse xlator does a revalidate upon
>> getting ESTALE) or NFS client can revalidate via path based resolution.
>>
>
> So wouldn't it be wrong not to send ESTALE to NFS-clients and map it to
> ENOENT, as was intended in the original mail.
>
> NFSv3 rfc [1] mentions that NFS3ERR_STALE is a valid error for REMOVE fop.
>
> Also (at least in gfapi) the resolve code path doesn't seem to be honoring
> ESTALE errors - glfs_resolve_component(..), glfs_refresh_inode_safe(..)
> etc.. Do we need to fix them?
>
>
> Thanks,
> Soumya
>
> [1] https://www.ietf.org/rfc/rfc1813.txt (section# 3.3.12)
>
>
>> Regards,
>> Raghavendra
>>
>>
>> On Thu, Mar 24, 2016 at 9:51 AM, Shyam <srang...@redhat.com
>> <mailto:srang...@redhat.com>> wrote:
>>
>> On 03/23/2016 12:07 PM, Ravishankar N wrote:
>>
>> On 03/23/2016 09:16 PM, Soumya Koduri wrote:
>>
>> If it occurs only when the file/dir is not actually present
>> at the
>> back-end, shouldn't we fix the server to send ENOENT then?
>>
>> I never fully understood it here is the answer:
>> http://review.gluster.org/#/c/6318/
>>
>>
>> The intention of ESTALE is to state that the inode#/GFID is stale,
>> when using that for any operations. IOW, we did not find the GFID in
>> the backend, that does not mean the name is not present. This hence
>> means, if you have a pGFID+bname, try resolving with that.
>>
>> For example, a client side cache can hang onto a GFID for a bname,
>> but another client could have, in the interim, unlinked the bname
>> and create a new file there.
>>
>> A presence test using GFID by the client that cached the result the
>> first time, is an ESTALE. But a resolution based on pGFID+bname
>> again by the same client would be a success.
>>
>> By extension, a GFID based resolution, when not really present in
>> the backend will return ESTALE, it could very well mean ENOENT, but
>> that has to be determined by the client again, if possible.
>>
>> See "A10. What does it mean when my application fails because of an
>> ESTALE error?" for NFS here [1] and [2] (there maybe better sources
>> for this information)
>>
>> [1] http://nfs.sourceforge.net/
>> [2] https://lwn.net/Articles/272684/
>>
>>
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org <mailto:Gluster-devel@gluster.org>
>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org <mailto:Gluster-devel@gluster.org>
>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>
>>
>>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Regression: Bitrot core generated by distribute/bug-1117851.t

[FNU Raghavendra Manjunath]

Hi,

I have raised a bug for it (
https://bugzilla.redhat.com/show_bug.cgi?id=1315465).

A patch has been sent for review (http://review.gluster.org/#/c/13628/).

Regards,
Raghavendra


On Mon, Mar 7, 2016 at 11:04 AM, Poornima Gurusiddaiah 
wrote:

> Hi,
>
> I see a bitrot crash caused by a dht test case
> ./tests/bugs/distribute/bug-1117851.t
> Regression link:
> https://build.gluster.org/job/rackspace-regression-2GB-triggered/18858/console
>
> Backtrace:
>
> /build/install/lib/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xf2)[0x7f8d873d93f6]
> /build/install/lib/libglusterfs.so.0(gf_print_trace+0x22b)[0x7f8d873df381]
> /build/install/sbin/glusterfsd(glusterfsd_print_trace+0x1f)[0x409962]
> /lib64/libc.so.6(+0x326a0)[0x7f8d85f6b6a0]
>
> /build/install/lib/glusterfs/3.8dev/xlator/features/bitrot-stub.so(+0x6829)[0x7f8d79409829]
>
> /build/install/lib/glusterfs/3.8dev/xlator/features/bitrot-stub.so(+0x11751)[0x7f8d79414751]
>
> /build/install/lib/glusterfs/3.8dev/xlator/features/changelog.so(+0x505c)[0x7f8d7961f05c]
>
> /build/install/lib/glusterfs/3.8dev/xlator/features/changetimerecorder.so(+0xa8aa)[0x7f8d79ceb8aa]
>
> /build/install/lib/glusterfs/3.8dev/xlator/features/trash.so(+0xa544)[0x7f8d79f07544]
>
> /build/install/lib/glusterfs/3.8dev/xlator/storage/posix.so(+0x10650)[0x7f8d7a737650]
>
> /build/install/lib/glusterfs/3.8dev/xlator/features/trash.so(+0xd489)[0x7f8d79f0a489]
>
> /build/install/lib/glusterfs/3.8dev/xlator/features/changetimerecorder.so(+0xb179)[0x7f8d79cec179]
>
> /build/install/lib/glusterfs/3.8dev/xlator/features/changelog.so(+0x5d1c)[0x7f8d7961fd1c]
>
> /build/install/lib/glusterfs/3.8dev/xlator/features/bitrot-stub.so(+0x11a72)[0x7f8d79414a72]
>
> /build/install/lib/glusterfs/3.8dev/xlator/features/access-control.so(+0x7d3f)[0x7f8d791f9d3f]
> /build/install/lib/libglusterfs.so.0(default_unlink+0xa8)[0x7f8d8746d114]
>
> /build/install/lib/glusterfs/3.8dev/xlator/features/upcall.so(+0x4eab)[0x7f8d78dc4eab]
>
> /build/install/lib/libglusterfs.so.0(default_unlink_resume+0x1dd)[0x7f8d87469ef7]
>
> /build/install/lib/libglusterfs.so.0(call_resume_wind+0x321)[0x7f8d873f6c4c]
> /build/install/lib/libglusterfs.so.0(call_resume+0xd2)[0x7f8d873ffa1a]
>
> /build/install/lib/glusterfs/3.8dev/xlator/performance/io-threads.so(+0x4727)[0x7f8d78bb9727]
> /lib64/libpthread.so.0(+0x7aa1)[0x7f8d866b8aa1]
> /lib64/libc.so.6(clone+0x6d)[0x7f8d8602193d]
>
> Can you please look into it. Do i need to raise a BZ for this?
>
>
> Regards,
> Poornima
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] glusterfs-3.6.9 released

[FNU Raghavendra Manjunath]

Hi,
glusterfs-3.6.9 has been released and the packages for RHEL/Fedora/Centos
can be found here.http://download.gluster.org/pub/gluster/glusterfs/3.6/LATEST/

Requesting people running 3.6.x to please try it out and let us know if
there are any issues.

This release supposedly fixes the bugs listed below since 3.6.8 was made
available. Thanks to all who submitted patches, reviewed the changes.

1302541 - Problem when enabling quota : Could not start quota auxiliary mount
1302310 - log improvements:- enabling quota on a volume reports
numerous entries of "contribution node
list is empty which is an error" in brick logs

1308806 - tests : Modifying tests for crypt xlator

1304668 - Add missing release-notes on the 3.6 branch
1296931 - Installation of glusterfs-3.6.8 fails on CentOS-7


Regards,

Raghavendra Bhat
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Need help with bitrot

[FNU Raghavendra Manjunath]

As of now, signing happens only upon data modification. Metadata changes
and xattr changes does not trigger signing.

For more information about gluster and its internals, you can look here "
https://gluster.readthedocs.org/en/latest/;.

Regards,
Raghavendra

On Thu, Feb 25, 2016 at 10:28 AM, Ajil Abraham <ajil95.abra...@gmail.com>
wrote:

> Thanks FNU Raghavendra.  Does the signing happen only when the file data
> changes or even when extended attribute changes?
>
> I am also trying to understand the Gluster internal data structures. Are
> there any materials for the same? Similarly for the translators, the way
> they are stacked on client & server side, how control flows between them.
> Can somebody please help?
>
> - Ajil
>
>
> On Thu, Feb 25, 2016 at 7:27 AM, FNU Raghavendra Manjunath <
> rab...@redhat.com> wrote:
>
>> Hi Ajil,
>>
>> Expiry policy tells the signer (Bit-rot Daemon) to wait for a specific
>> period of time before signing a object.
>>
>> Whenever a object is modified, a notification is sent to the signer by
>> brick process (bit-rot-stub xlator sitting in the I/O path) upon getting a
>> release (i.e. when all the fds of that object are closed). The expiry
>> policy tells the signer to wait for some time (by default its 120 seconds)
>> before signing that object. It is done because, suppose the signer starts
>> signing (i.e. read the object + calculate the checksum + store the
>> checksum) a object the object gets modified again, then a new notification
>> has to be sent and again signer has to sign the object by calculating the
>> checksum. Whereas if the signer waits for some time and receives a new
>> notification on the same object when its waiting, then it can avoid signing
>> for the first notification.
>>
>> Venky, do you want to add anything more?
>>
>> Regards,
>> Raghavendra
>>
>>
>>
>> On Wed, Feb 24, 2016 at 12:28 AM, Ajil Abraham <ajil95.abra...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am a student interested in GlusterFS.  Trying to understand the design
>>> of GlusterFS. Came across the Bitrot design document in Google. There is a
>>> mention of expiry policy used to sign the files. I did not clearly
>>> understand what the expiry policy is.  Can somebody please help?
>>>
>>> -Ajil
>>>
>>> ___
>>> Gluster-devel mailing list
>>> Gluster-devel@gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>
>>
>>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] glusterfs-3.6.9 release plans

[FNU Raghavendra Manjunath]

Hi, Kaushal,

I have been trying to merge few patches. But every time I try (i.e. do a
cherry pick in gerrit), a new patch set gets submitted. I need some help in
resolving it.

Regards,
Raghavendra


On Wed, Feb 17, 2016 at 8:31 AM, Kaushal M  wrote:

> Hey Johnny,
>
> Could you please provide an update on the 3.6.9 release plans?
>
> The GlusterFS release schedule has 3.6 releases happening every month
> on the week of the 20th. We're less than a week away to the release as
> per the schedule, so we'd like to know if it is still on track.
>
> Thanks.
>
> Kaushal
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Bitrot stub forget()

[FNU Raghavendra Manjunath]

Venky,

Yes. You are right. We should not remove the quarantine entry in forget.

We have to remove it upon getting -ve lookups in bit-rot-stub and upon
getting an unlink.

I have attached a patch for it.

Unfortunately rfc.sh is failing for me with the below error.


ssh: connect to host git.gluster.com port 22: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists."


Regards,
Raghavendra


On Tue, Feb 16, 2016 at 10:53 AM, Venky Shankar  wrote:

> Hey Raghu,
>
> Bitrot stub inode forget implementation (br_stub_forget()) deletes the bad
> object
> marker (under quarantine directory) if present. This looks incorrect as
> ->forget()
> can be trigerred when inode table LRU size exceeeds configured limit -
> check bug
> #1308961 which tracks this issue. I recall that protocol/server calls
> inode_forget()
> on negative lookup (that might not invoke ->forget() though) and that's
> the reason
> why br_stub_forget() has this code.
>
> So, would it make sense to purge bad object marker just in lookup()? There
> might be
> a need to do the same in unlink() in case the object was removed by the
> client.
>
> Thoughts?
>
> Thanks,
>
> Venky
>
From a0cc49172df24e263e0db25c53b57f58c19d2cab Mon Sep 17 00:00:00 2001
From: Raghavendra Bhat 
Date: Tue, 16 Feb 2016 20:22:36 -0500
Subject: [PATCH] features/bitrot: do not remove the quarantine handle in
 forget

If an object is marked as bad, then an entry is corresponding to the
bad object is created in the .glusterfs/quarantine directory to help
scrub status. The entry name is the gfid of the corrupted object.
The quarantine handle is removed in below 2 cases.

1) When protocol/server revceives the -ve lookup on an entry whose inode
   is there in the inode table (it can happen when the corrupted object
   is deleted directly from the backend for recovery purpose) it sends a
   forget on the inode and bit-rot-stub removes the quarantine handle in
   upon getting the forget.
   refer to the below commit
   f853ed9c61bf65cb39f859470a8ffe8973818868:
   http://review.gluster.org/12743)

2) When bit-rot-stub itself realizes that lookup on a corrupted object
   has failed with ENOENT.

But with step1, there is a problem when the bit-rot-stub receives forget
due to lru limit exceeding in the inode table. In such cases, though the
corrupted object is not deleted (either from the mount point or from the
backend), the handle in the quarantine directory is removed and that object
is not shown in the bad objects list in the scrub status command.

So it is better to follow only 2nd step (i.e. bit-rot-stub removing the handle
from the quarantine directory in -ve lookups). Also the handle has to be removed
when a corrupted object is unlinked from the mount point itself.

Change-Id: Ibc3bbaf4bc8a5f8986085e87b729ab912cbf8cf9
Signed-off-by: Raghavendra Bhat 
---
 xlators/features/bit-rot/src/stub/bit-rot-stub.c | 103 +--
 1 file changed, 95 insertions(+), 8 deletions(-)

diff --git a/xlators/features/bit-rot/src/stub/bit-rot-stub.c b/xlators/features/bit-rot/src/stub/bit-rot-stub.c
index 9c0f622..920aa7c 100644
--- a/xlators/features/bit-rot/src/stub/bit-rot-stub.c
+++ b/xlators/features/bit-rot/src/stub/bit-rot-stub.c
@@ -2841,19 +2841,46 @@ unwind:
 
 /** {{{ */
 
-/* forget() */
+/* unlink() */
 
 int
-br_stub_forget (xlator_t *this, inode_t *inode)
+br_stub_unlink_cbk (call_frame_t *frame, void *cookie, xlator_t *this,
+   int32_t op_ret, int32_t op_errno, struct iatt *preparent,
+   struct iatt *postparent, dict_t *xdata)
 {
-uint64_t ctx_addr = 0;
-br_stub_inode_ctx_t *ctx = NULL;
+br_stub_local_t *local= NULL;
+inode_t *inode= NULL;
+uint64_t ctx_addr = 0;
+br_stub_inode_ctx_t *ctx  = NULL;
+int32_t  ret  = -1;
 
-inode_ctx_del (inode, this, _addr);
-if (!ctx_addr)
-return 0;
+if (op_ret < 0)
+goto unwind;
 
-ctx = (br_stub_inode_ctx_t *) (long) ctx_addr;
+local = frame->local;
+frame->local = NULL;
+
+inode = local->u.context.inode;
+
+ret = br_stub_get_inode_ctx (this, inode, _addr);
+if (ret) {
+/**
+ * If the inode is bad AND context is not there, then there
+ * is a possibility of the gfid of the object being listed
+ * in the quarantine directory and will be shown in the
+ * bad objects list. So continuing with the fop with a
+ * warning log. The entry from the quarantine directory
+ * has to be removed manually. Its not a good idea to fail
+ * the fop, as the object has already been deleted.
+ */