date:20190220

Re: [Gluster-devel] GlusterFs v4.1.5: Need help on bitrot detection

2019-02-20 Thread Nithya Balachandran

On Wed, 20 Feb 2019 at 21:03, Amar Tumballi Suryanarayan <
atumb...@redhat.com> wrote:

> Hi Chandranana,
>
> We are trying to find a BigEndian platform to test this out at the moment,
> will get back to you on this.
>
> Meantime, did you run the entire regression suit? Is it the only test
> failing? To run the entire regression suite, please run `run-tests.sh -c`
> from glusterfs source repo.
>
>
They are seeing other issues as well [1] , mostly related to the hashed
values in Big endian systems and the hardcoded names and paths in the .t
files.  I have fixed 2 .t files and asked them to debug the remaining tests
and provide patches as it was taking a long time to go back and forth with
various suggested changes.

There are debug logs attached for all the failing tests, including one for
the failing bitrot case which indicates a very large value being returned
in fgetxattr (probably also related to endianess).

[2019-02-14 09:12:05.140750] D [MSGID: 0] [io-threads.c:372:iot_schedule]
0-patchy-io-threads: FGETXATTR scheduled as least priority fop
[2019-02-14 09:12:05.140828] A [MSGID: 0] [mem-pool.c:118:__gf_calloc] : no
memory available for size (176093659239) [call stack follows]
/usr/local/lib/libglusterfs.so.0(+0x28eaa)[0x3ffb2da8eaa]

/usr/local/lib/libglusterfs.so.0(_gf_msg_nomem+0x31c)[0x3ffb2da93c4]

/usr/local/lib/libglusterfs.so.0(__gf_calloc+0x13c)[0x3ffb2dd595c]

/usr/local/lib/glusterfs/4.1.5/xlator/features/bitrot-stub.so(+0xe3c4)[0x3ffae28e3c4]
/usr/local/lib/glusterfs/4.1.5/xlator/storage/posix.so(+0x32154)[0x3ffae7b2154]


Regards,
Nithya
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1672480



> -Amar
>
> On Tue, Feb 19, 2019 at 1:31 AM Chandranana Naik 
> wrote:
>
>> Hi Team,
>>
>> We are working with Glusterfs v4.1.5 on big endian platform(Ubuntu 16.04)
>> and encountered that the subtest 20 of test
>> ./tests/bitrot/bug-1207627-bitrot-scrub-status.t is failing.
>>
>> Subtest 20 is failing as below:
>> *trusted.bit-rot.bad-file check_for_xattr trusted.bit-rot.bad-file
>> //d/backends/patchy1/FILE1*
>> *not ok 20 Got "" instead of "trusted.bit-rot.bad-file", LINENUM:50*
>> *FAILED COMMAND: trusted.bit-rot.bad-file check_for_xattr
>> trusted.bit-rot.bad-file //d/backends/patchy1/FILE1*
>>
>> The test is failing with error "*remote operation failed [Cannot
>> allocate memory]"* logged in /var/log/glusterfs/scrub.log.
>> Could you please let us know if anything is missing in making this test
>> pass, PFA the logs for the test case
>>
>> *(See attached file: bug-1207627-bitrot-scrub-status.7z)*
>>
>> Note: *Enough memory is available on the system*.
>>
>> Regards,
>> Chandranana Naik
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
>
> --
> Amar Tumballi (amarts)
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] GlusterFs v4.1.5: Need help on bitrot detection

2019-02-20 Thread Amar Tumballi Suryanarayan

Hi Chandranana,

We are trying to find a BigEndian platform to test this out at the moment,
will get back to you on this.

Meantime, did you run the entire regression suit? Is it the only test
failing? To run the entire regression suite, please run `run-tests.sh -c`
from glusterfs source repo.

-Amar

On Tue, Feb 19, 2019 at 1:31 AM Chandranana Naik 
wrote:

> Hi Team,
>
> We are working with Glusterfs v4.1.5 on big endian platform(Ubuntu 16.04)
> and encountered that the subtest 20 of test
> ./tests/bitrot/bug-1207627-bitrot-scrub-status.t is failing.
>
> Subtest 20 is failing as below:
> *trusted.bit-rot.bad-file check_for_xattr trusted.bit-rot.bad-file
> //d/backends/patchy1/FILE1*
> *not ok 20 Got "" instead of "trusted.bit-rot.bad-file", LINENUM:50*
> *FAILED COMMAND: trusted.bit-rot.bad-file check_for_xattr
> trusted.bit-rot.bad-file //d/backends/patchy1/FILE1*
>
> The test is failing with error "*remote operation failed [Cannot allocate
> memory]"* logged in /var/log/glusterfs/scrub.log.
> Could you please let us know if anything is missing in making this test
> pass, PFA the logs for the test case
>
> *(See attached file: bug-1207627-bitrot-scrub-status.7z)*
>
> Note: *Enough memory is available on the system*.
>
> Regards,
> Chandranana Naik
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel



-- 
Amar Tumballi (amarts)
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Release 6: Branched and next steps

2019-02-20 Thread Shyam Ranganathan

On 2/20/19 7:45 AM, Amar Tumballi Suryanarayan wrote:
> 
> 
> On Tue, Feb 19, 2019 at 1:37 AM Shyam Ranganathan  > wrote:
> 
> In preparation for RC0 I have put up an intial patch for the release
> notes [1]. Request the following actions on the same (either a followup
> patchset, or a dependent one),
> 
> - Please review!
> - Required GD2 section updated to latest GD2 status
> 
> 
> I am inclined to drop the GD2 section for 'standalone' users. As the
> team worked with goals of making GD2 invisible with containers (GCS) in
> mind. So, should we call out any features of GD2 at all?

This is fine, we possibly need to add a note in the release notes, on
the GD2 future and where it would land, so that we can inform users
about the continued use of GD1 in non-GCS use cases.

I will add some text around the same in the release-notes.

> 
> Anyways, as per my previous email on GCS release updates, we are
> planning to have a container available with gd2 and glusterfs, which can
> be used by people who are trying out options with GD2.
>  
> 
> - Require notes on "Reduce the number or threads used in the brick
> process" and the actual status of the same in the notes
> 
> 
> This work is still in progress, and we are treating it as a bug fix for
> 'brick-multiplex' usecase, which is mainly required in scaled volume
> number usecase in container world. My guess is, we won't have much
> content to add for glusterfs-6.0 at the moment.

Ack!

>  
> 
> RC0 build target would be tomorrow or by Wednesday.
> 
> 
> Thanks, I was testing for few upgrade and different version clusters
> support. With 4.1.6 and latest release-6.0 branch, things works fine. I
> haven't done much of a load testing yet. 

Awesome! Helps write out the upgrade guide as well. As this time content
there would/should carry data regarding how to upgrade if any of the
deprecated xlators are in use by a deployment.

> 
> Requesting people to support in upgrade testing. From different volume
> options, and different usecase scenarios.
> 
> Regards,
> Amar
> 
>  
> 
> Thanks,
> Shyam
> 
> [1] Release notes patch: https://review.gluster.org/c/glusterfs/+/6
> 
> On 2/5/19 8:25 PM, Shyam Ranganathan wrote:
> > Hi,
> >
> > Release 6 is branched, and tracker bug for 6.0 is created [1].
> >
> > Do mark blockers for the release against [1].
> >
> > As of now we are only tracking [2] "core: implement a global
> thread pool
> > " for a backport as a feature into the release.
> >
> > We expect to create RC0 tag and builds for upgrade and other testing
> > close to mid-week next week (around 13th Feb), and the release is
> slated
> > for the first week of March for GA.
> >
> > I will post updates to this thread around release notes and other
> > related activity.
> >
> > Thanks,
> > Shyam
> >
> > [1] Tracker: https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-6.0
> >
> > [2] Patches tracked for a backport:
> >   - https://review.gluster.org/c/glusterfs/+/20636
> > ___
> > Gluster-devel mailing list
> > Gluster-devel@gluster.org 
> > https://lists.gluster.org/mailman/listinfo/gluster-devel
> >
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org 
> https://lists.gluster.org/mailman/listinfo/gluster-devel
> 
> 
> 
> 
> -- 
> Amar Tumballi (amarts)
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] md-cache: May bug found in md-cache.c

2019-02-20 Thread Amar Tumballi Suryanarayan

Hi David,

https://docs.gluster.org/en/latest/Developer-guide/Backport-Guidelines/
gives more details about it.

But easiest is to go to your patch (https://review.gluster.org/22234), and
then click on 'Cherry Pick' button. In the pop-up, 'branch:' field, give
'release-6' and Submit. If you want it in release-5 branch too, repeat the
same, with branch being 'release-5'. Siimlarly we need 'clone-of' bug for
both the branches (the original bug used in patch is for master branch).

That should be it. Rest, we can take care.

Thanks a lot!

Regards,
Amar

On Wed, Feb 20, 2019 at 6:58 PM David Spisla 
wrote:

> Hello Amar,
>
>
>
> no problem. How can I do that? Can you please tell me the procedure?
>
>
>
> Regards
>
> David
>
>
>
> *Von:* Amar Tumballi Suryanarayan 
> *Gesendet:* Mittwoch, 20. Februar 2019 14:18
> *An:* David Spisla 
> *Cc:* Gluster Devel 
> *Betreff:* Re: [Gluster-devel] md-cache: May bug found in md-cache.c
>
>
>
> Hi David,
>
>
>
> Thanks for the patch, it got merged in master now. Can you please post it
> into release branches, so we can take them in release-6, release-5 branch,
> so next releases can have them.
>
>
>
> Regards,
>
> Amar
>
>
>
> On Tue, Feb 19, 2019 at 8:49 PM David Spisla  wrote:
>
> Hello,
>
>
>
> I already open a bug:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1678726
>
>
>
> There is also a link to a bug fix patch
>
>
>
> Regards
>
> David Spisla
>
>
>
> Am Di., 19. Feb. 2019 um 13:07 Uhr schrieb David Spisla <
> spisl...@gmail.com>:
>
> Hi folks,
>
>
>
> The 'struct md_cache' in md-cache.c uses int data types which are not in
> common with the data types used in the 'struct iatt' in iatt.h . If one
> take a closer look to the implementations one can see that the struct in
> md-cache.c uses still the int data types like in the struct 'old_iatt' .
> This can lead to unexpected side effects and some values of iatt maybe will
> not mapped correctly. I would suggest to open a bug report. What do you
> think?
>
> Additional info:
>
> struct md_cache {
> ia_prot_t md_prot;
> uint32_t md_nlink;
> uint32_t md_uid;
> uint32_t md_gid;
> uint32_t md_atime;
> uint32_t md_atime_nsec;
> uint32_t md_mtime;
> uint32_t md_mtime_nsec;
> uint32_t md_ctime;
> uint32_t md_ctime_nsec;
> uint64_t md_rdev;
> uint64_t md_size;
> uint64_t md_blocks;
> uint64_t invalidation_time;
> uint64_t generation;
> dict_t *xattr;
> char *linkname;
> time_t ia_time;
> time_t xa_time;
> gf_boolean_t need_lookup;
> gf_boolean_t valid;
> gf_boolean_t gen_rollover;
> gf_boolean_t invalidation_rollover;
> gf_lock_t lock;
> };
>
> struct iatt {
> uint64_t ia_flags;
> uint64_t ia_ino; /* inode number */
> uint64_t ia_dev; /* backing device ID */
> uint64_t ia_rdev;/* device ID (if special file) */
> uint64_t ia_size;/* file size in bytes */
> uint32_t ia_nlink;   /* Link count */
> uint32_t ia_uid; /* user ID of owner */
> uint32_t ia_gid; /* group ID of owner */
> uint32_t ia_blksize; /* blocksize for filesystem I/O */
> uint64_t ia_blocks;  /* number of 512B blocks allocated */
> int64_t ia_atime;/* last access time */
> int64_t ia_mtime;/* last modification time */
> int64_t ia_ctime;/* last status change time */
> int64_t ia_btime;/* creation time. Fill using statx */
> uint32_t ia_atime_nsec;
> uint32_t ia_mtime_nsec;
> uint32_t ia_ctime_nsec;
> uint32_t ia_btime_nsec;
> uint64_t ia_attributes;  /* chattr related:compressed, immutable,
>   * append only, encrypted etc.*/
> uint64_t ia_attributes_mask; /* Mask for the attributes */
>
> uuid_t ia_gfid;
> ia_type_t ia_type; /* type of file */
> ia_prot_t ia_prot; /* protection */
> };
>
> struct old_iatt {
> uint64_t ia_ino; /* inode number */
> uuid_t ia_gfid;
> uint64_t ia_dev; /* backing device ID */
> ia_type_t ia_type;   /* type of file */
> ia_prot_t ia_prot;   /* protection */
> uint32_t ia_nlink;   /* Link count */
> uint32_t ia_uid; /* user ID of owner */
> uint32_t ia_gid; /* group ID of owner */
> uint64_t ia_rdev;/* device ID (if special file) */
> uint64_t ia_size;/* file size in bytes */
> uint32_t ia_blksize; /* blocksize for filesystem I/O */
> uint64_t ia_blocks;  /* number of 512B blocks allocated */
> uint32_t ia_atime;   /* last access time */
> uint32_t ia_atime_nsec;
> uint32_t ia_mtime; /* last modification time */
> uint32_t ia_mtime_nsec;
> uint32_t ia_ctime; /* last status change time */
> uint32_t ia_ctime_nsec;
> };
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
>
>
> --
>
> Amar Tumballi (amarts)
>


-- 
Amar Tumballi (amarts)
___

Re: [Gluster-devel] md-cache: May bug found in md-cache.c

2019-02-20 Thread Amar Tumballi Suryanarayan

Hi David,

Thanks for the patch, it got merged in master now. Can you please post it
into release branches, so we can take them in release-6, release-5 branch,
so next releases can have them.

Regards,
Amar

On Tue, Feb 19, 2019 at 8:49 PM David Spisla  wrote:

> Hello,
>
> I already open a bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=1678726
>
> There is also a link to a bug fix patch
>
> Regards
> David Spisla
>
> Am Di., 19. Feb. 2019 um 13:07 Uhr schrieb David Spisla <
> spisl...@gmail.com>:
>
>> Hi folks,
>>
>> The 'struct md_cache' in md-cache.c uses int data types which are not in
>> common with the data types used in the 'struct iatt' in iatt.h . If one
>> take a closer look to the implementations one can see that the struct in
>> md-cache.c uses still the int data types like in the struct 'old_iatt' .
>> This can lead to unexpected side effects and some values of iatt maybe will
>> not mapped correctly. I would suggest to open a bug report. What do you
>> think?
>>
>> Additional info:
>>
>> struct md_cache {
>> ia_prot_t md_prot;
>> uint32_t md_nlink;
>> uint32_t md_uid;
>> uint32_t md_gid;
>> uint32_t md_atime;
>> uint32_t md_atime_nsec;
>> uint32_t md_mtime;
>> uint32_t md_mtime_nsec;
>> uint32_t md_ctime;
>> uint32_t md_ctime_nsec;
>> uint64_t md_rdev;
>> uint64_t md_size;
>> uint64_t md_blocks;
>> uint64_t invalidation_time;
>> uint64_t generation;
>> dict_t *xattr;
>> char *linkname;
>> time_t ia_time;
>> time_t xa_time;
>> gf_boolean_t need_lookup;
>> gf_boolean_t valid;
>> gf_boolean_t gen_rollover;
>> gf_boolean_t invalidation_rollover;
>> gf_lock_t lock;
>> };
>>
>> struct iatt {
>> uint64_t ia_flags;
>> uint64_t ia_ino; /* inode number */
>> uint64_t ia_dev; /* backing device ID */
>> uint64_t ia_rdev;/* device ID (if special file) */
>> uint64_t ia_size;/* file size in bytes */
>> uint32_t ia_nlink;   /* Link count */
>> uint32_t ia_uid; /* user ID of owner */
>> uint32_t ia_gid; /* group ID of owner */
>> uint32_t ia_blksize; /* blocksize for filesystem I/O */
>> uint64_t ia_blocks;  /* number of 512B blocks allocated */
>> int64_t ia_atime;/* last access time */
>> int64_t ia_mtime;/* last modification time */
>> int64_t ia_ctime;/* last status change time */
>> int64_t ia_btime;/* creation time. Fill using statx */
>> uint32_t ia_atime_nsec;
>> uint32_t ia_mtime_nsec;
>> uint32_t ia_ctime_nsec;
>> uint32_t ia_btime_nsec;
>> uint64_t ia_attributes;  /* chattr related:compressed, immutable,
>>   * append only, encrypted etc.*/
>> uint64_t ia_attributes_mask; /* Mask for the attributes */
>>
>> uuid_t ia_gfid;
>> ia_type_t ia_type; /* type of file */
>> ia_prot_t ia_prot; /* protection */
>> };
>>
>> struct old_iatt {
>> uint64_t ia_ino; /* inode number */
>> uuid_t ia_gfid;
>> uint64_t ia_dev; /* backing device ID */
>> ia_type_t ia_type;   /* type of file */
>> ia_prot_t ia_prot;   /* protection */
>> uint32_t ia_nlink;   /* Link count */
>> uint32_t ia_uid; /* user ID of owner */
>> uint32_t ia_gid; /* group ID of owner */
>> uint64_t ia_rdev;/* device ID (if special file) */
>> uint64_t ia_size;/* file size in bytes */
>> uint32_t ia_blksize; /* blocksize for filesystem I/O */
>> uint64_t ia_blocks;  /* number of 512B blocks allocated */
>> uint32_t ia_atime;   /* last access time */
>> uint32_t ia_atime_nsec;
>> uint32_t ia_mtime; /* last modification time */
>> uint32_t ia_mtime_nsec;
>> uint32_t ia_ctime; /* last status change time */
>> uint32_t ia_ctime_nsec;
>> };
>>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel



-- 
Amar Tumballi (amarts)
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Release 6: Branched and next steps

2019-02-20 Thread Amar Tumballi Suryanarayan

On Tue, Feb 19, 2019 at 1:37 AM Shyam Ranganathan 
wrote:

> In preparation for RC0 I have put up an intial patch for the release
> notes [1]. Request the following actions on the same (either a followup
> patchset, or a dependent one),
>
> - Please review!
> - Required GD2 section updated to latest GD2 status
>

I am inclined to drop the GD2 section for 'standalone' users. As the team
worked with goals of making GD2 invisible with containers (GCS) in mind.
So, should we call out any features of GD2 at all?

Anyways, as per my previous email on GCS release updates, we are planning
to have a container available with gd2 and glusterfs, which can be used by
people who are trying out options with GD2.


> - Require notes on "Reduce the number or threads used in the brick
> process" and the actual status of the same in the notes
>
>
This work is still in progress, and we are treating it as a bug fix for
'brick-multiplex' usecase, which is mainly required in scaled volume number
usecase in container world. My guess is, we won't have much content to add
for glusterfs-6.0 at the moment.


> RC0 build target would be tomorrow or by Wednesday.
>
>
Thanks, I was testing for few upgrade and different version clusters
support. With 4.1.6 and latest release-6.0 branch, things works fine. I
haven't done much of a load testing yet.

Requesting people to support in upgrade testing. From different volume
options, and different usecase scenarios.

Regards,
Amar



> Thanks,
> Shyam
>
> [1] Release notes patch: https://review.gluster.org/c/glusterfs/+/6
>
> On 2/5/19 8:25 PM, Shyam Ranganathan wrote:
> > Hi,
> >
> > Release 6 is branched, and tracker bug for 6.0 is created [1].
> >
> > Do mark blockers for the release against [1].
> >
> > As of now we are only tracking [2] "core: implement a global thread pool
> > " for a backport as a feature into the release.
> >
> > We expect to create RC0 tag and builds for upgrade and other testing
> > close to mid-week next week (around 13th Feb), and the release is slated
> > for the first week of March for GA.
> >
> > I will post updates to this thread around release notes and other
> > related activity.
> >
> > Thanks,
> > Shyam
> >
> > [1] Tracker: https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-6.0
> >
> > [2] Patches tracked for a backport:
> >   - https://review.gluster.org/c/glusterfs/+/20636
> > ___
> > Gluster-devel mailing list
> > Gluster-devel@gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-devel
> >
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
>

-- 
Amar Tumballi (amarts)
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] GlusterFs v4.1.5: Need help on bitrot detection

Re: [Gluster-devel] GlusterFs v4.1.5: Need help on bitrot detection

Re: [Gluster-devel] Release 6: Branched and next steps

Re: [Gluster-devel] md-cache: May bug found in md-cache.c

Re: [Gluster-devel] md-cache: May bug found in md-cache.c

Re: [Gluster-devel] Release 6: Branched and next steps

6 matches

Site Navigation

Mail list logo

Footer information