[Gluster-devel] Test-case "./tests/basic/tier/tier-file-create.t" hung

[Vijaikumar Mallikarjuna]

Hi,

The test-case looks hung "./tests/basic/tier/tier-file-create.t".

Can someone look at this issue?

Below are the regression runs which are hung

https://build.gluster.org/job/rackspace-regression-2GB-triggered/19659/console

https://build.gluster.org/job/rackspace-regression-2GB-triggered/19660/console

https://build.gluster.org/job/rackspace-regression-2GB-triggered/19663/console

Thanks,
Vijay
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Testcase broken due to posix iatt commit

[Vijaikumar Mallikarjuna]

Submitted patch http://review.gluster.org/#/c/13837/, to keep the
previous behavior of posix_acl_ctx_get to include for the GS hotfix.

We will fix actual issue of sending lookup on root as a separate bug.

Thanks,
Vijay





On Tue, Mar 29, 2016 at 6:54 AM, Vijaikumar Mallikarjuna <
vmall...@redhat.com> wrote:

> Previously function 'posix_acl_ctx_get' was creating inode_ctx if it is
> not present. So if sometime lookup does not come for root inode, then
> inode-ctx was created here without any perm set.
> Now inode-ctx is created only in lookup_cbk and setattr_cbk to populate
> the ctx with valid perm from statbuf, so I think we need to send lookup
> for root-inode.
>
> Thanks,
> Vijay
>
>
> On Mon, Mar 28, 2016 at 8:40 PM, FNU Raghavendra Manjunath <
> rab...@redhat.com> wrote:
>
>> Root inode gets linked during mount itselsf.
>>
>> We get lookups on root. But since root inode is always present in the
>> inode table, after the mount if there is a fop on an entry whose parent is
>> root, then that fop will be able to find the parent in the inode table as
>> part of resolve. So it need not explicitly send a hard resolution for
>> resolving parent inode.
>>
>> Du? Did I miss anything?
>>
>> Regards,
>> Raghavendra
>> On Mar 28, 2016 11:02 AM, "Vijaikumar Mallikarjuna" 
>> wrote:
>>
>>> Hi Du/Johny,
>>>
>>> Don't we get a lookup on root inode? Please confirm.
>>>
>>> If we don't get lookup on root inode, then we need to create inode-ctx
>>> in a posix_acl_ctx_get() function.
>>>
>>> Thanks,
>>> Vijay
>>> On 28-Mar-2016 7:37 PM, "FNU Raghavendra Manjunath" 
>>> wrote:
>>>
>>>> CCing Vijay Kumar who made the acl related changes in that patch.
>>>>
>>>> Vijay? Can you please look into it?
>>>>
>>>> Regards,
>>>> Raghavendra
>>>>
>>>> On Mon, Mar 28, 2016 at 9:57 AM, Avra Sengupta 
>>>> wrote:
>>>>
>>>>> Hi Raghavendra,
>>>>>
>>>>> As part of the patch (http://review.gluster.org/#/c/13730/16), the
>>>>> inode_ctx is not created in posix_acl_ctx_get(). Because of this the
>>>>> testcase in http://review.gluster.org/#/c/13623/ breaks. It fails
>>>>> with the following logs:
>>>>>
>>>>> [2016-03-28 13:43:39.216168] D [MSGID: 0]
>>>>> [io-threads.c:346:iot_schedule] 0-patchy-io-threads: CREATE scheduled as
>>>>> normal fop
>>>>> [2016-03-28 13:43:39.216495] E [posix-acl.c:199:acl_permits] (-->
>>>>> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x1ba)[0x7f6fea72780a]
>>>>> (-->
>>>>> /usr/local/lib/glusterfs/3.8dev/xlator/features/access-control.so(+0x49c4)[0x7f6fde5499c4]
>>>>> (-->
>>>>> /usr/local/lib/glusterfs/3.8dev/xlator/features/access-control.so(+0xa855)[0x7f6fde54f855]
>>>>> (-->
>>>>> /usr/local/lib/glusterfs/3.8dev/xlator/features/locks.so(+0xd37e)[0x7f6fde33837e]
>>>>> (-->
>>>>> /usr/local/lib/glusterfs/3.8dev/xlator/features/upcall.so(+0x640f)[0x7f6fde12040f]
>>>>> ) 0-patchy-access-control: inode ctx is NULL for
>>>>> ----0001
>>>>> [2016-03-28 13:43:39.216544] I [MSGID: 115071]
>>>>> [server-rpc-fops.c:1612:server_create_cbk] 0-patchy-server: 8: CREATE
>>>>> /file1 (----0001/file1) ==> (Permission 
>>>>> denied)
>>>>> [Permission denied]
>>>>>
>>>>> Is it because we missed the inode_ctx creation that was being done by
>>>>> posix_acl_ctx_get() previously? Can you please shed some light on this
>>>>>
>>>>> Regards,
>>>>> Avra
>>>>>
>>>>>
>>>>>
>>>>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Testcase broken due to posix iatt commit

[Vijaikumar Mallikarjuna]

Previously function 'posix_acl_ctx_get' was creating inode_ctx if it is not
present. So if sometime lookup does not come for root inode, then inode-ctx
was created here without any perm set.
Now inode-ctx is created only in lookup_cbk and setattr_cbk to populate the
ctx with valid perm from statbuf, so I think we need to send lookup for
root-inode.

Thanks,
Vijay


On Mon, Mar 28, 2016 at 8:40 PM, FNU Raghavendra Manjunath <
rab...@redhat.com> wrote:

> Root inode gets linked during mount itselsf.
>
> We get lookups on root. But since root inode is always present in the
> inode table, after the mount if there is a fop on an entry whose parent is
> root, then that fop will be able to find the parent in the inode table as
> part of resolve. So it need not explicitly send a hard resolution for
> resolving parent inode.
>
> Du? Did I miss anything?
>
> Regards,
> Raghavendra
> On Mar 28, 2016 11:02 AM, "Vijaikumar Mallikarjuna" 
> wrote:
>
>> Hi Du/Johny,
>>
>> Don't we get a lookup on root inode? Please confirm.
>>
>> If we don't get lookup on root inode, then we need to create inode-ctx in
>> a posix_acl_ctx_get() function.
>>
>> Thanks,
>> Vijay
>> On 28-Mar-2016 7:37 PM, "FNU Raghavendra Manjunath" 
>> wrote:
>>
>>> CCing Vijay Kumar who made the acl related changes in that patch.
>>>
>>> Vijay? Can you please look into it?
>>>
>>> Regards,
>>> Raghavendra
>>>
>>> On Mon, Mar 28, 2016 at 9:57 AM, Avra Sengupta 
>>> wrote:
>>>
>>>> Hi Raghavendra,
>>>>
>>>> As part of the patch (http://review.gluster.org/#/c/13730/16), the
>>>> inode_ctx is not created in posix_acl_ctx_get(). Because of this the
>>>> testcase in http://review.gluster.org/#/c/13623/ breaks. It fails with
>>>> the following logs:
>>>>
>>>> [2016-03-28 13:43:39.216168] D [MSGID: 0]
>>>> [io-threads.c:346:iot_schedule] 0-patchy-io-threads: CREATE scheduled as
>>>> normal fop
>>>> [2016-03-28 13:43:39.216495] E [posix-acl.c:199:acl_permits] (-->
>>>> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x1ba)[0x7f6fea72780a]
>>>> (-->
>>>> /usr/local/lib/glusterfs/3.8dev/xlator/features/access-control.so(+0x49c4)[0x7f6fde5499c4]
>>>> (-->
>>>> /usr/local/lib/glusterfs/3.8dev/xlator/features/access-control.so(+0xa855)[0x7f6fde54f855]
>>>> (-->
>>>> /usr/local/lib/glusterfs/3.8dev/xlator/features/locks.so(+0xd37e)[0x7f6fde33837e]
>>>> (-->
>>>> /usr/local/lib/glusterfs/3.8dev/xlator/features/upcall.so(+0x640f)[0x7f6fde12040f]
>>>> ) 0-patchy-access-control: inode ctx is NULL for
>>>> ----0001
>>>> [2016-03-28 13:43:39.216544] I [MSGID: 115071]
>>>> [server-rpc-fops.c:1612:server_create_cbk] 0-patchy-server: 8: CREATE
>>>> /file1 (----0001/file1) ==> (Permission denied)
>>>> [Permission denied]
>>>>
>>>> Is it because we missed the inode_ctx creation that was being done by
>>>> posix_acl_ctx_get() previously? Can you please shed some light on this
>>>>
>>>> Regards,
>>>> Avra
>>>>
>>>>
>>>>
>>>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Testcase broken due to posix iatt commit

[Vijaikumar Mallikarjuna]

Hi Du/Johny,

Don't we get a lookup on root inode? Please confirm.

If we don't get lookup on root inode, then we need to create inode-ctx in a
posix_acl_ctx_get() function.

Thanks,
Vijay
On 28-Mar-2016 7:37 PM, "FNU Raghavendra Manjunath" 
wrote:

> CCing Vijay Kumar who made the acl related changes in that patch.
>
> Vijay? Can you please look into it?
>
> Regards,
> Raghavendra
>
> On Mon, Mar 28, 2016 at 9:57 AM, Avra Sengupta 
> wrote:
>
>> Hi Raghavendra,
>>
>> As part of the patch (http://review.gluster.org/#/c/13730/16), the
>> inode_ctx is not created in posix_acl_ctx_get(). Because of this the
>> testcase in http://review.gluster.org/#/c/13623/ breaks. It fails with
>> the following logs:
>>
>> [2016-03-28 13:43:39.216168] D [MSGID: 0] [io-threads.c:346:iot_schedule]
>> 0-patchy-io-threads: CREATE scheduled as normal fop
>> [2016-03-28 13:43:39.216495] E [posix-acl.c:199:acl_permits] (-->
>> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x1ba)[0x7f6fea72780a]
>> (-->
>> /usr/local/lib/glusterfs/3.8dev/xlator/features/access-control.so(+0x49c4)[0x7f6fde5499c4]
>> (-->
>> /usr/local/lib/glusterfs/3.8dev/xlator/features/access-control.so(+0xa855)[0x7f6fde54f855]
>> (-->
>> /usr/local/lib/glusterfs/3.8dev/xlator/features/locks.so(+0xd37e)[0x7f6fde33837e]
>> (-->
>> /usr/local/lib/glusterfs/3.8dev/xlator/features/upcall.so(+0x640f)[0x7f6fde12040f]
>> ) 0-patchy-access-control: inode ctx is NULL for
>> ----0001
>> [2016-03-28 13:43:39.216544] I [MSGID: 115071]
>> [server-rpc-fops.c:1612:server_create_cbk] 0-patchy-server: 8: CREATE
>> /file1 (----0001/file1) ==> (Permission denied)
>> [Permission denied]
>>
>> Is it because we missed the inode_ctx creation that was being done by
>> posix_acl_ctx_get() previously? Can you please shed some light on this
>>
>> Regards,
>> Avra
>>
>>
>>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Regression: ./tests/basic/quota.t fails

[Vijaikumar Mallikarjuna]

Test number 82 is a umount operation on NFS mount, not sure why umount
failed.

EXPECT_WITHIN $UMOUNT_TIMEOUT "Y" force_umount $N0



Thanks,
Vijay



On Wed, Mar 9, 2016 at 12:14 PM, Poornima Gurusiddaiah 
wrote:

> Hi,
>
> I see the below test failing for an unrelated patch:
>
> ./tests/basic/quota.t.
> Failed test : 82
>
> Reqgression: 
> https://build.gluster.org/job/rackspace-netbsd7-regression-triggered/15045/consoleFull
>
> Can you please take a look into it?
>
> Regards,
> Poornima
>
>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Spurious failures in ec/quota.t and distribute/bug-860663.t

[Vijaikumar Mallikarjuna]

Hi Poornima,

Below patch might solve the regression failure for
''./tests/basic/ec/quota.t'

http://review.gluster.org/#/c/13446/
http://review.gluster.org/#/c/13447/

Thanks,
Vijay


On Tue, Mar 1, 2016 at 4:49 PM, Poornima Gurusiddaiah 
wrote:

> Hi,
>
> I see these test cases failing spuriously,
>
> ./tests/basic/ec/quota.t Failed Tests: 1-13, 16, 18, 20, 2
>
> https://build.gluster.org/job/rackspace-regression-2GB-triggered/18637/consoleFull
>
> ./tests/bugs/distribute/bug-860663.t Failed Test: 13
> https://build.gluster.org/job/rackspace-regression-2GB-triggered/18622/consoleFull
>
> Could any one from Quota and dht look into it?
>
> Regards,
> Poornima
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] tests/basic/ec/quota.t failure

[Vijaikumar Mallikarjuna]

We will look into the issue.

Thanks,
Vijay
On Feb 24, 2016 10:39 AM, "Atin Mukherjee"  wrote:

> The above test failed in one of the regression run [1]. Mind having a
> look? I've filed a bug [2] for the same.
>
> [1]
>
> https://build.gluster.org/job/rackspace-regression-2GB-triggered/18454/consoleFull
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1311368
>
> ~Atin
>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Quota list not reflecting disk usage

[Vijaikumar Mallikarjuna]

 getting created for a single entry.
> >
> > We have fixed quite some rename issues and have refactored the marker
> approach. Also
> > as I have mentioned already we have also done Versioning of xattr's
> which solves the
> > issue you are facing in 3.7. It would be really helpful in a production
> environment if
> > you could upgrade to 3.7
> >
> > --
> > Thanks & Regards,
> > Manikandan Selvaganesh.
> >
> > - Original Message -
> > From: "Steve Dainard" 
> > To: "Manikandan Selvaganesh" 
> > Cc: "Vijaikumar Mallikarjuna" , "Gluster Devel" <
> gluster-devel@gluster.org>, "gluster-us...@gluster.org List" <
> gluster-us...@gluster.org>
> > Sent: Thursday, February 11, 2016 1:48:19 AM
> > Subject: Re: [Gluster-users] Quota list not reflecting disk usage
> >
> > So after waiting out the process of disabling quotas, waiting for the
> > xattrs to be cleaned up, re-enabling quotas and waiting for the
> > xattr's to be created, then applying quotas I'm running into the same
> > issue.
> >
> > Yesterday at ~2pm one of the quotas was listed as:
> > /modules|100.0GB|18.3GB|81.7GB
> >
> > I initiated a copy from that glusterfs fuse mount to another fuse
> > mount for a different volume, and now I'm seeing:
> > /modules|100.0GB|27.4GB|72.6GB
> >
> > So an increase of 9GB usage.
> >
> > There were no writes at all to this directory during or after the cp.
> >
> > I did a bit of digging through the /modules directory on one of the
> > gluster nodes and created this spreadsheet:
> >
> https://docs.google.com/spreadsheets/d/1l_6ze68TCOcx6LEh9MFwmqPZ9bM-70CUlSM_8tpQ654/edit?usp=sharing
> >
> > The /modules/R/3.2.2 directory quota value doesn't come close to
> > matching the du value.
> >
> > Funny bit, there are TWO quota contribution attributes:
> > # getfattr -d -m quota -e hex 3.2.2
> > # file: 3.2.2
> >
> trusted.glusterfs.quota.242dcfd9-6aea-4cb8-beb2-c0ed91ad70d3.contri=0x09af6000
> >
> trusted.glusterfs.quota.c890be20-1bb9-4aec-a8d0-eacab0446f16.contri=0x13fda800
> > trusted.glusterfs.quota.dirty=0x3000
> > trusted.glusterfs.quota.size=0x13fda800
> >
> > For reference, another directory /modules/R/2.14.2 has only one
> > contribution attribute:
> > # getfattr -d -m quota -e hex 2.14.2
> > # file: 2.14.2
> >
> trusted.glusterfs.quota.c890be20-1bb9-4aec-a8d0-eacab0446f16.contri=0x00692800
> > trusted.glusterfs.quota.dirty=0x3000
> > trusted.glusterfs.quota.size=0x00692800
> >
> > Questions:
> > 1. Why wasn't the
> >
> trusted.glusterfs.quota.242dcfd9-6aea-4cb8-beb2-c0ed91ad70d3.contri=0x09af6000
> > cleaned up?
> > 2A. How can I remove old attributes from the fs, and then force a
> > re-calculation of contributions for the quota path /modules once I've
> > done this on all gluster nodes?
> > 2B. Or am I stuck yet again removing quotas completely, waiting for
> > the automated setfattr to remove the quotas for
> > c890be20-1bb9-4aec-a8d0-eacab0446f16 ID, manually removing attrs for
> > 242dcfd9-6aea-4cb8-beb2-c0ed91ad70d3, re-enabling quotas, waiting for
> > xattrs to be generated, then enabling limits?
> > 3. Shouldn't there be a command to re-trigger quota accounting on a
> > directory that confirms the attrs are set correctly and checks that
> > the contribution attr actually match disk usage?
> >
> > On Tue, Feb 2, 2016 at 3:00 AM, Manikandan Selvaganesh
> >  wrote:
> >> Hi Steve,
> >>
> >> As you have mentioned, if you are using a glusterfs version lesser than
> 3.7,
> >> then you are doing it right. We are sorry to say but unfortunately
> that's the only
> >> way(manually going and cleaning up the xattr's before enabling quota or
> wait for
> >> the process to complete itself, which would take quite some time
> depending upon the
> >> files) that can be done so as not to mess up quota
> enforcing/accounting. Also, we could
> >> not find anything that could help us with the logs too. Thanks for the
> >> point. We are in the process of writing blogs and documenting clearly
> about quota and
> >> it's internal working. There is an initial blog[1] which we have
> written. More blogs will
> >> follow.
> >>
> >> With glusterfs-3.7, we have introduced something called "Quota
> versioning".
> >> So whenever you enable quota, we are

Re: [Gluster-devel] NetBSD hang in quota-anon-fd-nfs.t

[Vijaikumar Mallikarjuna]

All quota test-cases uses 'tests/basic/quota.c' to write data

Does sync flags have any impact?


File:tests/basic/quota.c

 30 int
 31 file_write (char *filename, int bs, int count)
 32 {


 *46 fd = open (filename, O_RDWR|O_CREAT|O_SYNC, 0600);*
 47 while (i < count) {
 48 ret = nwrite(fd, buf, bs);
 49 if (ret == -1) {
 50 close (fd);
 51 goto out;
 52 }
 53 i++;
 54 }
 55
* 56 ret = fdatasync(fd);*


Thanks,
Vijay


On Mon, Jan 11, 2016 at 10:47 AM, Raghavendra Gowdappa 
wrote:

>
>
> - Original Message -
> > From: "Emmanuel Dreyfus" 
> > To: "Gluster Devel" 
> > Sent: Monday, January 11, 2016 8:56:58 AM
> > Subject: Re: [Gluster-devel] NetBSD hang in quota-anon-fd-nfs.t
> >
> > Emmanuel Dreyfus  wrote:
> >
> > > ps -axl shows the quota helper program is waiting on genput:
> > > UID  PID  PPID CPU PRI NI  VSZ  RSS WCHAN  STAT TTY  TIME COMMAND
> > >   0 9660 23707   0 124  0 3360 1080 genput D+   pts/2 0:00.01
> > > ./tests/basic/quota /mnt/nfs/0//0/1/2/3/4/5/6/7/8/9/new_file_2 256 4
> > >
> > > The process is stuck in kernel awaiting for a memory page to get
> > > unlocked.
> >
> > I reproduced the situation, and discovered the process is not really
> > hung. Tracing system calls in the quota procss shows that it does
> > complete write operations, thought ater a very long time. One write
> > system call that last 963s, for instance.
> >
> > It does not hang, but it does not look sane either.
>
> +vijay mallikarjuna, Manikandan.
>
> >
> > --
> > Emmanuel Dreyfus
> > http://hcpnet.free.fr/pubz
> > m...@netbsd.org
> > ___
> > Gluster-devel mailing list
> > Gluster-devel@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-devel
> >
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Quota-2

[Vijaikumar Mallikarjuna]

On Mon, Dec 21, 2015 at 9:43 AM, Nagaprasad Sathyanarayana <
nsath...@redhat.com> wrote:

> Not clear as to why quota 2 must support both DHT v1 and v2.  If customers
> continue to use dht v1, can we not allow them to use quota 1? We need not
> actively maintain quota v1 but only do reactive fixes.
>
>
> Idea is to have project quota in the early release and DHT-2 may not be
fully ready in the first release. so it is good to support DHT1 with
quota-2 till DHT-2 is stable.

Thanks,
Vijay



> Thanks
> Naga
>
>
> > On Dec 18, 2015, at 2:30 PM, Vijaikumar Mallikarjuna <
> vmall...@redhat.com> wrote:
> >
> > Hi All,
> >
> > Here is the summary of discussion we had today on Quota-v2
> >
> > Project, user and group quotas will use the same logic of accounting the
> usage
> >
> > Quota-v2 should be compatible with both DHT-v1 and DHT-v2
> >
> > Project quotas will use gfid of the given path as the project-id.
> > each inode will have a associated project-id which needs to be
> embedded within inode.
> > when creating new file, it will inherit project-id from its parent
> inode.
> > In case if parent inode doesn't contain project-id, then there
> should be a mechanism to
> > find the project-id given a gfid, so back-pointers can solve this
> problem
> > which xlator DHT/marker will set the project-id in the inode? this
> needs to be decided
> >
> > User and group quotas will use uid and gid to account the usage
> >
> > Quota has below set of operation which should be performed as a single
> transaction
> >   read current size
> >   read current contribution
> >   update delta to the contribution
> >   update size to user/project inode
> > In the current directory quota, to make this transaction crash
> consistent we use a mechanism of setting dirty flag in a parent inode.
> > This mechanism will not work efficiently with project quotas. Journaling
> will be the good solution for crash-consistency.
> >
> > Quota 2 depends on:
> >   back-pointers: to find project-id from gfid
> >   journaling: for crash consistency of accounting
> >
> >
> > Thanks,
> > Vijay
> > ___
> > Gluster-devel mailing list
> > Gluster-devel@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Quota-2

[Vijaikumar Mallikarjuna]

On Fri, Dec 18, 2015 at 8:28 PM, Shyam  wrote:

> 
>
> On 12/18/2015 04:00 AM, Vijaikumar Mallikarjuna wrote:
>
>> Hi All,
>>
>> Here is the summary of discussion we had today on Quota-v2
>>
>> Project, user and group quotas will use the same logic of accounting the
>> usage
>>
>> Quota-v2 should be compatible with both DHT-v1 and DHT-v2
>>
>> Project quotas will use gfid of the given path as the project-id.
>>  each inode will have a associated project-id which needs to be
>> embedded within inode.
>>  when creating new file, it will inherit project-id from its parent
>> inode.
>>  In case if parent inode doesn't contain project-id, then there
>> should be a mechanism to
>>  find the project-id given a gfid, so back-pointers can solve this
>> problem
>>
>
> Why would the parent not contain a project-id? and if it does not contain
> a project-id, why would you need to find a project-id?
>
> What I can visualize is that, the parent is not yet part of a quota
> project, hence it does not contain a project-id. When this becomes a part
> of some quota restriction it would start getting a quota. This seems to be
> one case where the project-id would be missing, but by choice.
>
> What am I missing here?
>
> There are two scenarios where project-id can be missing
1) When quota is enabled on a pre-existing data,
2) Directory created when one of the brick is down

Thanks,
Vijay


>  which xlator DHT/marker will set the project-id in the inode? this
>> needs to be decided
>> User and group quotas will use uid and gid to account the usage
>>
>> Quota has below set of operation which should be performed as a single
>> transaction
>>read current size
>>read current contribution
>>update delta to the contribution
>>update size to user/project inode
>> In the current directory quota, to make this transaction crash
>> consistent we use a mechanism of setting dirty flag in a parent inode.
>> This mechanism will not work efficiently with project quotas. Journaling
>> will be the good solution for crash-consistency.
>>
>> Quota 2 depends on:
>>back-pointers: to find project-id from gfid
>>journaling: for crash consistency of accounting
>>
>>
>> Thanks,
>> Vijay
>>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Quota-2

[Vijaikumar Mallikarjuna]

Hi All,

Here is the summary of discussion we had today on Quota-v2

Project, user and group quotas will use the same logic of accounting the
usage

Quota-v2 should be compatible with both DHT-v1 and DHT-v2

Project quotas will use gfid of the given path as the project-id.
each inode will have a associated project-id which needs to be embedded
within inode.
when creating new file, it will inherit project-id from its parent
inode.
In case if parent inode doesn't contain project-id, then there should
be a mechanism to
find the project-id given a gfid, so back-pointers can solve this
problem
which xlator DHT/marker will set the project-id in the inode? this
needs to be decided

User and group quotas will use uid and gid to account the usage

Quota has below set of operation which should be performed as a single
transaction
  read current size
  read current contribution
  update delta to the contribution
  update size to user/project inode
In the current directory quota, to make this transaction crash consistent
we use a mechanism of setting dirty flag in a parent inode.
This mechanism will not work efficiently with project quotas. Journaling
will be the good solution for crash-consistency.

Quota 2 depends on:
  back-pointers: to find project-id from gfid
  journaling: for crash consistency of accounting


Thanks,
Vijay
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] GlusterFS User and Group Quotas

[Vijaikumar Mallikarjuna]

Hi All,

Below is the design for '*GlusterFS User and Group Quotas', *please provide
your feedback on the same.


*Developers:*
Vijaikumar.M and Manikandan.S


*Introduction:*
User and Group quotas is to limit the amount of disk space for a
specified user/group ID.
This documents provides some details about how the accounting (marker
xlator) can be done
for user and group quotas


*Design:*
We have three different approaches, each has pros and cons

*Approach-1)*
T1 - For each file/dir 'file_x', create a contribution extended attribute
say 'trusted.glusterfs.quota.-contri'
T2 - In a lookup/write operation read the actual size from the stat-buf,
add the delta size to the contribution xattr
T3 - Create a file .glusterfs/quota/users/.
 Update size extended attribute say 'trusted.glusterfs.quota.size' by
adding the delta size calculated in T2

Same for group quotas a size xattr is updated under
.glusterfs/quota/groups/.

cons:
If the brick crashes after executing T2 and before T3. Now accounting
information is in-correct.
To recover and correct the accounting information, entire file-systems
needs to be crawled to fix the trusted.glusterfs.quota.size
value by summing up the contribution of all files with UID. But is a
slow process.


*Approach-2)*
T1 - For each file/dir 'file_x', create a contribution extended attribute
say 'trusted.glusterfs.quota.-contri'
T2 - create a directory '.glusterfs/quota/users/'
 create a hardlink for file file_x under this directories
T3 - In a lookup/write operation, set dirty flag
'trusted.glusterfs.quota.dirty' for directory '.glusterfs/quota/users/'
T4 - Read the actual size of a file from the stat-buf, add the delta size
to the contribution xattr
T5 - update size extended attribute say for directory
'.glusterfs/quota/users/'
T6 - unset the dirty flag

Same for group quotas a size xattr is updated under
.glusterfs/quota/groups/.

Problem of approach 1 of crawling entire brick is solved by only crawling
the directory which is set dirty.

cons:
Need to make sure that the hard-link for a file is consistent when
having another hardlinks
under .glusterfs/quota/users/ and .glusterfs/quota/groups/


*Approach-3)*
T1 - For each file/dir 'file_x', update a contribution entry in the
SQL-LITE DB (Create a DB file under .glusterfs/quota/)
T2 - In a lookup/write operation read the actual size from the statbuf, add
the update the size in the USER-QUOTA schema in the DB
T3 - In a lookup/write operation, set dirty flag
'trusted.glusterfs.quota.dirty' for directory '.glusterfs/quota/users/'

Atomicity problem found in approach 1 and 2 is solved by using DB
transactions.

Note: need to test the consistency of the SQL-LITE DB.

We feel approach-3 is more simpler and efficient way of implementing
user/group quotas.


Thanks,
Vijay
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Another peer probe problem after quota enable

[Vijaikumar Mallikarjuna]

Hi Sungsik,

We have filed a bug to track this issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1287996

Thanks,
Vijay


On Thu, Dec 3, 2015 at 12:31 PM, 박성식  wrote:

> Thanks to Vijay answer.
>
> Volume 'info' and 'cksum' file copy already know.
>
> I wanted to know if there is another solution for the 'quota-version'.
>
> It's now planning to create test automation scripts for basic
> full-features.
>
> I'll post another when issue is detected.
>
> Thank you once again for answers.
>
> Sungsik, Park.
>
>
>
> --
>
> Sung Sik, Park [박성식, 朴宬植]
>
> Email: mulg...@gmail.com 
>
> KR: +82 10-4518-2647
>
>
> ------
>
> On Thu, Dec 3, 2015 at 2:56 PM, Vijaikumar Mallikarjuna <
> vmall...@redhat.com> wrote:
>
>> Hi Sung Sik Park,
>>
>> Please find my comments inline.
>>
>>
>>
>> On Thu, Dec 3, 2015 at 7:19 AM, 박성식  wrote:
>>
>>> Hi all
>>>
>>> Gluster-3.7.6 in 'Peer Rejected' problem occurs in the following test
>>> case.(it doesn't occur if don't enable the volume quota)
>>>
>>> The problem is caused from the commit point
>>> '3d3176958b7da48dbacb1a5a0fedf26322a38297'.
>>>
>>> Can you help with the following problems?
>>>
>>> # Test enviroment
>>>
>>> 1. Ubuntu 14.04 Server
>>>(1) c2-dn1: 192.16.63.21
>>>(2) c2-dn2: 192.16.63.22
>>>(3) c2-dn3: 192.16.63.23
>>> 2. Glusterfs-3.7.6
>>>
>>> # Test procedure
>>>
>>> 1. Gluster installation
>>>
>>> $ git clone git://git.gluster.com/glusterfs.git git.gluster.com
>>> $ cd git.gluster.com/
>>> $ git checkout v3.7.6
>>> $ ./autogen.sh ; ./configure ; make ; make install ; ldconfig
>>>
>>> 2. Glusterd startup
>>>
>>> $ /etc/init.d/glusterd start
>>>
>>> 3. Gluster peer: 2 Server peer
>>>
>>> [c2-dn1] $ gluster peer probe 192.16.63.22
>>> [c2-dn1] $ gluster volume create tvol0001
>>> c2-dn1:/brick/tvol0001 c2-dn2:/brick/tvol0001 force
>>> [c2-dn1] $ gluster volume start tvol0001
>>> [c2-dn1] $ gluster volume quota tvol0001 enable
>>>
>>> 4. Another server peer
>>>
>>> [c2-dn1] $ gluster peer probe 192.16.63.23
>>> peer probe: success.
>>>
>>> [c2-dn1] $ gluster peer status
>>> Uuid: dd749b9b-d4e1-4b94-8f36-c03064bc5a57
>>> State: Peer Rejected (Connected)
>>>
>>> # c2-dn1 log
>>> [2015-12-03 00:37:52.464600] I [MSGID: 106487]
>>> [glusterd-handler.c:1422:__glusterd_handle_cli_list_friends] 0-glusterd:
>>> Received cli list req
>>> [2015-12-03 00:38:51.035018] I [MSGID: 106487]
>>> [glusterd-handler.c:1301:__glusterd_handle_cli_deprobe] 0-glusterd:
>>> Received CLI deprobe req
>>> [2015-12-03 00:38:51.036338] I [MSGID: 106493]
>>> [glusterd-rpc-ops.c:599:__glusterd_friend_remove_cbk] 0-glusterd: Received
>>> ACC from uuid: dd749b9b-d4e1-4b94-8f36-c03064bc5a57, host: c2-dn1,
>>> port: 0
>>> [2015-12-03 00:38:51.041859] I [MSGID: 106493]
>>> [glusterd-rpc-ops.c:695:__glusterd_friend_update_cbk] 0-management:
>>> Received ACC from uuid: cb22d272-d563-4bda-802b-c93798130108
>>> [2015-12-03 00:38:56.441189] I [MSGID: 106487]
>>> [glusterd-handler.c:1189:__glusterd_handle_cli_probe] 0-glusterd: Received
>>> CLI probe req 192.16.63.23 24007
>>> [2015-12-03 00:38:56.441828] I [MSGID: 106129]
>>> [glusterd-handler.c:3611:glusterd_probe_begin] 0-glusterd: Unable to find
>>> peerinfo for host: 192.16.63.23 (24007)
>>> [2015-12-03 00:38:56.445471] I [rpc-clnt.c:984:rpc_clnt_connection_init]
>>> 0-management: setting frame-timeout to 600
>>> [2015-12-03 00:38:56.445954] W [socket.c:869:__socket_keepalive]
>>> 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 23, Invalid
>>> argument
>>> [2015-12-03 00:38:56.445977] E [socket.c:2965:socket_connect]
>>> 0-management: Failed to set keep-alive: Invalid argument
>>> [2015-12-03 00:38:56.446232] I [MSGID: 106498]
>>> [glusterd-handler.c:3539:glusterd_friend_add] 0-management: connect
>>> returned 0
>>> [2015-12-03 00:38:56.457933] I [M

Re: [Gluster-devel] Another peer probe problem after quota enable

[Vijaikumar Mallikarjuna]

Hi Sung Sik Park,

Please find my comments inline.



On Thu, Dec 3, 2015 at 7:19 AM, 박성식  wrote:

> Hi all
>
> Gluster-3.7.6 in 'Peer Rejected' problem occurs in the following test
> case.(it doesn't occur if don't enable the volume quota)
>
> The problem is caused from the commit point
> '3d3176958b7da48dbacb1a5a0fedf26322a38297'.
>
> Can you help with the following problems?
>
> # Test enviroment
>
> 1. Ubuntu 14.04 Server
>(1) c2-dn1: 192.16.63.21
>(2) c2-dn2: 192.16.63.22
>(3) c2-dn3: 192.16.63.23
> 2. Glusterfs-3.7.6
>
> # Test procedure
>
> 1. Gluster installation
>
> $ git clone git://git.gluster.com/glusterfs.git git.gluster.com
> $ cd git.gluster.com/
> $ git checkout v3.7.6
> $ ./autogen.sh ; ./configure ; make ; make install ; ldconfig
>
> 2. Glusterd startup
>
> $ /etc/init.d/glusterd start
>
> 3. Gluster peer: 2 Server peer
>
> [c2-dn1] $ gluster peer probe 192.16.63.22
> [c2-dn1] $ gluster volume create tvol0001
> c2-dn1:/brick/tvol0001 c2-dn2:/brick/tvol0001 force
> [c2-dn1] $ gluster volume start tvol0001
> [c2-dn1] $ gluster volume quota tvol0001 enable
>
> 4. Another server peer
>
> [c2-dn1] $ gluster peer probe 192.16.63.23
> peer probe: success.
>
> [c2-dn1] $ gluster peer status
> Uuid: dd749b9b-d4e1-4b94-8f36-c03064bc5a57
> State: Peer Rejected (Connected)
>
> # c2-dn1 log
> [2015-12-03 00:37:52.464600] I [MSGID: 106487]
> [glusterd-handler.c:1422:__glusterd_handle_cli_list_friends] 0-glusterd:
> Received cli list req
> [2015-12-03 00:38:51.035018] I [MSGID: 106487]
> [glusterd-handler.c:1301:__glusterd_handle_cli_deprobe] 0-glusterd:
> Received CLI deprobe req
> [2015-12-03 00:38:51.036338] I [MSGID: 106493]
> [glusterd-rpc-ops.c:599:__glusterd_friend_remove_cbk] 0-glusterd: Received
> ACC from uuid: dd749b9b-d4e1-4b94-8f36-c03064bc5a57, host: c2-dn1,
> port: 0
> [2015-12-03 00:38:51.041859] I [MSGID: 106493]
> [glusterd-rpc-ops.c:695:__glusterd_friend_update_cbk] 0-management:
> Received ACC from uuid: cb22d272-d563-4bda-802b-c93798130108
> [2015-12-03 00:38:56.441189] I [MSGID: 106487]
> [glusterd-handler.c:1189:__glusterd_handle_cli_probe] 0-glusterd: Received
> CLI probe req 192.16.63.23 24007
> [2015-12-03 00:38:56.441828] I [MSGID: 106129]
> [glusterd-handler.c:3611:glusterd_probe_begin] 0-glusterd: Unable to find
> peerinfo for host: 192.16.63.23 (24007)
> [2015-12-03 00:38:56.445471] I [rpc-clnt.c:984:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2015-12-03 00:38:56.445954] W [socket.c:869:__socket_keepalive] 0-socket:
> failed to set TCP_USER_TIMEOUT -1000 on socket 23, Invalid argument
> [2015-12-03 00:38:56.445977] E [socket.c:2965:socket_connect]
> 0-management: Failed to set keep-alive: Invalid argument
> [2015-12-03 00:38:56.446232] I [MSGID: 106498]
> [glusterd-handler.c:3539:glusterd_friend_add] 0-management: connect
> returned 0
> [2015-12-03 00:38:56.457933] I [MSGID: 106511]
> [glusterd-rpc-ops.c:257:__glusterd_probe_cbk] 0-management: Received probe
> resp from uuid: dd749b9b-d4e1-4b94-8f36-c03064bc5a57, host: 192.16.63.23
> [2015-12-03 00:38:56.457992] I [MSGID: 106511]
> [glusterd-rpc-ops.c:417:__glusterd_probe_cbk] 0-glusterd: Received resp to
> probe req
> [2015-12-03 00:38:56.519048] I [MSGID: 106493]
> [glusterd-rpc-ops.c:481:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
> from uuid: dd749b9b-d4e1-4b94-8f36-c03064bc5a57, host: 192.16.63.23, port: 0
> [2015-12-03 00:38:56.524556] I [MSGID: 106163]
> [glusterd-handshake.c:1193:__glusterd_mgmt_hndsk_versions_ack]
> 0-management: using the op-version 30707
> [2015-12-03 00:38:56.527701] I [MSGID: 106490]
> [glusterd-handler.c:2895:__glusterd_handle_probe_query] 0-glusterd:
> Received probe from uuid: dd749b9b-d4e1-4b94-8f36-c03064bc5a57
> [2015-12-03 00:38:56.528598] I [MSGID: 106493]
> [glusterd-handler.c:2958:__glusterd_handle_probe_query] 0-glusterd:
> Responded to c2-dn3, op_ret: 0, op_errno: 0, ret: 0
> [2015-12-03 00:38:56.529154] I [MSGID: 106490]
> [glusterd-handler.c:2550:__glusterd_handle_incoming_friend_req] 0-glusterd:
> Received probe from uuid: dd749b9b-d4e1-4b94-8f36-c03064bc5a57
> [2015-12-03 00:38:56.529743] E [MSGID: 106010]
> [glusterd-utils.c:2727:glusterd_compare_friend_volume] 0-management:
> Version of Cksums tvol0001 differ. local cksum = 870323923, remote cksum =
> 870586066 on peer 192.16.63.23
> [2015-12-03 00:38:56.529905] I [MSGID: 106493]
> [glusterd-handler.c:3791:glusterd_xfer_friend_add_resp] 0-glusterd:
> Responded to 192.16.63.23 (0), ret: 0
> [2015-12-03 00:38:59.243087] I [MSGID: 106487]
> [glusterd-handler.c:1422:__glusterd_handle_cli_list_friends] 0-glusterd:
> Received cli list req
>
> # c2-dn3 log
> [2015-12-03 00:38:45.556047] I [MSGID: 106491]
> [glusterd-handler.c:2600:__glusterd_handle_incoming_unfriend_req]
> 0-glusterd: Received unfriend from uuid:
> 090e2d18-7a6a-4d11-89cd-7c583180cba3
> [2015-12-03 0

Re: [Gluster-devel] Object Quotas

[Vijaikumar Mallikarjuna]

On Thu, Nov 26, 2015 at 10:25 AM, Prashanth Pai  wrote:

>
> > We have sent an upstream patch[1] that enhances the list command by
> showing
> > used space in case of 'gluster v quota  list ' and shows
> > 'dir_count,file_count' with gluster v quota  list-objects
> '
> > command even when quota limit is not set(But quota/inode-quota should be
> > configured for the first time).
>
> Would enabling quota for the volume be enough ? For an existing volume
> with data, When are the xattrs populated ?
>
Yes, enabling quota for a volume will be enough.
When quota is enabled on existing data, back-end quota-crawl process is
initiated to set the xattrs. This process can take from secs to few hours
to complete the task, depending on the number of files present in the
volume.




>
> >
> > [1] http://review.gluster.org/#/c/12741/
>
> Cool, thanks.
>
> >
> > > > Commands supported for 'Object Quotas'
> > > > # volume quota  {limit-objects   []}
> > > > # volume quota  list-objects []
> > > > # volume quota  remove-objects 
> >
> > > The command "list-objects" and "remove-objects" is confusing. What it
> > > really does is something like "list-object-limit" and
> > > "remove-object-limit".
> >
> > > Yes, list-objects is to list the count, remove-objects is to remove the
> > > object limit set
>
> Can this be changed too ? From "list-objects" and "remove-objects" to
> "list-object-limit" and "remove-object-limit".
>
As part of patch submitted #12741, list-objects now will list the quota
usage count even if limit is not set, so list-object-limit may not be the
right command.
Yes, remove-objects can be changed to remove-object-limit

Thanks,
Vijay


> Thanks.
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Object Quotas

[Vijaikumar Mallikarjuna]

Hi Prashanth,

Thank you for the feedback, please find my comments inline.

On Tue, Nov 24, 2015 at 10:23 AM, Prashanth Pai  wrote:

> It would be good to expose the object count (dir and file separate) to
> client via a virtual xattr. I have a use-case to know the number of files
> without even having to set/use quota.
>
> If quota is enabled, you can get this data by doing getxattr
'trusted.glusterfs.quota.size.xx' from a auxiliary mount point.
As of now quota 'list/list-objects' will list the usage only if limit is
set. We will enhance this command to list the usage even if quota limit is
not set, I have file a bug for the same bug# 1284752.
You can get this data in xml format as well



> > Commands supported for 'Object Quotas'
> > # volume quota  {limit-objects   []}
> > # volume quota  list-objects []
> > # volume quota  remove-objects 
>
> The command "list-objects" and "remove-objects" is confusing. What it
> really does is something like "list-object-limit" and "remove-object-limit".
>
> Yes, list-objects is to list the count, remove-objects is to remove the
object limit set


> Thanks.
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Object Quotas

[Vijaikumar Mallikarjuna]

Hi All,

We have implemented a new feature 'Object quotas' in glusterfs-3.7, this
feature is same like 'Usage quota' and accounts on the numbers of inodes.

The mechanism is to store count of objects (inodes: files/directories) as
part of an extended attribute of a directory (same extended attribute used
for usage quotas).

Each directory's extended attribute value will indicate the number of
objects present in a tree with the directory being considered as the root
of the tree.

Commands supported for 'Object Quotas'

# volume quota  {limit-objects   []}

# volume quota  list-objects []

# volume quota  remove-objects 

'Object Quota' will be enabled as part of 'gluster volume quota 
enable'. If the glusterfs is upgraded to 3.7 where is the quota is alreday
enabled in the previous version before upgrade,

execute below command to enable 'Object Quota':

gluster volume inode-quota  enable

Please provide your feedback.

Thanks,

Vijay
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Object Quotas

[Vijaikumar Mallikarjuna]

Hi All,

We have implemented a new feature 'Object quotas' in glusterfs-3.7, this
feature is same like 'Usage quota' and accounts on the numbers of inodes.
The mechanism is to store count of objects (inodes: files/directories) as
part of an extended attribute of a directory (same extended attribute used
for usage quotas).
Each directory's extended attribute value will indicate the number of
objects present in a tree with the directory being considered as the root
of the tree.

Commands supported for 'Object Quotas'
# volume quota  {limit-objects   []}
# volume quota  list-objects []
# volume quota  remove-objects 

'Object Quota' will be enabled as part of 'gluster volume quota 
enable'. If the glusterfs is upgraded to 3.7 where is the quota is alreday
enabled in the previous version before upgrade,
execute below command to enable 'Object Quota':
gluster volume inode-quota  enable


Please provide your feedback.

Thanks,
Vijay
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Need advice re some major issues with glusterfind

[Vijaikumar Mallikarjuna]

Hi Kotresh/Venky,

Could you please provide your inputs on the change-log issues mentioned
below?

Thanks,
Vijay


On Fri, Oct 23, 2015 at 9:54 AM, Sincock, John [FLCPTY]  wrote:

>
> Hi Vijay, pls see below again (I'm wondering if top-posting would be
> easier, that's usually what I do, though I know some ppl don’t like it)
>
>
> On Wed, Oct 21, 2015 at 5:53 AM, Sincock, John [FLCPTY] <
> j.sinc...@fugro.com> wrote:
> Hi Everybody,
>
> We have recently upgraded our 220 TB gluster to 3.7.4, and we've been
> trying to use the new glusterfind feature but have been having some serious
> problems with it. Overall the glusterfind looks very promising, so I don't
> want to offend anyone by raising these issues.
>
> If these issues can be resolved or worked around, glusterfind will be a
> great feature.  So I would really appreciate any information or advice:
>
> 1) What can be done about the vast number of tiny changelogs? We are
> seeing often 5+ small 89 byte changelog files per minute on EACH brick.
> Larger files if busier. We've been generating these changelogs for a few
> weeks and have in excess of 10,000 or 12,000 on most bricks. This makes
> glusterfinds very, very slow, especially on a node which has a lot of
> bricks, and looks unsustainable in the long run. Why are these files so
> small, and why are there so many of them, and how are they supposed to be
> managed in the long run? The sheer number of these files looks sure to
> impact performance in the long run.
>
> 2) Pgfid xattribute is wreaking havoc with our backup scheme - when
> gluster adds this extended attribute to files it changes the ctime, which
> we were using to determine which files need to be archived. There should be
> a warning added to release notes & upgrade notes, so people can make a plan
> to manage this if required.
>
> Also, we ran a rebalance immediately after the 3.7.4 upgrade, and the
> rebalance took 5 days or so to complete, which looks like a major speed
> improvement over the more serial rebalance algorithm, so that's good. But I
> was hoping that the rebalance would also have had the side-effect of
> triggering all files to be labelled with the pgfid attribute by the time
> the rebalance completed, or failing that, after creation of an mlocate
> database across our entire gluster (which would have accessed every file,
> unless it is getting the info it needs only from directory inodes). Now it
> looks like ctimes are still being modified, and I think this can only be
> caused by files still being labelled with pgfids.
>
> How can we force gluster to get this pgfid labelling over and done with,
> for all files that are already on the volume? We can't have gluster
> continuing to add pgfids in bursts here and there, eg when files are read
> for the first time since the upgrade. We need to get it over and done with.
> We have just had to turn off pgfid creation on the volume until we can
> force gluster to get it over and done with in one go.
>
>
> Hi John,
>
> Was quota turned on/off before/after performing re-balance? If the pgfid
> is  missing, this can be healed by performing 'find  | xargs
> stat', all the files will get looked-up once and the pgfid healing will
> happen.
> Also could you please provide all the volume files under
> '/var/lib/glusterd/vols//*.vol'?
>
> Thanks,
> Vijay
>
>
> Hi Vijay
>
> Quota has never been turned on in our gluster, so it can’t be any
> quota-related xattrs which are resetting our ctimes, so I’m pretty sure it
> must be due to pgfids still being added.
>
> Thanks for the tip re using stat, if that should trigger the pgfid build
> on each file, then I will run that when I have a chance. We’ll have to get
> our archiving of data back up to date, re-enable pgfid build option, and
> then run the stat over a weekend or something, as it will take a while.
>
> I’m still quite concerned about the number of changelogs being generated.
> Do you know if there any plans to change the way changelogs are generated
> so there aren’t so many of them, and to process them more efficiently? I
> think this will be vital to improving performance of glusterfind in future,
> as there are currently an enormous number of these small changelogs being
> generated on each of our gluster bricks.
>
> Below is the volfile for one brick, the others are all equivalent. We
> haven’t tweaked the volume options much, besides increasing the io thread
> count to 32, and client/event threads to 6 (since we have a lot of small
> files on our gluster (30 million files, a lot of which are small, and some
> of which are large to very large):
>
>
> Hi John,
>
> PGFID xattrs are updated only when update-link-count-parent is enabled in
> the brick volume file. This option is enabled when quota is enabled on a
> volume.
> In the volume file you provided below
> has update-link-count-parent disabled, I am wondering why PGFID xattrs are
> updated.
>
> Thanks,
> Vijay
>
>
> Hi Vijay,
> somewhere in the 3.7.5 upgrade instructions or

Re: [Gluster-devel] Need advice re some major issues with glusterfind

[Vijaikumar Mallikarjuna]

On Thu, Oct 22, 2015 at 8:41 AM, Sincock, John [FLCPTY]  wrote:

> Pls see below
>
>
>
> *From:* Vijaikumar Mallikarjuna [mailto:vmall...@redhat.com]
> *Sent:* Wednesday, 21 October 2015 6:37 PM
> *To:* Sincock, John [FLCPTY]
> *Cc:* gluster-devel@gluster.org
> *Subject:* Re: [Gluster-devel] Need advice re some major issues with
> glusterfind
>
>
>
>
>
>
>
> On Wed, Oct 21, 2015 at 5:53 AM, Sincock, John [FLCPTY] <
> j.sinc...@fugro.com> wrote:
>
> Hi Everybody,
>
> We have recently upgraded our 220 TB gluster to 3.7.4, and we've been
> trying to use the new glusterfind feature but have been having some serious
> problems with it. Overall the glusterfind looks very promising, so I don't
> want to offend anyone by raising these issues.
>
> If these issues can be resolved or worked around, glusterfind will be a
> great feature.  So I would really appreciate any information or advice:
>
> 1) What can be done about the vast number of tiny changelogs? We are
> seeing often 5+ small 89 byte changelog files per minute on EACH brick.
> Larger files if busier. We've been generating these changelogs for a few
> weeks and have in excess of 10,000 or 12,000 on most bricks. This makes
> glusterfinds very, very slow, especially on a node which has a lot of
> bricks, and looks unsustainable in the long run. Why are these files so
> small, and why are there so many of them, and how are they supposed to be
> managed in the long run? The sheer number of these files looks sure to
> impact performance in the long run.
>
> 2) Pgfid xattribute is wreaking havoc with our backup scheme - when
> gluster adds this extended attribute to files it changes the ctime, which
> we were using to determine which files need to be archived. There should be
> a warning added to release notes & upgrade notes, so people can make a plan
> to manage this if required.
>
> Also, we ran a rebalance immediately after the 3.7.4 upgrade, and the
> rebalance took 5 days or so to complete, which looks like a major speed
> improvement over the more serial rebalance algorithm, so that's good. But I
> was hoping that the rebalance would also have had the side-effect of
> triggering all files to be labelled with the pgfid attribute by the time
> the rebalance completed, or failing that, after creation of an mlocate
> database across our entire gluster (which would have accessed every file,
> unless it is getting the info it needs only from directory inodes). Now it
> looks like ctimes are still being modified, and I think this can only be
> caused by files still being labelled with pgfids.
>
> How can we force gluster to get this pgfid labelling over and done with,
> for all files that are already on the volume? We can't have gluster
> continuing to add pgfids in bursts here and there, eg when files are read
> for the first time since the upgrade. We need to get it over and done with.
> We have just had to turn off pgfid creation on the volume until we can
> force gluster to get it over and done with in one go.
>
>
>
>
>
> Hi John,
>
>
>
> Was quota turned on/off before/after performing re-balance? If the pgfid
> is  missing, this can be healed by performing 'find  | xargs
> stat', all the files will get looked-up once and the pgfid healing will
> happen.
>
> Also could you please provide all the volume files under
> '/var/lib/glusterd/vols//*.vol'?
>
>
>
> Thanks,
>
> Vijay
>
>
>
>
>
> Hi Vijay
>
>
>
> Quota has never been turned on in our gluster, so it can’t be any
> quota-related xattrs which are resetting our ctimes, so I’m pretty sure it
> must be due to pgfids still being added.
>
>
>
> Thanks for the tip re using stat, if that should trigger the pgfid build
> on each file, then I will run that when I have a chance. We’ll have to get
> our archiving of data back up to date, re-enable pgfid build option, and
> then run the stat over a weekend or something, as it will take a while.
>
>
>
> I’m still quite concerned about the number of changelogs being generated.
> Do you know if there any plans to change the way changelogs are generated
> so there aren’t so many of them, and to process them more efficiently? I
> think this will be vital to improving performance of glusterfind in future,
> as there are currently an enormous number of these small changelogs being
> generated on each of our gluster bricks.
>
>
>
Below is the volfile for one brick, the others are all equivalent. We
> haven’t tweaked the volume options much, besides increasing the io thread
> count to 32, and client/event threads to 6 (since we have a lot of small
> files on our gluster 

Re: [Gluster-devel] Need advice re some major issues with glusterfind

[Vijaikumar Mallikarjuna]

On Wed, Oct 21, 2015 at 5:53 AM, Sincock, John [FLCPTY]  wrote:

> Hi Everybody,
>
> We have recently upgraded our 220 TB gluster to 3.7.4, and we've been
> trying to use the new glusterfind feature but have been having some serious
> problems with it. Overall the glusterfind looks very promising, so I don't
> want to offend anyone by raising these issues.
>
> If these issues can be resolved or worked around, glusterfind will be a
> great feature.  So I would really appreciate any information or advice:
>
> 1) What can be done about the vast number of tiny changelogs? We are
> seeing often 5+ small 89 byte changelog files per minute on EACH brick.
> Larger files if busier. We've been generating these changelogs for a few
> weeks and have in excess of 10,000 or 12,000 on most bricks. This makes
> glusterfinds very, very slow, especially on a node which has a lot of
> bricks, and looks unsustainable in the long run. Why are these files so
> small, and why are there so many of them, and how are they supposed to be
> managed in the long run? The sheer number of these files looks sure to
> impact performance in the long run.
>
> 2) Pgfid xattribute is wreaking havoc with our backup scheme - when
> gluster adds this extended attribute to files it changes the ctime, which
> we were using to determine which files need to be archived. There should be
> a warning added to release notes & upgrade notes, so people can make a plan
> to manage this if required.
>
> Also, we ran a rebalance immediately after the 3.7.4 upgrade, and the
> rebalance took 5 days or so to complete, which looks like a major speed
> improvement over the more serial rebalance algorithm, so that's good. But I
> was hoping that the rebalance would also have had the side-effect of
> triggering all files to be labelled with the pgfid attribute by the time
> the rebalance completed, or failing that, after creation of an mlocate
> database across our entire gluster (which would have accessed every file,
> unless it is getting the info it needs only from directory inodes). Now it
> looks like ctimes are still being modified, and I think this can only be
> caused by files still being labelled with pgfids.
>
> How can we force gluster to get this pgfid labelling over and done with,
> for all files that are already on the volume? We can't have gluster
> continuing to add pgfids in bursts here and there, eg when files are read
> for the first time since the upgrade. We need to get it over and done with.
> We have just had to turn off pgfid creation on the volume until we can
> force gluster to get it over and done with in one go.
>
> Hi John,

Was quota turned on/off before/after performing re-balance? If the pgfid is
 missing, this can be healed by performing 'find  | xargs
stat', all the files will get looked-up once and the pgfid healing will
happen.
Also could you please provide all the volume files under
'/var/lib/glusterd/vols//*.vol'?

Thanks,
Vijay






> 3) Files modified just before a glusterfind pre are often not included in
> the changed files list, unless pre command is run again a bit later - I
> think changelogs are missing very recent changes and need to be flushed or
> something before the pre command uses them?
>
> 4) BUG: Glusterfind follows symlinks off bricks and onto NFS mounted
> directories (and will cause these shares to be mounted if you have autofs
> enabled). Glusterfind should definitely not follow symlinks, but it does.
> For now, we are getting around this by turning off autofs when re run
> glusterfinds, but this should not be necessary. Glusterfind must be fixed
> so it never follows symlinks and never leaves the brick it is currently
> searching.
>
> 5) We have one of our nodes  with 16 bricks, and on this machine,
> glusterfind pre command seems to get stuck pegging all 8 cores to 100%, an
> strace of an offending processes gives an endless stream of these lseeks
> and reads and very little else. What is going on here? It doesn't look
> right... :
>
> lseek(13, 17188864, SEEK_SET)   = 17188864
> read(13,
> "\r\0\0\0\4\0J\0\3\25\2\"\0013\0J\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
> 1024) = 1024
> lseek(13, 17189888, SEEK_SET)   = 17189888
> read(13,
> "\r\0\0\0\4\0\"\0\3\31\0020\1#\0\"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
> 1024) = 1024
> lseek(13, 17190912, SEEK_SET)   = 17190912
> read(13,
> "\r\0\0\0\3\0\365\0\3\1\1\372\0\365\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
> 1024) = 1024
> lseek(13, 17191936, SEEK_SET)   = 17191936
> read(13,
> "\r\0\0\0\4\0F\0\3\17\2\"\0017\0F\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
> 1024) = 1024
> lseek(13, 17192960, SEEK_SET)   = 17192960
> read(13,
> "\r\0\0\0\4\0006\0\2\371\2\4\1\31\0006\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
> 1024) = 1024
> lseek(13, 17193984, SEEK_SET)   = 17193984
> read(13,
> "\r\0\0\0\4\0L\0\3\31\2\36\1/\0L\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024)
> = 1024
>
> I saved one of these straces for 20 or 30 secs or so, and then 

[Gluster-devel] [release-3.7] seeing multiple crashes with 3.7 linux regression

[Vijaikumar Mallikarjuna]

*https://build.gluster.org/job/rackspace-regression-2GB-triggered/14753/consoleFull
*
#gdb -ex 'set sysroot ./' -ex 'core-file ./build/install/cores/core.7122'
 /build/install/sbin/glusterfsd

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00407eae in emancipate (ctx=0x0, ret=-1) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/glusterfsd/src/glusterfsd.c:1329
1329 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/glusterfsd/src/glusterfsd.c:
No such file or directory.
(gdb) bt
#0  0x00407eae in emancipate (ctx=0x0, ret=-1) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/glusterfsd/src/glusterfsd.c:1329
#1  0x0040f806 in mgmt_pmap_signin_cbk (req=0x7f51e806fdec,
iov=0x7f51eece15e0, count=1, myframe=0x7f51e806ee1c) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/glusterfsd/src/glusterfsd-mgmt.c:2174
#2  0x7f51fab1a6c7 in saved_frames_unwind (saved_frames=0x1535fb0) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-clnt.c:366
#3  0x7f51fab1a766 in saved_frames_destroy (frames=0x1535fb0) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-clnt.c:383
#4  0x7f51fab1abf8 in rpc_clnt_connection_cleanup (conn=0x1534b70) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-clnt.c:536
#5  0x7f51fab1b670 in rpc_clnt_notify (trans=0x1534fc0,
mydata=0x1534b70, event=RPC_TRANSPORT_DISCONNECT, data=0x1534fc0) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-clnt.c:856
#6  0x7f51fab17af3 in rpc_transport_notify (this=0x1534fc0,
event=RPC_TRANSPORT_DISCONNECT, data=0x1534fc0) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-transport.c:544
#7  0x7f51f0305621 in socket_event_poll_err (this=0x1534fc0) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:1151
#8  0x7f51f030a34c in socket_event_handler (fd=9, idx=1,
data=0x1534fc0, poll_in=1, poll_out=0, poll_err=24) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:2356
#9  0x7f51fadcb7c0 in event_dispatch_epoll_handler
(event_pool=0x14fac90, event=0x7f51eece1e70) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:575
#10 0x7f51fadcbbae in event_dispatch_epoll_worker (data=0x1536180) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:678
#11 0x7f51fa032a51 in start_thread () from ./lib64/libpthread.so.0
#12 0x7f51f999c93d in clone () from ./lib64/libc.so.6


*https://build.gluster.org/job/rackspace-regression-2GB-triggered/14748/consoleFull
*
#gdb -ex 'set sysroot ./' -ex 'core-file ./build/install/cores/core.25320'
./build/install/sbin/glusterfs
#0  0x7fae978ccb0f in afr_local_replies_wipe (local=0x0,
priv=0x7fae900125b0) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/afr/src/afr-common.c:1241
1241 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/afr/src/afr-common.c:
No such file or directory.
(gdb) bt
#0  0x7fae978ccb0f in afr_local_replies_wipe (local=0x0,
priv=0x7fae900125b0) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/afr/src/afr-common.c:1241
#1  0x7fae978b7aaf in afr_selfheal_inodelk (frame=0x7fae8c000c0c,
this=0x7fae9000a6d0, inode=0x7fae8c00609c, dom=0x7fae900099f0
"patchy-replicate-0", off=8126464, size=131072, locked_on=0x7fae96b4f110 "")
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/afr/src/afr-self-heal-common.c:879
#2  0x7fae978bbeb5 in afr_selfheal_data_block (frame=0x7fae8c000c0c,
this=0x7fae9000a6d0, fd=0x7fae8c006e6c, source=0,
healed_sinks=0x7fae96b4f8a0 "", offset=8126464, size=131072, type=1,
replies=0x7fae96b4f2b0)
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/afr/src/afr-self-heal-data.c:243
#3  0x7fae978bc91d in afr_selfheal_data_do (frame=0x7fae8c006c9c,
this=0x7fae9000a6d0, fd=0x7fae8c006e6c, source=0,
healed_sinks=0x7fae96b4f8a0 "", replies=0x7fae96b4f2b0)
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/afr/src/afr-self-heal-data.c:365
#4  0x7fae978bdc7b in __afr_selfheal_data (frame=0x7fae8c006c9c,
this=0x7fae9000a6d0, fd=0x7fae8c006e6c, locked_on=0x7fae96b4fa00
"\001\001\240")
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/afr/src/afr-self-heal-data.c:719
#5  0x7fae978be0a0 in afr_selfheal_data (frame=0x7fae8c006c9c,
this=0x7fae9000a6d0, inode=0x7fae8c00609c) a