Re: [Gluster-devel] glusterd crashing

[Atin Mukherjee]

-Atin
Sent from one plus one
On 05-Mar-2016 11:46 am, "Ajil Abraham"  wrote:
>
> Thanks for all the support.  After handling the input validation in my
code, Glusterd no longer crashes.  I am still waiting for clearance from my
superior to pass on all the details. Expecting him to revert by this Sunday.
Great to know that and we appreciate your contribution, if you happen to
find any issues feel free to send patches :)
>
> - Ajil
>
> On Fri, Mar 4, 2016 at 10:20 AM, Joseph Fernandes 
wrote:
>>
>> Well that may not be completely correct !
>>
>> Its  "gluster volume status all", unlike volume maintenance operation
which are rare.
>>
>> Status can be issued multiple times in a day or might be put in a
script/cron-job to check the health of the
>> cluster.
>> But anyways the fix is ready as the bug says.
>>
>> Crash is what we need to worry about.
>>
>> ~Joe
>>
>> - Original Message -
>> > From: "Atin Mukherjee" 
>> > To: "Joseph Fernandes" , "Atin Mukherjee" <
atin.mukherje...@gmail.com>
>> > Cc: "Gluster Devel" , "Ajil Abraham" <
ajil95.abra...@gmail.com>
>> > Sent: Friday, March 4, 2016 9:37:43 AM
>> > Subject: Re: [Gluster-devel] glusterd crashing
>> >
>> >
>> >
>> > On 03/04/2016 07:10 AM, Joseph Fernandes wrote:
>> > > Might be this bug can give some context on the mem-leak (fix recently
>> > > merged on master but not on 3.7.x)
>> > >
>> > > https://bugzilla.redhat.com/show_bug.cgi?id=1287517
>> > Yes, this is what we'd be fixing in 3.7.x too, but if you refer to [1]
>> > the hike is seen when a command is run in a loop which is typically not
>> > a use case in any production setup.
>> >
>> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1287517#c15
>> > >
>> > > ~Joe
>> > >
>> > >
>> > > - Original Message -
>> > >> From: "Atin Mukherjee" 
>> > >> To: "Joseph Fernandes" 
>> > >> Cc: "Gluster Devel" , "Ajil Abraham"
>> > >> 
>> > >> Sent: Friday, March 4, 2016 7:01:54 AM
>> > >> Subject: Re: [Gluster-devel] glusterd crashing
>> > >>
>> > >> -Atin
>> > >> Sent from one plus one
>> > >> On 04-Mar-2016 6:12 am, "Joseph Fernandes" 
wrote:
>> > >>>
>> > >>> Hi Ajil,
>> > >>>
>> > >>> Well few things,
>> > >>>
>> > >>> 1. Whenever you see a crash its better to send across the
Backtrace(BT)
>> > >> using gdb and attach the log files (or share it via some cloud
drive)
>> > >>>
>> > >>> 2. About the memory leak, What kind of tools are you using for
profiling
>> > >> memory, valgrind ? if so please attach the valgrind reports.
>> > >>>$> glusterd --xlator-option *.run-with-valgrind=yes
>> > >>>
>> > >>> 3. Well I am not sure if glusterd uses any of the mempools as we
do in
>> > >> client and brick processes, Atin can shed some light on this.
>> > >>>Well In that case you can used the statedump mechanism check for
>> > >> mem-leaks check the glusterfs/doc/debugging/statedump.md
>> > >> GlusterD does use mempool and it has infra for capturing statedump
as
>> > >> well.
>> > >> I am aware of few bytes of memory leaks in few paths which is
really not a
>> > >> huge concern but it shouldn't crash.
>> > >>>
>> > >>> Hope this helps
>> > >>>
>> > >>> ~Joe
>> > >>>
>> > >>>
>> > >>> - Original Message -
>> >  From: "Ajil Abraham" 
>> >  To: "Atin Mukherjee" 
>> >  Cc: "Gluster Devel" 
>> >  Sent: Thursday, March 3, 2016 10:48:56 PM
>> >  Subject: Re: [Gluster-devel] glusterd crashing
>> > 
>> >  Hi Atin,
>> > 
>> >  The inputs I use are as per the requirements of a project I am
working
>> > >> on for
>> >  one of the large finance institutions in Dubai. I will try to
handle the
>> >  input validation within my code. I uncovered some of the issues
while
>> > >> doing
>> >  a thorough testing of my code.
>> > 
>> >  I tried with 3.7.6 and also my own build from master branch. I
will
>> > >> check
>> >  with my superiors before sending you backtrace and other details.
So
>> > >> far, I
>> >  have seen memory leak in 100s of KBs.
>> > 
>> >  -Ajil
>> > 
>> > 
>> >  On Thu, Mar 3, 2016 at 10:17 PM, Atin Mukherjee <
>> > >> atin.mukherje...@gmail.com
>> > > wrote:
>> > 
>> > 
>> > 
>> > 
>> >  Hi Ajil,
>> > 
>> >  Its good to see that you are doing a thorough testing gluster.
From
>> > >> your mail
>> >  it looks like your automation focuses on mostly negative tests. I
need
>> > >> few
>> >  additional details to get to know whether they are known:
>> > 
>> >  1. Version of gluster
>> >  2. Backtrace of the crash along with reproducer
>> >  3. Amount of memory leak in terms of bytes/KB/MB?? Have you
already
>> >  identified them?
>> > 

Re: [Gluster-devel] [Gluster-users] Default quorum for 2 way replication

[Pranith Kumar Karampuri]

On 03/04/2016 10:05 PM, Diego Remolina wrote:

I run a few two node glusterfs instances, but always have a third
machine acting as an arbiter. I am with Jeff on this one, better safe
than sorry.

Setting up a 3rd system without bricks to achieve quorum is very easy.


This is server side quorum. This is good. But what we are discussing 
here is for just 2 nodes, what should be the default.


Pranith


Diego



On Fri, Mar 4, 2016 at 10:40 AM, Jeff Darcy  wrote:

I like the default to be 'none'. Reason: If we have 'auto' as quorum for
2-way replication and first brick dies, there is no HA. If users are
fine with it, it is better to use plain distribute volume

"Availability" is a tricky word.  Does it mean access to data now, or
later despite failure?  Taking a volume down due to loss of quorum might
be equivalent to having no replication in the first sense, but certainly
not in the second.  When the possibility (likelihood?) of split brain is
considered, enforcing quorum actually does a *better* job of preserving
availability in the second sense.  I believe this second sense is most
often what users care about, and therefore quorum enforcement should be
the default.

I think we all agree that quorum is a bit slippery when N=2.  That's
where there really is a tradeoff between (immediate) availability and
(highest levels of) data integrity.  That's why arbiters showed up first
in the NSR specs, and later in AFR.  We should definitely try to push
people toward N>=3 as much as we can.  However, the ability to "scale
down" is one of the things that differentiate us vs. both our Ceph
cousins and our true competitors.  Many of our users will stop at N=2 no
matter what we say.  However unwise that might be, we must still do what
we can to minimize harm when things go awry.
___
Gluster-users mailing list
gluster-us...@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] glusterd crashing

[Ajil Abraham]

Thanks for all the support.  After handling the input validation in my
code, Glusterd no longer crashes.  I am still waiting for clearance from my
superior to pass on all the details. Expecting him to revert by this Sunday.

- Ajil

On Fri, Mar 4, 2016 at 10:20 AM, Joseph Fernandes 
wrote:

> Well that may not be completely correct !
>
> Its  "gluster volume status all", unlike volume maintenance operation
> which are rare.
>
> Status can be issued multiple times in a day or might be put in a
> script/cron-job to check the health of the
> cluster.
> But anyways the fix is ready as the bug says.
>
> Crash is what we need to worry about.
>
> ~Joe
>
> - Original Message -
> > From: "Atin Mukherjee" 
> > To: "Joseph Fernandes" , "Atin Mukherjee" <
> atin.mukherje...@gmail.com>
> > Cc: "Gluster Devel" , "Ajil Abraham" <
> ajil95.abra...@gmail.com>
> > Sent: Friday, March 4, 2016 9:37:43 AM
> > Subject: Re: [Gluster-devel] glusterd crashing
> >
> >
> >
> > On 03/04/2016 07:10 AM, Joseph Fernandes wrote:
> > > Might be this bug can give some context on the mem-leak (fix recently
> > > merged on master but not on 3.7.x)
> > >
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1287517
> > Yes, this is what we'd be fixing in 3.7.x too, but if you refer to [1]
> > the hike is seen when a command is run in a loop which is typically not
> > a use case in any production setup.
> >
> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1287517#c15
> > >
> > > ~Joe
> > >
> > >
> > > - Original Message -
> > >> From: "Atin Mukherjee" 
> > >> To: "Joseph Fernandes" 
> > >> Cc: "Gluster Devel" , "Ajil Abraham"
> > >> 
> > >> Sent: Friday, March 4, 2016 7:01:54 AM
> > >> Subject: Re: [Gluster-devel] glusterd crashing
> > >>
> > >> -Atin
> > >> Sent from one plus one
> > >> On 04-Mar-2016 6:12 am, "Joseph Fernandes" 
> wrote:
> > >>>
> > >>> Hi Ajil,
> > >>>
> > >>> Well few things,
> > >>>
> > >>> 1. Whenever you see a crash its better to send across the
> Backtrace(BT)
> > >> using gdb and attach the log files (or share it via some cloud drive)
> > >>>
> > >>> 2. About the memory leak, What kind of tools are you using for
> profiling
> > >> memory, valgrind ? if so please attach the valgrind reports.
> > >>>$> glusterd --xlator-option *.run-with-valgrind=yes
> > >>>
> > >>> 3. Well I am not sure if glusterd uses any of the mempools as we do
> in
> > >> client and brick processes, Atin can shed some light on this.
> > >>>Well In that case you can used the statedump mechanism check for
> > >> mem-leaks check the glusterfs/doc/debugging/statedump.md
> > >> GlusterD does use mempool and it has infra for capturing statedump as
> > >> well.
> > >> I am aware of few bytes of memory leaks in few paths which is really
> not a
> > >> huge concern but it shouldn't crash.
> > >>>
> > >>> Hope this helps
> > >>>
> > >>> ~Joe
> > >>>
> > >>>
> > >>> - Original Message -
> >  From: "Ajil Abraham" 
> >  To: "Atin Mukherjee" 
> >  Cc: "Gluster Devel" 
> >  Sent: Thursday, March 3, 2016 10:48:56 PM
> >  Subject: Re: [Gluster-devel] glusterd crashing
> > 
> >  Hi Atin,
> > 
> >  The inputs I use are as per the requirements of a project I am
> working
> > >> on for
> >  one of the large finance institutions in Dubai. I will try to
> handle the
> >  input validation within my code. I uncovered some of the issues
> while
> > >> doing
> >  a thorough testing of my code.
> > 
> >  I tried with 3.7.6 and also my own build from master branch. I will
> > >> check
> >  with my superiors before sending you backtrace and other details. So
> > >> far, I
> >  have seen memory leak in 100s of KBs.
> > 
> >  -Ajil
> > 
> > 
> >  On Thu, Mar 3, 2016 at 10:17 PM, Atin Mukherjee <
> > >> atin.mukherje...@gmail.com
> > > wrote:
> > 
> > 
> > 
> > 
> >  Hi Ajil,
> > 
> >  Its good to see that you are doing a thorough testing gluster. From
> > >> your mail
> >  it looks like your automation focuses on mostly negative tests. I
> need
> > >> few
> >  additional details to get to know whether they are known:
> > 
> >  1. Version of gluster
> >  2. Backtrace of the crash along with reproducer
> >  3. Amount of memory leak in terms of bytes/KB/MB?? Have you already
> >  identified them?
> > 
> >  -Atin
> >  Sent from one plus one
> >  On 03-Mar-2016 10:01 pm, "Ajil Abraham" < ajil95.abra...@gmail.com
> >
> > >> wrote:
> > 
> > 
> > 
> >  For my project, I am trying to do some automation using glusterd.
> It is
> > >> very
> >  frustrating to see it crashing frequently. Looks 

[Gluster-devel] GlusterFS readdir-ahead

[Keiviw]

hi,
Only if a preload(a readdir request not initiated by application, but instead 
triggered by readdir-ahead in an attempt to pre-emptively fill the 
readdir-ahead buffer) is in process,the client will wait for its completion. If 
the preload has completed, and the client's readdir requests wolud reply from 
the buffer,not the bricks.And  here is my question, when the first buffer was 
completed, another buffer will continually prefetch the denties, at the same 
time, the client fetches denties from the fisrt buffer?___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Default quorum for 2 way replication

[Diego Remolina]

I run a few two node glusterfs instances, but always have a third
machine acting as an arbiter. I am with Jeff on this one, better safe
than sorry.

Setting up a 3rd system without bricks to achieve quorum is very easy.

Diego



On Fri, Mar 4, 2016 at 10:40 AM, Jeff Darcy  wrote:
>> I like the default to be 'none'. Reason: If we have 'auto' as quorum for
>> 2-way replication and first brick dies, there is no HA. If users are
>> fine with it, it is better to use plain distribute volume
>
> "Availability" is a tricky word.  Does it mean access to data now, or
> later despite failure?  Taking a volume down due to loss of quorum might
> be equivalent to having no replication in the first sense, but certainly
> not in the second.  When the possibility (likelihood?) of split brain is
> considered, enforcing quorum actually does a *better* job of preserving
> availability in the second sense.  I believe this second sense is most
> often what users care about, and therefore quorum enforcement should be
> the default.
>
> I think we all agree that quorum is a bit slippery when N=2.  That's
> where there really is a tradeoff between (immediate) availability and
> (highest levels of) data integrity.  That's why arbiters showed up first
> in the NSR specs, and later in AFR.  We should definitely try to push
> people toward N>=3 as much as we can.  However, the ability to "scale
> down" is one of the things that differentiate us vs. both our Ceph
> cousins and our true competitors.  Many of our users will stop at N=2 no
> matter what we say.  However unwise that might be, we must still do what
> we can to minimize harm when things go awry.
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Default quorum for 2 way replication

[Jeff Darcy]

> I like the default to be 'none'. Reason: If we have 'auto' as quorum for
> 2-way replication and first brick dies, there is no HA. If users are
> fine with it, it is better to use plain distribute volume

"Availability" is a tricky word.  Does it mean access to data now, or
later despite failure?  Taking a volume down due to loss of quorum might
be equivalent to having no replication in the first sense, but certainly
not in the second.  When the possibility (likelihood?) of split brain is
considered, enforcing quorum actually does a *better* job of preserving
availability in the second sense.  I believe this second sense is most
often what users care about, and therefore quorum enforcement should be
the default.

I think we all agree that quorum is a bit slippery when N=2.  That's
where there really is a tradeoff between (immediate) availability and
(highest levels of) data integrity.  That's why arbiters showed up first
in the NSR specs, and later in AFR.  We should definitely try to push
people toward N>=3 as much as we can.  However, the ability to "scale
down" is one of the things that differentiate us vs. both our Ceph
cousins and our true competitors.  Many of our users will stop at N=2 no
matter what we say.  However unwise that might be, we must still do what
we can to minimize harm when things go awry.
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] glusterfs-3.6.9 released

[FNU Raghavendra Manjunath]

Hi,
glusterfs-3.6.9 has been released and the packages for RHEL/Fedora/Centos
can be found here.http://download.gluster.org/pub/gluster/glusterfs/3.6/LATEST/

Requesting people running 3.6.x to please try it out and let us know if
there are any issues.

This release supposedly fixes the bugs listed below since 3.6.8 was made
available. Thanks to all who submitted patches, reviewed the changes.

1302541 - Problem when enabling quota : Could not start quota auxiliary mount
1302310 - log improvements:- enabling quota on a volume reports
numerous entries of "contribution node
list is empty which is an error" in brick logs

1308806 - tests : Modifying tests for crypt xlator

1304668 - Add missing release-notes on the 3.6 branch
1296931 - Installation of glusterfs-3.6.8 fails on CentOS-7


Regards,

Raghavendra Bhat
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] gluster-nagios-common missing dependencies?

[Niels de Vos]

On Fri, Mar 04, 2016 at 09:11:33AM -0500, Glomski, Patrick wrote:
> Just an FYI, I installed gluster-nagios-common on a CentOS7 machine
> (started with minimal server, so there wasn't much there) to monitor the
> bricks in a gluster array. python-ethtool seems to be missing as a
> dependency in the rpm spec:
> 
> import glusternagios.glustercli as gcli
> >   File "/usr/lib64/python2.7/site-packages/glusternagios/glustercli.py",
> > line 21, in 
> > import ethtool
> >
> 
> I was using the latest 1.1 version:
> 
> http://download.gluster.org/pub/gluster/glusterfs-nagios/1.1.0/EPEL.repo/gluster-nagios-110-epel.repo

Thanks for reporting this. The nagios packages are currently undergoing
a Fedora packaging review so that they can get included in the
distribution.  Hopefully things like this get caught during the review
and additional testing.

The bug for the review of gluster-nagios-common can be found here:
  https://bugzilla.redhat.com/show_bug.cgi?id=1313159

Other related packeges are linked in the "See Also" section of the bug.

Cheers,
Niels


signature.asc
Description: PGP signature
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Default quorum for 2 way replication

[Shyam]

On 03/04/2016 07:30 AM, Pranith Kumar Karampuri wrote:



On 03/04/2016 05:47 PM, Bipin Kunal wrote:

HI Pranith,

Thanks for starting this mail thread.

Looking from a user perspective most important is to get a "good copy"
of data.  I agree that people use replication for HA but having stale
data with HA will not have any value.
So I will suggest to make auto quorum as default configuration even
for 2-way replication.

If user is willing to lose data at the cost of HA, he always have
option disable it. But default preference should be data and its
integrity.


I think we need to consider *maintenance* activities on the volume, like 
replacing a brick in a replica pair, or upgrading one half of the 
replica and then the other, at which time the replica group would 
function read-only, if we choose 'auto' in a 2-way replicated state, is 
this correct?


Having said the above, we already have the option in place, right? I.e 
admins can already choose 'auto', it is just the default that we are 
discussing. This could also be tackled via documentation/best practices 
("yeah right! who reads those again?" is a valid comment here).


I guess we need to be clear (in documentation or otherwise) what they 
get when they choose one over the other (like the HA point below and 
also upgrade concerns etc.), irrespective of how this discussion ends 
(just my 2 c's).




That is the point. There is an illusion of choice between Data integrity
and HA. But we are not *really* giving HA, are we? HA will be there only
if second brick in the replica pair goes down. In your typical


@Pranith, can you elaborate on this? I am not so AFR savvy, so unable to 
comprehend why HA is available if only when the second brick goes down 
and is not when the first does. Just helps in understanding the issue at 
hand.



deployment, we can't really give any guarantees about what brick will go
down when. So I am not sure if we can consider it as HA. But I would
love to hear what others have to say about this as well. If majority of
users say they need it to be auto, you will definitely see a patch :-).

Pranith


Thanks,
Bipin Kunal

On Fri, Mar 4, 2016 at 5:43 PM, Ravishankar N 
wrote:

On 03/04/2016 05:26 PM, Pranith Kumar Karampuri wrote:

hi,
  So far default quorum for 2-way replication is 'none' (i.e.
files/directories may go into split-brain) and for 3-way replication
and
arbiter based replication it is 'auto' (files/directories won't go into
split-brain). There are requests to make default as 'auto' for 2-way
replication as well. The line of reasoning is that people value data
integrity (files not going into split-brain) more than HA (operation of
mount even when bricks go down). And admins should explicitly change
it to
'none' when they are fine with split-brains in 2-way replication. We
were
wondering if you have any inputs about what is a sane default for 2-way
replication.

I like the default to be 'none'. Reason: If we have 'auto' as quorum
for
2-way replication and first brick dies, there is no HA.



+1.  Quorum does not make sense when there are only 2 parties. There
is no
majority voting. Arbiter volumes are a better option.
If someone wants some background, please see 'Client quorum' and
'Replica 2
and Replica 3 volumes' section of
http://gluster.readthedocs.org/en/latest/Administrator%20Guide/arbiter-volumes-and-quorum/


-Ravi


If users are fine with it, it is better to use plain distribute volume
rather than replication with quorum as 'auto'. What are your
thoughts on the
matter? Please guide us in the right direction.

Pranith





___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] gluster-nagios-common missing dependencies?

[Glomski, Patrick]

Just an FYI, I installed gluster-nagios-common on a CentOS7 machine
(started with minimal server, so there wasn't much there) to monitor the
bricks in a gluster array. python-ethtool seems to be missing as a
dependency in the rpm spec:

import glusternagios.glustercli as gcli
>   File "/usr/lib64/python2.7/site-packages/glusternagios/glustercli.py",
> line 21, in 
> import ethtool
>

I was using the latest 1.1 version:

http://download.gluster.org/pub/gluster/glusterfs-nagios/1.1.0/EPEL.repo/gluster-nagios-110-epel.repo

Regards,
Patrick
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] REST API authentication: JWT - Shared Token vs Shared Secret

[Luis Pabon]

For shared secret, create a simple 10 line python program. Here is an example: 
https://github.com/heketi/heketi/wiki/API#example

- Luis

- Original Message -
From: "Aravinda" 
To: "Kaushal M" 
Cc: "Luis Pabon" , "Kanagaraj Mayilsamy" 
, "Gluster Devel" , "Kaushal 
Madappa" 
Sent: Thursday, March 3, 2016 11:04:17 PM
Subject: Re: [Gluster-devel] REST API authentication: JWT - Shared Token vs 
Shared Secret


regards
Aravinda

On 03/03/2016 05:58 PM, Kaushal M wrote:
> On Thu, Mar 3, 2016 at 2:39 PM, Aravinda  wrote:
>> Thanks.
>>
>> We can use Shared secret if https requirement can be completely
>> avoided. I am not sure how to use same SSL certificates in all the
>> nodes of the Cluster.(REST API server patch set 2 was written based on
>> shared secret method based on custom HMAC signing
>> http://review.gluster.org/#/c/13214/2/in_progress/management_rest_api.md)
>>
>> Listing the steps involved in each side with both the
>> approaches. (Skipping Register steps since it is common to both)
>>
>> Shared Token:
>> -
>> Client side:
>> 1. Add saved token Authorization header and initiate a REST call.
>> 2. If UnAuthorized, call /token and get access_token again and repeat
>> the step 1
>>
>> Server side:
>> 1. Verify JWT using the Server's secret.
> You forgot the part where server generates the token. :)
Oh Yes. I missed that step :)
>
>>
>> Shared Secret:
>> --
>> Client side:
>> 1. Hash the Method + URL + Params and include in qsh claim of JWT
>> 2. Using shared secret, create JWT.
>> 3. Add previously generated JWT in Authorization header and initiate
>> REST call
>>
>> Server side:
>> 1. Recalculate the hash using same details (Method + URL + Params) and
>> verify with received qsh
>> 2. Do not trust any claims, validate against the values stored in
>> Server(role/group/capabilities)
>> 3. Verify JWT using the shared secret
>>
> Anyways, I'm still not sure which of the two approaches I like better.
> My google research on this topic (ReST api authentication) led to many
> results which followed a token approach.
> This causes me to lean slightly towards shared tokens.
Yeah, Shared token is widely used method. But https is mandatory to 
avoid URL tampering. But using SSL certs in multi node setup is 
tricky.(Using same cert files across all nodes)

Shared secret approach it is difficult to test using curl or 
Postman(https://www.getpostman.com/)

>
> Since, I can't decide, I plan to write down workflows involved for
> both and try to compare them that way.
> It would probably help arrive at a decision. I'll try to share this
> ASAP (probably this weekend).
Thanks.
>
>> regards
>> Aravinda
>>
>>
>> On 03/03/2016 11:49 AM, Luis Pabon wrote:
>>> Hi Aravinda,
>>> Very good summary.  I would like to rephrase a few parts.
>>>
>>> On the shared token approach, the disadvantage is that the server will be
>>> more complicated (not *really* complicated, just more than the shared
>>> token), because it would need a login mechanism.  Server would have to both
>>> authenticate and authorize the user.  Once this has occurred a token with an
>>> expiration date can be handed back to the caller.
>>>
>>> On the shared secret approach, I do not consider the client creating a JWT
>>> a disadvantage (unless you are doing it in C), it is pretty trivial for
>>> programs written in Python, Go, Javascript etc to create a JWT on each call.
>>>
>>> - Luis
>>>
>>> - Original Message -
>>> From: "Aravinda" 
>>> To: "Gluster Devel" 
>>> Cc: "Kaushal Madappa" , "Atin Mukherjee"
>>> , "Luis Pabon" ,
>>> kmayi...@redhat.com, "Prashanth Pai" 
>>> Sent: Wednesday, March 2, 2016 1:53:00 AM
>>> Subject: REST API authentication: JWT - Shared Token vs Shared Secret
>>>
>>> Hi,
>>>
>>> For Gluster REST project we are planning to use JSON Web Token for
>>> authentication. There are two approaches to use JWT, please help us to
>>> evaluate between these two options.
>>>
>>> http://jwt.io/
>>>
>>> For both approach, user/app will register with Username and Secret.
>>>
>>> Shared Token Approach:(Default as per JWT website
>>> http://jwt.io/introduction/)
>>>
>>> --
>>> Server will generate JWT with pre-configured expiry once user login to
>>> server by providing Username and Secret. Secret is encrypted and
>>> stored in Server. Clients should include that JWT in all requests.
>>>
>>> Advantageous:
>>> 1. Clients need not worry anything about JWT signing.
>>> 2. Single secret at server side can be used for all token verification.
>>> 3. This is a stateless authentication mechanism as the user state is
>>>   never saved in the server 

Re: [Gluster-devel] Is anyone paying attention to centos regressions on jenkins?

[Jeff Darcy]

- Original Message -
> There are a handful of centos regressions that have been running for over
> eight hours.
> 
> I don't know if that's contributing to the short backlog of centos
> regressions waiting to run.

I'm going to kill these in a moment, but here's more specific info
in case anyone wants to take a look.

  18780, 9:58pm
  79b6474 dht:remember locked subvol and send unlock to the same
  Running tests in file ./tests/basic/tier/frequency-counters.t

  18782, 11:26pm
  717e9e8 Cli/tier: separating services from cold bricks in xml
  Running tests in file ./tests/basic/tier/tier-file-create.t

  18783, 11:38pm
  633a05e vagrant-tests: Fix CFLAGS not passed to configure error
  Running tests in file ./tests/basic/tier/tier-file-create.t

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Is anyone paying attention to centos regressions on jenkins?

[Atin Mukherjee]

I think Raghavendra Talur is already aware of the problem and he sent a
mail on devel to disable regression trigger on code review +2 or verified
+1.

-Atin
Sent from one plus one
On 04-Mar-2016 6:49 pm, "Kaleb Keithley"  wrote:

> There are a handful of centos regressions that have been running for over
> eight hours.
>
> I don't know if that's contributing to the short backlog of centos
> regressions waiting to run.
>
> --
>
> Kaleb
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] WORM/Retention Feature - 04/03/2016

[Karthik Subrahmanya]

Hi all,

This week's status:

-The implementation of buffered way of storing the WORM-Retention profile:
In this way we will have more space for storing additional profile entries
for a file, for which the requirements may arise in the future.

-Tested the program with distributed and replicated type of volumes.

-Working on handling the utime changes for the WORM-Retained files:
Based on the "Mode of Retention" (Relax/Enterprise) I'm trying to handle
the utime changes.
  -If the mode is set to "Relax", then it allows to both increase
   and decrease the retention time of a WORM-Retained file.
  -If it is "Enterprise" then it allows only to increase the retention time
   of a WORM-Retained file.

Plan for next week:
-Complete the utime handling task
-Implementing the "auto-commit" feature
-Writing a blog on WORM/Retention Semantics
-Addressing the review comments by Prashanth Pai  on the
 WORM/Retention Semantics specs document

Current work:
POC: http://review.gluster.org/#/c/13429/
Spec: http://review.gluster.org/13538
Feature page: 
http://www.gluster.org/community/documentation/index.php/Features/gluster_compliance_archive

Your valuable reviews and suggestions are most welcome

Thanks & Regards,
Karthik Subrahmanya
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Is anyone paying attention to centos regressions on jenkins?

[Kaleb Keithley]

There are a handful of centos regressions that have been running for over eight 
hours.

I don't know if that's contributing to the short backlog of centos regressions 
waiting to run.

--

Kaleb
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Query on healing process

[Ravishankar N]

On 03/04/2016 06:23 PM, ABHISHEK PALIWAL wrote:



Ok, just to confirm, glusterd  and other brick processes are
running after this node rebooted?
When you run the above command, you need to check
/var/log/glusterfs/glfsheal-volname.log logs errros. Setting
client-log-level to DEBUG would give you a more verbose message

Yes, glusterd and other brick processes running fine. I have check the 
/var/log/glusterfs/glfsheal-volname.log file without the log-level= 
DEBUG. Here is the logs from that file


[2016-03-02 13:51:39.059440] I [MSGID: 101190] 
[event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started 
thread with index 1
[2016-03-02 13:51:39.072172] W [MSGID: 101012] 
[common-utils.c:2776:gf_get_reserved_ports] 0-glusterfs: could not 
open the file /proc/sys/net/ipv4/ip_local_reserved_ports for getting 
reserved ports info [No such file or directory]
[2016-03-02 13:51:39.072228] W [MSGID: 101081] 
[common-utils.c:2810:gf_process_reserved_ports] 0-glusterfs: Not able 
to get reserved ports, hence there is a possibility that glusterfs may 
consume reserved port
[2016-03-02 13:51:39.072583] E [socket.c:2278:socket_connect_finish] 
0-gfapi: connection to 127.0.0.1:24007  failed 
(Connection refused)


Not sure why ^^ occurs. You could try flushing iptables (iptables -F), 
restart glusterd and run the heal info command again .


[2016-03-02 13:51:39.072663] E [MSGID: 104024] 
[glfs-mgmt.c:738:mgmt_rpc_notify] 0-glfs-mgmt: failed to connect with 
remote-host: localhost (Transport endpoint is not connected) 
[Transport endpoint is not connected]
[2016-03-02 13:51:39.072700] I [MSGID: 104025] 
[glfs-mgmt.c:744:mgmt_rpc_notify] 0-glfs-mgmt: Exhausted all volfile 
servers [Transport endpoint is not connected]



# gluster volume heal c_glusterfs info split-brain
c_glusterfs: Not able to fetch volfile from glusterd
Volume heal failed.





And based on the your observation I understood that this is not
the problem of split-brain but *is there any way through which
can find out the file which is not in split-brain as well as not
in sync?*


`gluster volume heal c_glusterfs info split-brain`  should give
you files that need heal.



Sorry  I meant 'gluster volume heal c_glusterfs info' should give you 
the files that need heal and 'gluster volume heal c_glusterfs info 
split-brain' the list of files in split-brain.
The commands are detailed in 
https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md




I have run "gluster volume heal c_glusterfs info split-brain" command 
but it is not showing that file which is out of sync that is the issue 
file is not in sync on both of the brick and split-brain is not 
showing that command in output for heal required.


Thats is why I am asking that is there any command other than this 
split brain command so that I can find out the files those are 
required the heal operation but not displayed in the output of 
"gluster volume heal c_glusterfs info split-brain" command.








___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Warning while buidling "master"

[Joseph Fernandes]

Hi All,

There is a warning while building master code 

Making install in fdl
Making install in src
  CC   fdl.lo
  CCLD fdl.la
  CC   logdump.o
  CC   libfdl.o
  CCLD gf_logdump
  CC   recon.o
  CC   librecon.o
librecon.c: In function ‘fdl_replay_rename’:
librecon.c:158:9: warning: ‘xdata’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (xdata);
 ^
librecon.c: In function ‘fdl_replay_setxattr’:
librecon.c:275:9: warning: ‘xdata’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (xdata);
 ^
librecon.c: In function ‘fdl_replay_fsetxattr’:
librecon.c:495:9: warning: ‘xdata’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (xdata);
 ^
librecon.c:492:9: warning: ‘dict’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (dict);
 ^
librecon.c: In function ‘fdl_replay_fremovexattr’:
librecon.c:587:9: warning: ‘xdata’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (xdata);
 ^
librecon.c: In function ‘fdl_replay_xattrop’:
librecon.c:704:9: warning: ‘xdata’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (xdata);
 ^
librecon.c: In function ‘fdl_replay_create’:
librecon.c:827:9: warning: ‘xdata’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (xdata);
 ^
librecon.c: In function ‘fdl_replay_discard’:
librecon.c:916:9: warning: ‘xdata’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (xdata);
 ^
librecon.c: In function ‘fdl_replay_writev’:
librecon.c:1127:9: warning: ‘xdata’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (xdata);
 ^
librecon.c:1124:9: warning: ‘iobref’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 iobref_unref (iobref);
 ^
librecon.c: In function ‘fdl_replay_fallocate’:
librecon.c:1310:9: warning: ‘xdata’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (xdata);
 ^
librecon.c: In function ‘fdl_replay_zerofill’:
librecon.c:1584:9: warning: ‘xdata’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (xdata);
 ^
librecon.c: In function ‘fdl_replay_link’:
librecon.c:1690:9: warning: ‘xdata’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (xdata);
 ^
librecon.c: In function ‘fdl_replay_fxattrop’:
librecon.c:1810:9: warning: ‘xdata’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (xdata);
 ^
librecon.c:1807:9: warning: ‘dict’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (dict);
 ^
librecon.c: In function ‘fdl_replay_ftruncate’:
librecon.c:1896:9: warning: ‘xdata’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (xdata);
 ^
librecon.c: In function ‘fdl_replay_fsetattr’:
librecon.c:2194:9: warning: ‘xdata’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (xdata);
 ^
librecon.c: In function ‘fdl_replay_removexattr’:
librecon.c:2283:9: warning: ‘xdata’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
 dict_unref (xdata);
 ^
  CCLD gf_recon
 /usr/bin/mkdir -p '/usr/local/sbin'
  /bin/sh ../../../../libtool --quiet  --mode=install /usr/bin/install -c 
gf_logdump gf_recon '/usr/local/sbin'
 /usr/bin/mkdir -p '/usr/local/lib/glusterfs/3.8dev/xlator/experimental'
 /bin/sh ../../../../libtool --quiet  --mode=install /usr/bin/install -c   
fdl.la '/usr/local/lib/glusterfs/3.8dev/xlator/experimental'
libtool: install: warning: relinking `fdl.la'



Regards,
Joe
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-infra] Code-Review+2 and Verified+1 cause multiple retriggers on Jenkins

[Raghavendra Talur]

On Thu, Feb 4, 2016 at 7:13 PM, Niels de Vos  wrote:

> On Thu, Feb 04, 2016 at 04:15:16PM +0530, Raghavendra Talur wrote:
> > On Thu, Feb 4, 2016 at 4:13 PM, Niels de Vos  wrote:
> >
> > > On Thu, Feb 04, 2016 at 03:34:05PM +0530, Raghavendra Talur wrote:
> > > > Hi,
> > > >
> > > > We recently changed the jenkins builds to be triggered on the
> following
> > > > triggers.
> > > >
> > > > 1. Verified+1
> > > > 2. Code-review+2
> > > > 3. recheck (netbsd|centos|smoke)
> > > >
> > > > There is a bug in 1 and 2.
> > > >
> > > > Multiple triggers of 1 or 2 would result in re-runs even when not
> > > intended.
> > > >
> > > > I would like to replace 1 and 2 with a comment "run-all-regression"
> or
> > > > something like that.
> > > > Thoughts?
> > >
> > > Maybe starting regressions on Code-Review +1 (or +2) only?
> > >
> >
> > Multiple code-reviews would do multiple triggers. Won't work.
>
> How can we make this to work, without the need of providing magic
> comments?
>

I investigated but couldn't find a way to make it work. Discussed with
Kaushal and we feel it should be ok to go with a "check all" comment for
initial regression run and deprecate Code-Review+2 and Verified+1 triggers.

I would like to go ahead and do it as the build queue is increasing again
just because of Code-Review+2's given just before a patch is merged; they
don't serve any purpose.


>
> Niels
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Default quorum for 2 way replication

[Pranith Kumar Karampuri]

On 03/04/2016 05:47 PM, Bipin Kunal wrote:

HI Pranith,

Thanks for starting this mail thread.

Looking from a user perspective most important is to get a "good copy"
of data.  I agree that people use replication for HA but having stale
data with HA will not have any value.
So I will suggest to make auto quorum as default configuration even
for 2-way replication.

If user is willing to lose data at the cost of HA, he always have
option disable it. But default preference should be data and its
integrity.


That is the point. There is an illusion of choice between Data integrity 
and HA. But we are not *really* giving HA, are we? HA will be there only 
if second brick in the replica pair goes down. In your typical 
deployment, we can't really give any guarantees about what brick will go 
down when. So I am not sure if we can consider it as HA. But I would 
love to hear what others have to say about this as well. If majority of 
users say they need it to be auto, you will definitely see a patch :-).


Pranith


Thanks,
Bipin Kunal

On Fri, Mar 4, 2016 at 5:43 PM, Ravishankar N  wrote:

On 03/04/2016 05:26 PM, Pranith Kumar Karampuri wrote:

hi,
  So far default quorum for 2-way replication is 'none' (i.e.
files/directories may go into split-brain) and for 3-way replication and
arbiter based replication it is 'auto' (files/directories won't go into
split-brain). There are requests to make default as 'auto' for 2-way
replication as well. The line of reasoning is that people value data
integrity (files not going into split-brain) more than HA (operation of
mount even when bricks go down). And admins should explicitly change it to
'none' when they are fine with split-brains in 2-way replication. We were
wondering if you have any inputs about what is a sane default for 2-way
replication.

I like the default to be 'none'. Reason: If we have 'auto' as quorum for
2-way replication and first brick dies, there is no HA.



+1.  Quorum does not make sense when there are only 2 parties. There is no
majority voting. Arbiter volumes are a better option.
If someone wants some background, please see 'Client quorum' and 'Replica 2
and Replica 3 volumes' section of
http://gluster.readthedocs.org/en/latest/Administrator%20Guide/arbiter-volumes-and-quorum/

-Ravi


If users are fine with it, it is better to use plain distribute volume
rather than replication with quorum as 'auto'. What are your thoughts on the
matter? Please guide us in the right direction.

Pranith





___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Cores generated with ./tests/geo-rep/georep-basic-dr-tarssh.t

[Soumya Koduri]

Hi Raghavendra,

On 03/04/2016 03:09 PM, Raghavendra G wrote:



On Fri, Mar 4, 2016 at 2:02 PM, Raghavendra G > wrote:



On Thu, Mar 3, 2016 at 6:26 PM, Kotresh Hiremath Ravishankar
> wrote:

Hi,

Yes, with this patch we need not set conn->trans to NULL in
rpc_clnt_disable


While [1] fixes the crash, things can be improved in the way how
changelog is using rpc.

1. In the current code, there is an rpc_clnt object leak during
disconnect event.
2. Also, freed "mydata" of changelog is still associated with
rpc_clnt object (corollary of 1), though change log might not get
any events with "mydata" (as connection is dead).

I've discussed with Kotresh about changes needed, offline. So,
following are the action items.
1. Soumya's patch [2] is valid and is needed for 3.7 branch too.
2. [2] can be accepted. However, someone might want to re-use an rpc
object after disabling it, like introducing a new api
rpc_clnt_enable_again (though no of such use-cases is very less).
But [2] doesn't allow it. The point is as long as rpc-clnt object is
alive, transport object is alive (though disconnected) and we can
re-use it. So, I would prefer not to accept it.


[2] will be accepted now.

The link mentioned seems to be invalid. Just to be clear, we have 3 
patches in question here -


[1] Original patch (merged in master but not in release-3.7) 
http://review.gluster.org/13456


There are two patches proposed to fix the regression
[2] http://review.gluster.org/#/c/13587/
[3] http://review.gluster.org/#/c/13592/

Since the patch by Kotresh [3] completely fixes the regression, we have 
decided to abandon [2]. In addition, since these fixes look very 
intricate, till we are sure that the code is stable, we thought it may 
be better not to back-port these patches to stable release branch.


Please confirm if you are implying to revive [2] and  back-port all 
these 3 patches to release-3.7 branch?


Thanks,
Soumya


3. Kotresh will work on new changes to make sure changelog makes
correct use of rpc-clnt.

[1] http://review.gluster.org/#/c/13592
[2] http://review.gluster.org/#/c/1359

regards,
Raghavendra.


Thanks and Regards,
Kotresh H R

- Original Message -
> From: "Soumya Koduri" >
> To: "Kotresh Hiremath Ravishankar" >, "Raghavendra
G" >
> Cc: "Gluster Devel" >
 > Sent: Thursday, March 3, 2016 5:06:00 PM
 > Subject: Re: [Gluster-devel] Cores generated with
./tests/geo-rep/georep-basic-dr-tarssh.t
 >
 >
 >
 > On 03/03/2016 04:58 PM, Kotresh Hiremath Ravishankar wrote:
 > > [Replying on top of my own reply]
 > >
 > > Hi,
 > >
 > > I have submitted the below patch [1] to avoid the issue of
 > > 'rpc_clnt_submit'
 > > getting reconnected. But it won't take care of memory leak
problem you were
 > > trying to fix. That we have to carefully go through all
cases and fix it.
 > > Please have a look at it.
 > >
 > Looks good. IIUC, with this patch, we need not set
conn->trans to NULL
 > in 'rpc_clnt_disable()'. Right? If yes, then it takes care of
memleak as
 > the transport object shall then get freed as part of
 > 'rpc_clnt_trigger_destroy'.
 >
 >
 > > http://review.gluster.org/#/c/13592/
 > >
 > > Thanks and Regards,
 > > Kotresh H R
 > >
 > > - Original Message -
 > >> From: "Kotresh Hiremath Ravishankar" >
 > >> To: "Soumya Koduri" >
 > >> Cc: "Raghavendra G" >, "Gluster Devel"
 > >> >
 > >> Sent: Thursday, March 3, 2016 3:39:11 PM
 > >> Subject: Re: [Gluster-devel] Cores generated with
 > >> ./tests/geo-rep/georep-basic-dr-tarssh.t
 > >>
 > >> Hi Soumya,
 > >>
 > >> I tested the lastes patch [2] on master where your
previous patch [1] in
 > >> merged.
 > >> I see crashes at different places.
 > >>
 > >> 1. If there are code paths that are holding rpc object
without taking ref
 > >> on
 > >> it, all those
 > >> code path will crash on invoking rpc submit on that

Re: [Gluster-devel] Default quorum for 2 way replication

[Bipin Kunal]

HI Pranith,

Thanks for starting this mail thread.

Looking from a user perspective most important is to get a "good copy"
of data.  I agree that people use replication for HA but having stale
data with HA will not have any value.
So I will suggest to make auto quorum as default configuration even
for 2-way replication.

If user is willing to lose data at the cost of HA, he always have
option disable it. But default preference should be data and its
integrity.

Thanks,
Bipin Kunal

On Fri, Mar 4, 2016 at 5:43 PM, Ravishankar N  wrote:
> On 03/04/2016 05:26 PM, Pranith Kumar Karampuri wrote:
>>
>> hi,
>>  So far default quorum for 2-way replication is 'none' (i.e.
>> files/directories may go into split-brain) and for 3-way replication and
>> arbiter based replication it is 'auto' (files/directories won't go into
>> split-brain). There are requests to make default as 'auto' for 2-way
>> replication as well. The line of reasoning is that people value data
>> integrity (files not going into split-brain) more than HA (operation of
>> mount even when bricks go down). And admins should explicitly change it to
>> 'none' when they are fine with split-brains in 2-way replication. We were
>> wondering if you have any inputs about what is a sane default for 2-way
>> replication.
>>
>> I like the default to be 'none'. Reason: If we have 'auto' as quorum for
>> 2-way replication and first brick dies, there is no HA.
>
>
>
> +1.  Quorum does not make sense when there are only 2 parties. There is no
> majority voting. Arbiter volumes are a better option.
> If someone wants some background, please see 'Client quorum' and 'Replica 2
> and Replica 3 volumes' section of
> http://gluster.readthedocs.org/en/latest/Administrator%20Guide/arbiter-volumes-and-quorum/
>
> -Ravi
>
>> If users are fine with it, it is better to use plain distribute volume
>> rather than replication with quorum as 'auto'. What are your thoughts on the
>> matter? Please guide us in the right direction.
>>
>> Pranith
>
>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Default quorum for 2 way replication

[Ravishankar N]

On 03/04/2016 05:26 PM, Pranith Kumar Karampuri wrote:

hi,
 So far default quorum for 2-way replication is 'none' (i.e. 
files/directories may go into split-brain) and for 3-way replication 
and arbiter based replication it is 'auto' (files/directories won't go 
into split-brain). There are requests to make default as 'auto' for 
2-way replication as well. The line of reasoning is that people value 
data integrity (files not going into split-brain) more than HA 
(operation of mount even when bricks go down). And admins should 
explicitly change it to 'none' when they are fine with split-brains in 
2-way replication. We were wondering if you have any inputs about what 
is a sane default for 2-way replication.


I like the default to be 'none'. Reason: If we have 'auto' as quorum 
for 2-way replication and first brick dies, there is no HA.



+1.  Quorum does not make sense when there are only 2 parties. There is 
no majority voting. Arbiter volumes are a better option.
If someone wants some background, please see 'Client quorum' and 
'Replica 2 and Replica 3 volumes' section of 
http://gluster.readthedocs.org/en/latest/Administrator%20Guide/arbiter-volumes-and-quorum/ 



-Ravi
If users are fine with it, it is better to use plain distribute volume 
rather than replication with quorum as 'auto'. What are your thoughts 
on the matter? Please guide us in the right direction.


Pranith



___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Query on healing process

[Ravishankar N]

On 03/04/2016 12:10 PM, ABHISHEK PALIWAL wrote:

Hi Ravi,

3. On the rebooted node, do you have ssl enabled by any chance? There 
is a bug for "Not able to fetch volfile' when ssl is enabled: 
https://bugzilla.redhat.com/show_bug.cgi?id=1258931


-> I have checked but ssl is disabled but still getting these errors

# gluster volume heal c_glusterfs info
c_glusterfs: Not able to fetch volfile from glusterd
Volume heal failed.



Ok, just to confirm, glusterd  and other brick processes are running 
after this node rebooted?
When you run the above command, you need to check 
/var/log/glusterfs/glfsheal-volname.log logs errros. Setting 
client-log-level to DEBUG would give you a more verbose message



# gluster volume heal c_glusterfs info split-brain
c_glusterfs: Not able to fetch volfile from glusterd
Volume heal failed.





And based on the your observation I understood that this is not the 
problem of split-brain but *is there any way through which can find 
out the file which is not in split-brain as well as not in sync?*


`gluster volume heal c_glusterfs info split-brain` should give you files 
that need heal.




# getfattr -m . -d -e hex 
/opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml

getfattr: Removing leading '/' from absolute path names
# file: 
opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml

trusted.afr.c_glusterfs-client-0=0x
trusted.afr.c_glusterfs-client-2=0x
trusted.afr.c_glusterfs-client-4=0x
trusted.afr.c_glusterfs-client-6=0x
trusted.afr.c_glusterfs-client-8=*0x0006**//because client8 
is the latest client in our case and starting 8 digits **

*
*0006are saying like there is something in changelog data.
*
trusted.afr.dirty=0x
trusted.bit-rot.version=0x001356d86c0c000217fd
trusted.gfid=0x9f5e354ecfda40149ddce7d5ffe760ae

# lhsh 002500 getfattr -m . -d -e hex 
/opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml

getfattr: Removing leading '/' from absolute path names
# file: 
opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml
trusted.afr.c_glusterfs-client-1=*0x**// and 
here we can say that there is no split brain but the file is out of sync*

trusted.afr.dirty=0x
trusted.bit-rot.version=0x001156d86c290005735c
trusted.gfid=0x9f5e354ecfda40149ddce7d5ffe760ae

# gluster volume info

Volume Name: c_glusterfs
Type: Replicate
Volume ID: c6a61455-d378-48bf-ad40-7a3ce897fc9c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.32.0.48:/opt/lvmdir/c2/brick
Brick2: 10.32.1.144:/opt/lvmdir/c2/brick
Options Reconfigured:
performance.readdir-ahead: on
network.ping-timeout: 4
nfs.disable: on


# gluster volume info

Volume Name: c_glusterfs
Type: Replicate
Volume ID: c6a61455-d378-48bf-ad40-7a3ce897fc9c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.32.0.48:/opt/lvmdir/c2/brick
Brick2: 10.32.1.144:/opt/lvmdir/c2/brick
Options Reconfigured:
performance.readdir-ahead: on
network.ping-timeout: 4
nfs.disable: on

# gluster --version
glusterfs 3.7.8 built on Feb 17 2016 07:49:49
Repository revision: git://git.gluster.com/glusterfs.git 

Copyright (c) 2006-2011 Gluster Inc. > 


GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU 
General Public License.

# gluster volume heal info heal-failed
Usage: volume heal  [enable | disable | full |statistics 
[heal-count [replica ]] |info [healed | 
heal-failed | split-brain] |split-brain {bigger-file  
|source-brick  []}]

# gluster volume heal c_glusterfs info heal-failed
Command not supported. Please use "gluster volume heal c_glusterfs 
info" and logs to find the heal information.

# lhsh 002500
 ___  _ _  _ __   _ _ _ _ _
 |   |_] |_]  ||   | \  | | |  \___/
 |_  |   |  |_ __|__ |  \_| |_| _/   \_

002500> gluster --version
glusterfs 3.7.8 built on Feb 17 2016 07:49:49
Repository revision: git://git.gluster.com/glusterfs.git 

Copyright (c) 2006-2011 Gluster Inc. > 


GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU 
General Public License.

002500>

[Gluster-devel] Default quorum for 2 way replication

[Pranith Kumar Karampuri]

hi,
 So far default quorum for 2-way replication is 'none' (i.e. 
files/directories may go into split-brain) and for 3-way replication and 
arbiter based replication it is 'auto' (files/directories won't go into 
split-brain). There are requests to make default as 'auto' for 2-way 
replication as well. The line of reasoning is that people value data 
integrity (files not going into split-brain) more than HA (operation of 
mount even when bricks go down). And admins should explicitly change it 
to 'none' when they are fine with split-brains in 2-way replication. We 
were wondering if you have any inputs about what is a sane default for 
2-way replication.


I like the default to be 'none'. Reason: If we have 'auto' as quorum for 
2-way replication and first brick dies, there is no HA. If users are 
fine with it, it is better to use plain distribute volume rather than 
replication with quorum as 'auto'. What are your thoughts on the matter? 
Please guide us in the right direction.


Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Bugs with incorrect status

[ndevos]

1008839 (mainline) POST: Certain blocked entry lock info not retained after the 
lock is granted
  [master] Ie37837 features/locks : Certain blocked entry lock info not 
retained after the lock is granted (ABANDONED)
  ** ata...@redhat.com: Bug 1008839 is in POST, but all changes have been 
abandoned **

1062437 (mainline) POST: stripe does not work with empty xlator
  [master] I778699 stripe: fix FIRST_CHILD checks to be more general (ABANDONED)
  ** jda...@redhat.com: Bug 1062437 is in POST, but all changes have been 
abandoned **

1074947 (mainline) ON_QA: add option to bulld rpm without server
  [master] Iaa1498 build: add option to bulld rpm without server (NEW)
  ** b...@gluster.org: Bug 1074947 should be in POST, change Iaa1498 under 
review **

1089642 (mainline) POST: Quotad doesn't load io-stats xlator, which implies 
none of the logging options have any effect on it.
  [master] Iccc033 glusterd: add io-stats to all quotad's sub-graphs (ABANDONED)
  ** spa...@redhat.com: Bug 1089642 is in POST, but all changes have been 
abandoned **

1092414 (mainline) ASSIGNED: Disable NFS by default
  [master] If25502 [WIP] glusterd: make Gluster/NFS an optional component (NEW)
  ** nde...@redhat.com: Bug 1092414 should be in POST, change If25502 under 
review **

1093768 (3.5.0) POST: Comment typo in gf-history.changelog.c
  ** kschi...@redhat.com: No change posted, but bug 1093768 is in POST **

1094478 (3.5.0) POST: Bad macro in changelog-misc.h
  ** kschi...@redhat.com: No change posted, but bug 1094478 is in POST **

1099294 (3.5.0) POST: Incorrect error message in 
/features/changelog/lib/src/gf-history-changelog.c
  ** kschi...@redhat.com: No change posted, but bug 1099294 is in POST **

1099460 (3.5.0) NEW: file locks are not released within an acceptable time when 
a fuse-client uncleanly disconnects
  [release-3.5] I5e5f54 socket: use TCP_USER_TIMEOUT to detect client failures 
quicker (NEW)
  ** nde...@redhat.com: Bug 1099460 should be in POST, change I5e5f54 under 
review **

1099683 (3.5.0) POST: Silent error from call to realpath in 
features/changelog/lib/src/gf-history-changelog.c
  ** vshan...@redhat.com: No change posted, but bug 1099683 is in POST **

110 (mainline) MODIFIED: [RFE] Add regression tests for the component 
geo-replication
  [master] Ie27848 tests/geo-rep: Automated configuration for geo-rep 
regression. (NEW)
  [master] I9c9ae8 geo-rep: Regression tests improvements (ABANDONED)
  [master] I433dd8 Geo-rep: Adding regression tests for geo-rep (MERGED)
  [master] Ife8201 Geo-rep: Adding regression tests for geo-rep (ABANDONED)
  ** khire...@redhat.com: Bug 110 should be in POST, change Ie27848 under 
review **

020 (3.5.0) POST: Unused code changelog_entry_length
  ** kschi...@redhat.com: No change posted, but bug 020 is in POST **

031 (3.5.0) POST: CHANGELOG_FILL_HTIME_DIR macro fills buffer without size 
limits
  ** kschi...@redhat.com: No change posted, but bug 031 is in POST **

1114415 (mainline) MODIFIED: There is no way to monitor if the healing is 
successful when the brick is erased
  ** pkara...@redhat.com: No change posted, but bug 1114415 is in MODIFIED **

1116714 (3.5.0) POST: indices/xattrop directory contains stale entries
  [release-3.5] I470cf8 afr : Added xdata flags to indicate probable existence 
of stale index. (ABANDONED)
  ** ata...@redhat.com: Bug 1116714 is in POST, but all changes have been 
abandoned **

1117886 (mainline) MODIFIED: Gluster not resolving hosts with IPv6 only lookups
  [master] Icbaa3c Reinstate ipv6 support (NEW)
  [master] Iebc96e glusterd: Bug fixes for IPv6 support (MERGED)
  ** nithind1...@yahoo.in: Bug 1117886 should be in POST, change Icbaa3c under 
review **

1122120 (3.5.1) MODIFIED: Bricks crashing after disable and re-enabled quota on 
a volume
  ** b...@gluster.org: No change posted, but bug 1122120 is in MODIFIED **

1131846 (mainline) POST: remove-brick - once you stop remove-brick using stop 
command,  status says '  failed: remove-brick not started.'
  ** gg...@redhat.com: No change posted, but bug 1131846 is in POST **

1132074 (mainline) POST: Document steps to perform for replace-brick
  [master] Ic7292b doc: Steps for Replacing brick in gluster volume (ABANDONED)
  ** pkara...@redhat.com: Bug 1132074 is in POST, but all changes have been 
abandoned **

1134305 (mainline) POST: rpc actor failed to complete successfully messages in 
Glusterd
  [master] I094516 protocol/client: Add explicit states for connection sequence 
(ABANDONED)
  ** pkara...@redhat.com: Bug 1134305 is in POST, but all changes have been 
abandoned **

1142423 (mainline) MODIFIED: [DHT-REBALANCE]-DataLoss: The data appended to a 
file during its migration will be lost once the migration is done
  [master] I044a83 cluster/dht Use additional dst_info in inode_ctx (NEW)
  [master] I5c8810 cluster/dht: Fix stale subvol cache for files under 
migration (NEW)
  [master] I05a2e9 cluster/dht: fix incorrect dst subvol info in 

Re: [Gluster-devel] Cores generated with ./tests/geo-rep/georep-basic-dr-tarssh.t

[Raghavendra G]

On Fri, Mar 4, 2016 at 2:02 PM, Raghavendra G 
wrote:

>
>
> On Thu, Mar 3, 2016 at 6:26 PM, Kotresh Hiremath Ravishankar <
> khire...@redhat.com> wrote:
>
>> Hi,
>>
>> Yes, with this patch we need not set conn->trans to NULL in
>> rpc_clnt_disable
>>
>
> While [1] fixes the crash, things can be improved in the way how changelog
> is using rpc.
>
> 1. In the current code, there is an rpc_clnt object leak during disconnect
> event.
> 2. Also, freed "mydata" of changelog is still associated with rpc_clnt
> object (corollary of 1), though change log might not get any events with
> "mydata" (as connection is dead).
>
> I've discussed with Kotresh about changes needed, offline. So, following
> are the action items.
> 1. Soumya's patch [2] is valid and is needed for 3.7 branch too.
> 2. [2] can be accepted. However, someone might want to re-use an rpc
> object after disabling it, like introducing a new api rpc_clnt_enable_again
> (though no of such use-cases is very less). But [2] doesn't allow it. The
> point is as long as rpc-clnt object is alive, transport object is alive
> (though disconnected) and we can re-use it. So, I would prefer not to
> accept it.
>

[2] will be accepted now.


> 3. Kotresh will work on new changes to make sure changelog makes correct
> use of rpc-clnt.
>
> [1] http://review.gluster.org/#/c/13592
> [2] http://review.gluster.org/#/c/1359
>
> regards,
> Raghavendra.
>
>
>> Thanks and Regards,
>> Kotresh H R
>>
>> - Original Message -
>> > From: "Soumya Koduri" 
>> > To: "Kotresh Hiremath Ravishankar" , "Raghavendra
>> G" 
>> > Cc: "Gluster Devel" 
>> > Sent: Thursday, March 3, 2016 5:06:00 PM
>> > Subject: Re: [Gluster-devel] Cores generated with
>> ./tests/geo-rep/georep-basic-dr-tarssh.t
>> >
>> >
>> >
>> > On 03/03/2016 04:58 PM, Kotresh Hiremath Ravishankar wrote:
>> > > [Replying on top of my own reply]
>> > >
>> > > Hi,
>> > >
>> > > I have submitted the below patch [1] to avoid the issue of
>> > > 'rpc_clnt_submit'
>> > > getting reconnected. But it won't take care of memory leak problem
>> you were
>> > > trying to fix. That we have to carefully go through all cases and fix
>> it.
>> > > Please have a look at it.
>> > >
>> > Looks good. IIUC, with this patch, we need not set conn->trans to NULL
>> > in 'rpc_clnt_disable()'. Right? If yes, then it takes care of memleak as
>> > the transport object shall then get freed as part of
>> > 'rpc_clnt_trigger_destroy'.
>> >
>> >
>> > > http://review.gluster.org/#/c/13592/
>> > >
>> > > Thanks and Regards,
>> > > Kotresh H R
>> > >
>> > > - Original Message -
>> > >> From: "Kotresh Hiremath Ravishankar" 
>> > >> To: "Soumya Koduri" 
>> > >> Cc: "Raghavendra G" , "Gluster Devel"
>> > >> 
>> > >> Sent: Thursday, March 3, 2016 3:39:11 PM
>> > >> Subject: Re: [Gluster-devel] Cores generated with
>> > >> ./tests/geo-rep/georep-basic-dr-tarssh.t
>> > >>
>> > >> Hi Soumya,
>> > >>
>> > >> I tested the lastes patch [2] on master where your previous patch
>> [1] in
>> > >> merged.
>> > >> I see crashes at different places.
>> > >>
>> > >> 1. If there are code paths that are holding rpc object without
>> taking ref
>> > >> on
>> > >> it, all those
>> > >> code path will crash on invoking rpc submit on that object as rpc
>> > >> object
>> > >> would have freed
>> > >> by last unref on DISCONNECT event. I see this kind of use-case in
>> > >> chagnelog rpc code.
>> > >> Need to check on other users of rpc.
>> > Agree. We should fix all such code-paths. Since this seem to be an
>> > intricate fix, shall we take these patches only in master branch and not
>> > in 3.7 release for now till we fix all such paths as we encounter?
>> >
>> > >>
>> > >> 2. And also we need to take care of reconnect timers that are being
>> set
>> > >> and
>> > >> are re-tried to
>> > >> connect back on expiration. In those cases also, we might crash
>> as rpc
>> > >> object would have freed.
>> > Your patch addresses this..right?
>> >
>> > Thanks,
>> > Soumya
>> >
>> > >>
>> > >>
>> > >> [1] http://review.gluster.org/#/c/13507/
>> > >> [2] http://review.gluster.org/#/c/13587/
>> > >>
>> > >> Thanks and Regards,
>> > >> Kotresh H R
>> > >>
>> > >> - Original Message -
>> > >>> From: "Soumya Koduri" 
>> > >>> To: "Raghavendra G" , "Kotresh Hiremath
>> > >>> Ravishankar" 
>> > >>> Cc: "Gluster Devel" 
>> > >>> Sent: Thursday, March 3, 2016 12:24:00 PM
>> > >>> Subject: Re: [Gluster-devel] Cores generated with
>> > >>> ./tests/geo-rep/georep-basic-dr-tarssh.t
>> > >>>
>> > >>> Thanks a lot Kotresh.
>> > >>>
>> > >>> On 03/03/2016 08:47 AM, Raghavendra G wrote:
>> >  Hi Soumya,
>> > 
>> >  Can you send a fix to this 

Re: [Gluster-devel] Cores generated with ./tests/geo-rep/georep-basic-dr-tarssh.t

[Raghavendra G]

On Thu, Mar 3, 2016 at 6:26 PM, Kotresh Hiremath Ravishankar <
khire...@redhat.com> wrote:

> Hi,
>
> Yes, with this patch we need not set conn->trans to NULL in
> rpc_clnt_disable
>

While [1] fixes the crash, things can be improved in the way how changelog
is using rpc.

1. In the current code, there is an rpc_clnt object leak during disconnect
event.
2. Also, freed "mydata" of changelog is still associated with rpc_clnt
object (corollary of 1), though change log might not get any events with
"mydata" (as connection is dead).

I've discussed with Kotresh about changes needed, offline. So, following
are the action items.
1. Soumya's patch [2] is valid and is needed for 3.7 branch too.
2. [2] can be accepted. However, someone might want to re-use an rpc object
after disabling it, like introducing a new api rpc_clnt_enable_again
(though no of such use-cases is very less). But [2] doesn't allow it. The
point is as long as rpc-clnt object is alive, transport object is alive
(though disconnected) and we can re-use it. So, I would prefer not to
accept it.
3. Kotresh will work on new changes to make sure changelog makes correct
use of rpc-clnt.

[1] http://review.gluster.org/#/c/13592
[2] http://review.gluster.org/#/c/1359

regards,
Raghavendra.


> Thanks and Regards,
> Kotresh H R
>
> - Original Message -
> > From: "Soumya Koduri" 
> > To: "Kotresh Hiremath Ravishankar" , "Raghavendra
> G" 
> > Cc: "Gluster Devel" 
> > Sent: Thursday, March 3, 2016 5:06:00 PM
> > Subject: Re: [Gluster-devel] Cores generated with
> ./tests/geo-rep/georep-basic-dr-tarssh.t
> >
> >
> >
> > On 03/03/2016 04:58 PM, Kotresh Hiremath Ravishankar wrote:
> > > [Replying on top of my own reply]
> > >
> > > Hi,
> > >
> > > I have submitted the below patch [1] to avoid the issue of
> > > 'rpc_clnt_submit'
> > > getting reconnected. But it won't take care of memory leak problem you
> were
> > > trying to fix. That we have to carefully go through all cases and fix
> it.
> > > Please have a look at it.
> > >
> > Looks good. IIUC, with this patch, we need not set conn->trans to NULL
> > in 'rpc_clnt_disable()'. Right? If yes, then it takes care of memleak as
> > the transport object shall then get freed as part of
> > 'rpc_clnt_trigger_destroy'.
> >
> >
> > > http://review.gluster.org/#/c/13592/
> > >
> > > Thanks and Regards,
> > > Kotresh H R
> > >
> > > - Original Message -
> > >> From: "Kotresh Hiremath Ravishankar" 
> > >> To: "Soumya Koduri" 
> > >> Cc: "Raghavendra G" , "Gluster Devel"
> > >> 
> > >> Sent: Thursday, March 3, 2016 3:39:11 PM
> > >> Subject: Re: [Gluster-devel] Cores generated with
> > >> ./tests/geo-rep/georep-basic-dr-tarssh.t
> > >>
> > >> Hi Soumya,
> > >>
> > >> I tested the lastes patch [2] on master where your previous patch [1]
> in
> > >> merged.
> > >> I see crashes at different places.
> > >>
> > >> 1. If there are code paths that are holding rpc object without taking
> ref
> > >> on
> > >> it, all those
> > >> code path will crash on invoking rpc submit on that object as rpc
> > >> object
> > >> would have freed
> > >> by last unref on DISCONNECT event. I see this kind of use-case in
> > >> chagnelog rpc code.
> > >> Need to check on other users of rpc.
> > Agree. We should fix all such code-paths. Since this seem to be an
> > intricate fix, shall we take these patches only in master branch and not
> > in 3.7 release for now till we fix all such paths as we encounter?
> >
> > >>
> > >> 2. And also we need to take care of reconnect timers that are being
> set
> > >> and
> > >> are re-tried to
> > >> connect back on expiration. In those cases also, we might crash
> as rpc
> > >> object would have freed.
> > Your patch addresses this..right?
> >
> > Thanks,
> > Soumya
> >
> > >>
> > >>
> > >> [1] http://review.gluster.org/#/c/13507/
> > >> [2] http://review.gluster.org/#/c/13587/
> > >>
> > >> Thanks and Regards,
> > >> Kotresh H R
> > >>
> > >> - Original Message -
> > >>> From: "Soumya Koduri" 
> > >>> To: "Raghavendra G" , "Kotresh Hiremath
> > >>> Ravishankar" 
> > >>> Cc: "Gluster Devel" 
> > >>> Sent: Thursday, March 3, 2016 12:24:00 PM
> > >>> Subject: Re: [Gluster-devel] Cores generated with
> > >>> ./tests/geo-rep/georep-basic-dr-tarssh.t
> > >>>
> > >>> Thanks a lot Kotresh.
> > >>>
> > >>> On 03/03/2016 08:47 AM, Raghavendra G wrote:
> >  Hi Soumya,
> > 
> >  Can you send a fix to this regression on upstream master too? This
> patch
> >  is merged there.
> > 
> > >>> I have submitted below patch.
> > >>>   http://review.gluster.org/#/c/13587/
> > >>>
> > >>> Kindly review the same.
> > >>>
> > >>> Thanks,
> > >>> Soumya
> > >>>
> >  regards,
> >