Re: [Gluster-users] Feedback on CLI message outputs

[James]

Haha, actually it was this guy ;)

Here is the example code (python) i'm using:

#   EXAMPLE:
#   $ gluster volume --xml info  | ./xml.py 
#   

import sys
import lxml.etree as etree

if len(sys.argv) != 2:
sys.exit(1)

t = etree.parse(sys.stdin)
r = t.getroot()
v = [x.find('value').text for x in r.findall('.//option') if
x.find('name').text == str(sys.argv[1])]
if len(v) == 1:
print v[0]
sys.exit(0)
else:   # more than one value found
sys.exit(1)

While this is somewhat ridiculous, I don't mind doing it, because it's
done, and it's the only way I know of to "get" an option value. Hint,
hint: feature request.

This will appear in my git repo shortly...

James

PS: It seems that this only shows values which are set differently than
the defaults. It would be excellent to change this approach, so that
scripts (like puppet) can reliably know if the key they're watching is
different from what it wants to set it to, without having to have a
table of the defaults) If this doesn't make sense I can re-explain...

On Mon, 2012-08-06 at 20:11 -0700, Joe Julian wrote:
> Those darned IRC people, always advising people to use undocumented
> options... Sheesh.
> 
> On 08/06/2012 07:10 PM, James wrote: 
> > On Tue, 2012-08-07 at 07:33 +0530, Kaushal M wrote:
> > > Hi James,
> > > This is the first time I've heard of someone in the community using
> > > the '--xml' flag.
> > Cool :) I'll be posting my updated code that uses this in my
> > https://github.com/purpleidea/puppet-gluster shortly. The --xml flag was
> > suggested by someone in irc, as a stable alternative to parsing the
> > normal command line output (messy).
> > 
> > > This flag has not been tested extensively, so there are bound to be
> > > some errors, inconsistencies and incompleteness. We are trying to fix
> > > it and stabilize it as much as possible. So if you do find any bugs
> > > (which you will) please report them.
> > Will do :) It seems to be working at the moment 3.3
> > 
> > James
> > 
> > > Cheers,
> > > Kaushal
> > > 
> > > On 7 Aug 2012 01:58, "James"  wrote:
> > > Hi there,
> > > 
> > > I'm currently using the --xml flag when running gluster via
> > > automatic
> > > scripts such as via puppet exec. Please keep this stable :)
> > > 
> > > Thank you!
> > > James
> > > 
> > > 
> > > On Mon, 2012-08-06 at 12:31 +0530, Amar Tumballi wrote:
> > > > Hi All,
> > > >
> > > > We are in the process of improving our log messages, and
> > > message outputs
> > > > through CLI. As a first step, we want to make sure our CLI
> > > outputs are
> > > > both meaningful and script friendly (for people who want to
> > > use the CLI
> > > > output in their scripts).
> > > >
> > > > We are taking few baby steps in this regard in upstream
> > > (master branch).
> > > > I recently sent a patch http://review.gluster.com/3753 which
> > > tries to
> > > > separate out most of the error messages by outputting it to
> > > stderr
> > > > instead of stdout.
> > > >
> > > > In the meantime, I tried to keep it script friendly too.
> > > With that patch
> > > > going in, the problem is, if any of you have written scripts
> > > which greps
> > > > for specific pattern in the CLI output, will be broken when
> > > the next
> > > > major release happens.
> > > >
> > > > So, I want your feed back on getting it right by involving
> > > you all to
> > > > make the messages more friendlier to both script writers,
> > > and also to
> > > > admins who would understand the messages properly. (ie,
> > > meaning of the
> > > > messages should be unambiguous).
> > > >
> > > > To try to see the difference in CLI outputs, you need to
> > > checkout git
> > > > repo, and build locally, as there are no releases available
> > > at the moment.
> > > >
> > > >
> > > > Thanks and Regards,
> > > > Amar
> > > > ___
> > > > Gluster-users mailing list
> > > > Gluster-users@gluster.org
> > > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> > > 
> > > 
> > > ___
> > > Gluster-users mailing list
> > > Gluster-users@gluster.org
> > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> > > 
> > > 
> > > 
> > > ___
> > > Gluster-users mailing list
> > > Gluster-users@gluster.org
> > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> ___
> Glus

Re: [Gluster-users] Feedback on CLI message outputs

[Joe Julian]

Those darned IRC people, always advising people to use undocumented 
options... Sheesh.


On 08/06/2012 07:10 PM, James wrote:

On Tue, 2012-08-07 at 07:33 +0530, Kaushal M wrote:

Hi James,
This is the first time I've heard of someone in the community using
the '--xml' flag.

Cool :) I'll be posting my updated code that uses this in my
https://github.com/purpleidea/puppet-gluster shortly. The --xml flag was
suggested by someone in irc, as a stable alternative to parsing the
normal command line output (messy).


This flag has not been tested extensively, so there are bound to be
some errors, inconsistencies and incompleteness. We are trying to fix
it and stabilize it as much as possible. So if you do find any bugs
(which you will) please report them.

Will do :) It seems to be working at the moment 3.3

James


Cheers,
Kaushal

On 7 Aug 2012 01:58, "James"  wrote:
 Hi there,

 I'm currently using the --xml flag when running gluster via
 automatic
 scripts such as via puppet exec. Please keep this stable :)

 Thank you!
 James


 On Mon, 2012-08-06 at 12:31 +0530, Amar Tumballi wrote:
 >  Hi All,
 >
 >  We are in the process of improving our log messages, and
 message outputs
 >  through CLI. As a first step, we want to make sure our CLI
 outputs are
 >  both meaningful and script friendly (for people who want to
 use the CLI
 >  output in their scripts).
 >
 >  We are taking few baby steps in this regard in upstream
 (master branch).
 >  I recently sent a patch http://review.gluster.com/3753 which
 tries to
 >  separate out most of the error messages by outputting it to
 stderr
 >  instead of stdout.
 >
 >  In the meantime, I tried to keep it script friendly too.
 With that patch
 >  going in, the problem is, if any of you have written scripts
 which greps
 >  for specific pattern in the CLI output, will be broken when
 the next
 >  major release happens.
 >
 >  So, I want your feed back on getting it right by involving
 you all to
 >  make the messages more friendlier to both script writers,
 and also to
 >  admins who would understand the messages properly. (ie,
 meaning of the
 >  messages should be unambiguous).
 >
 >  To try to see the difference in CLI outputs, you need to
 checkout git
 >  repo, and build locally, as there are no releases available
 at the moment.
 >
 >
 >  Thanks and Regards,
 >  Amar
 >  ___
 >  Gluster-users mailing list
 >  Gluster-users@gluster.org
 >  http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://gluster.org/cgi-bin/mailman/listinfo/gluster-users



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Feedback on CLI message outputs

[James]

On Tue, 2012-08-07 at 07:33 +0530, Kaushal M wrote:
> Hi James,
> This is the first time I've heard of someone in the community using
> the '--xml' flag.
Cool :) I'll be posting my updated code that uses this in my
https://github.com/purpleidea/puppet-gluster shortly. The --xml flag was
suggested by someone in irc, as a stable alternative to parsing the
normal command line output (messy).

> This flag has not been tested extensively, so there are bound to be
> some errors, inconsistencies and incompleteness. We are trying to fix
> it and stabilize it as much as possible. So if you do find any bugs
> (which you will) please report them.
Will do :) It seems to be working at the moment 3.3

James

> 
> Cheers,
> Kaushal
> 
> On 7 Aug 2012 01:58, "James"  wrote:
> Hi there,
> 
> I'm currently using the --xml flag when running gluster via
> automatic
> scripts such as via puppet exec. Please keep this stable :)
> 
> Thank you!
> James
> 
> 
> On Mon, 2012-08-06 at 12:31 +0530, Amar Tumballi wrote:
> > Hi All,
> >
> > We are in the process of improving our log messages, and
> message outputs
> > through CLI. As a first step, we want to make sure our CLI
> outputs are
> > both meaningful and script friendly (for people who want to
> use the CLI
> > output in their scripts).
> >
> > We are taking few baby steps in this regard in upstream
> (master branch).
> > I recently sent a patch http://review.gluster.com/3753 which
> tries to
> > separate out most of the error messages by outputting it to
> stderr
> > instead of stdout.
> >
> > In the meantime, I tried to keep it script friendly too.
> With that patch
> > going in, the problem is, if any of you have written scripts
> which greps
> > for specific pattern in the CLI output, will be broken when
> the next
> > major release happens.
> >
> > So, I want your feed back on getting it right by involving
> you all to
> > make the messages more friendlier to both script writers,
> and also to
> > admins who would understand the messages properly. (ie,
> meaning of the
> > messages should be unambiguous).
> >
> > To try to see the difference in CLI outputs, you need to
> checkout git
> > repo, and build locally, as there are no releases available
> at the moment.
> >
> >
> > Thanks and Regards,
> > Amar
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> 



signature.asc
Description: This is a digitally signed message part
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Feedback on CLI message outputs

[Kaushal M]

Hi James,
This is the first time I've heard of someone in the community using the
'--xml' flag. This flag has not been tested extensively, so there are bound
to be some errors, inconsistencies and incompleteness. We are trying to fix
it and stabilize it as much as possible. So if you do find any bugs (which
you will) please report them.

Cheers,
Kaushal
On 7 Aug 2012 01:58, "James"  wrote:

> Hi there,
>
> I'm currently using the --xml flag when running gluster via automatic
> scripts such as via puppet exec. Please keep this stable :)
>
> Thank you!
> James
>
>
> On Mon, 2012-08-06 at 12:31 +0530, Amar Tumballi wrote:
> > Hi All,
> >
> > We are in the process of improving our log messages, and message outputs
> > through CLI. As a first step, we want to make sure our CLI outputs are
> > both meaningful and script friendly (for people who want to use the CLI
> > output in their scripts).
> >
> > We are taking few baby steps in this regard in upstream (master branch).
> > I recently sent a patch http://review.gluster.com/3753 which tries to
> > separate out most of the error messages by outputting it to stderr
> > instead of stdout.
> >
> > In the meantime, I tried to keep it script friendly too. With that patch
> > going in, the problem is, if any of you have written scripts which greps
> > for specific pattern in the CLI output, will be broken when the next
> > major release happens.
> >
> > So, I want your feed back on getting it right by involving you all to
> > make the messages more friendlier to both script writers, and also to
> > admins who would understand the messages properly. (ie, meaning of the
> > messages should be unambiguous).
> >
> > To try to see the difference in CLI outputs, you need to checkout git
> > repo, and build locally, as there are no releases available at the
> moment.
> >
> >
> > Thanks and Regards,
> > Amar
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] EBADFD with large number of concurrent files

[Brian Candler]

I have an application where there are 48 processes, and each one has opens
1000 files (different files for all 48 processes).  They are opened onto a
distributed gluster volume, distributed between two nodes.

It works initially, but after a while, some of the processes abort. perror
prints "File descriptor in bad state" (I think this means EBADFD)

This is with glusterfs 3.3.0 under Ubuntu 12.04 (both the storage nodes and
the application servers)

Looking on the two backend bricks, each has two glusterfsd processes.  On
both bricks, the one with the lower pid has 24168 open FDs
(ls /proc//fd | wc -l), and also 1.5-2.5GB of RSS.  So it's pretty
clear that glusterfsd keeps one open file handle per file opened by the
client. That's pretty reasonable.

I don't think I'm hitting a system limit for this:

# cat /proc/sys/fs/file-max
808870

and it's clearly working for the first few minutes.  So I wonder if anyone
has any other suggestions for why EBADFD is getting returned after a while?

Thanks,

Brian.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster-users Digest, Vol 51, Issue 46

[Jeff Darcy]

On 08/06/2012 09:51 AM, John Mark Walker wrote:
> 
> 
> ---
> 
> Hi Ben,
> 
> Thanks for the expert advice.
> 
> On Fri, Aug 3, 2012 at 2:35 PM, Ben England  > wrote:
> 
> 4. Re: kernel parameters for improving gluster writes on millions of
> small writes (long) (Harry Mangalam)
> 
> Harry, You are correct, Glusterfs throughput with small write transfer
> sizes is a client-side problem, here are workarounds that at least 
> some
> applications could use.
> 
> 
> Not to be impertinent nor snarky, but why is the gluster client written in
> this way and is that a high priority for fixing?  It seems that
> caching/buffering is one of the great central truths of computer science 
> in
> general.  Is there a countering argument for not doing this?

I'd say that caching/buffering is also one of the great central *battlegrounds*
of computer science.  ;)  While they're useful and often necessary performance
enhancers, when done without proper attention to issues such as
ordering/consistency and fault handling they can also lead to some pretty awful
problems.  The GlusterFS philosophy has generally been to handle performance
issues via horizontal scaling - which works even for data sizes greater than
any cache - and be conservative about the other issues.  The consistency issue
alone could be dealt with in other ways, but most of those would require more
complexity overall and particularly more complexity and overhead on the servers
(violating another principle of moving work to the more numerous clients).

Clearly these tradeoffs could have been made differently, but I'd argue that
the additional effort necessary would have required sacrifice in other areas.
Witness e.g. Ceph, which is technically superior in several ways but *as a
direct consequence* has taken longer to mature and stabilize.  The idea of
trying to implement e.g. dynamic reconfiguration in a more stateful and complex
system makes me a bit queasy.

> As a general point, you'll find that glusterfs always (almost always?) errs on
> the side of data consistency, even if it adversely affects performance. An NFS
> client can cache because it doesn't have to worry about HA - which has to be
> implemented with other tools. With recent changes in GlusterFS code, including
> further development of server-side code, it should be possible to create some
> type of client-side caching in the near future. There are also developments in
> fuse to think about, but mostly, it has to do with glusterfs' new server code.
> Previously, all the intelligence was in the client, so data consistency on the
> client was absolutely essential. Now, with "smarter" server-side translators,
> eg. self-heal and rebalancing, this is shifting. 3.3 was the first release 
> with
> the shift in that direction, and more is coming. I know this doesn't help you
> *now*, but I wanted to give you an idea of why it is this way, and how it's
> changing going forward.

To elaborate a bit on that point, I think some of the changes are likely to
involve support for explicit and deliberate sacrifices of consistency for the
sake of performance, for users that want to make that choice.  The defaults
will probably remain fairly conservative, to support current users/workloads.
Examples might include things like TTL-based caching or asynchronous
replication implemented in a modular way, not deep changes such as an
MPFS-style cache protocol.

Apparently I'll be out west in a few weeks, with a trip up the coast preceding
the Gluster workshop at LinuxCon.  I'd be glad to discuss these issues in
person with anyone who's interested, preferably over some sort of beverage.  ;)


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Feedback on CLI message outputs

[James]

Hi there,

I'm currently using the --xml flag when running gluster via automatic
scripts such as via puppet exec. Please keep this stable :)

Thank you!
James


On Mon, 2012-08-06 at 12:31 +0530, Amar Tumballi wrote:
> Hi All,
> 
> We are in the process of improving our log messages, and message outputs 
> through CLI. As a first step, we want to make sure our CLI outputs are 
> both meaningful and script friendly (for people who want to use the CLI 
> output in their scripts).
> 
> We are taking few baby steps in this regard in upstream (master branch). 
> I recently sent a patch http://review.gluster.com/3753 which tries to 
> separate out most of the error messages by outputting it to stderr 
> instead of stdout.
> 
> In the meantime, I tried to keep it script friendly too. With that patch 
> going in, the problem is, if any of you have written scripts which greps 
> for specific pattern in the CLI output, will be broken when the next 
> major release happens.
> 
> So, I want your feed back on getting it right by involving you all to 
> make the messages more friendlier to both script writers, and also to 
> admins who would understand the messages properly. (ie, meaning of the 
> messages should be unambiguous).
> 
> To try to see the difference in CLI outputs, you need to checkout git 
> repo, and build locally, as there are no releases available at the moment.
> 
> 
> Thanks and Regards,
> Amar
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users



signature.asc
Description: This is a digitally signed message part
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] brick online or not? Don't trust 'gluster peer status'

[Harry Mangalam]

As a final(?) follow-up to my problem, after restarting the rebalance with:

 gluster volume rebalance [vol-name] fix-layout start

it finished up last night after plowing thru the entirety of the filesystem
- fixing about ~1M files (apparently ~2.2TB), all while the fs remained
live (tho probably a bit slower than users would have liked).   That's a
strong '+' in the gluster column for resiliency.

I started the rebalance without waiting for any advice to the contrary.
 3.3 is supposed to have a built-in rebalance operator, but I saw no
evidence of it.  Other info from gluster.org suggested that it wouldn't do
any harm to do this, so I went ahead and started it.  Do the gluster
wizards have any final words on this before I write this up in our trouble
report?

best wishes
harry


On Thu, Aug 2, 2012 at 4:37 PM, Harry Mangalam  wrote:

> Further to what I wrote before:
> gluster server overload; recovers, now "Transport endpoint is not
> connected" for some files
> 
>
> I'm getting conflicting info here.  On one hand, the peer that had its
> glusterfsd  lock up seems to be in the gluster system, according to
> the frequently referenced 'gluster peer status'
>
> Thu Aug 02 15:48:46 [1.00 0.89 0.92]  root@pbs1:~
> 729 $ gluster peer status
> Number of Peers: 3
>
> Hostname: pbs4ib
> Uuid: 2a593581-bf45-446c-8f7c-212c53297803
> State: Peer in Cluster (Connected)
>
> Hostname: pbs2ib
> Uuid: 26de63bd-c5b7-48ba-b81d-5d77a533d077
> State: Peer in Cluster (Connected)
>
> Hostname: pbs3ib
> Uuid: c79c4084-d6b9-4af9-b975-40dd6aa99b42
> State: Peer in Cluster (Connected)
>
> On the other hand, some errors that I provided yesterday:
> ===
> [2012-08-01 18:07:26.104910] W
> [dht-selfheal.c:875:dht_selfheal_directory] 0-gli-dht: 1 subvolumes
> down -- not fixing
> ===
>
> as well as this information:
> $ gluster volume status all detail
>
> [top 2 brick stanzas trimmed; they're online]
>
> --
> Brick: Brick pbs3ib:/bducgl
> Port : 24018
> Online   : N   <<=
> Pid  : 20953
> File System  : xfs
> Device   : /dev/md127
> Mount Options: rw
> Inode Size   : 256
> Disk Space Free  : 6.1TB
> Total Disk Space : 8.2TB
> Inode Count  : 1758158080
> Free Inodes  : 1752326373
>
> --
> Brick: Brick pbs4ib:/bducgl
> Port : 24009
> Online   : Y
> Pid  : 20948
> File System  : xfs
> Device   : /dev/sda
> Mount Options: rw
> Inode Size   : 256
> Disk Space Free  : 4.6TB
> Total Disk Space : 6.4TB
> Inode Count  : 1367187392
> Free Inodes  : 1361305613
>
> The above implies fairly strongly that the brick did not re-establish
> connection to the volume, altho the gluster peer info did.
>
> Strangely enough, when I RE-restarted the glusterd, it DID come back
> and re-joined the gluster volume and now the (restarted) fix-layout
> job is proceeding without those  "subvolumes
> down -- not fixing" errors, just a steady stream of 'found
> anomalies/fixing the layout' messages, tho at the rate that it's going
> it looks like it will take several days.
>
> Still better several days to fix the data on-disk and having the fs
> live than having to tell users that their data is gone and then having
> to rebuild from zero.  Luckily, it's officially a /scratch filesystem.
>
> Harry
>
> --
> Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
> [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
> 415 South Circle View Dr, Irvine, CA, 92697 [shipping]
> MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
>



-- 
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster-users Digest, Vol 51, Issue 46

[John Mark Walker]

- Original Message -

> Hi Ben,

> Thanks for the expert advice.

> On Fri, Aug 3, 2012 at 2:35 PM, Ben England < bengl...@redhat.com >
> wrote:

> > 4. Re: kernel parameters for improving gluster writes on millions
> > of
> > small writes (long) (Harry Mangalam)
> 

> > Harry, You are correct, Glusterfs throughput with small write
> > transfer sizes is a client-side problem, here are workarounds that
> > at least some applications could use.
> 

> Not to be impertinent nor snarky, but why is the gluster client
> written in this way and is that a high priority for fixing? It seems
> that caching/buffering is one of the great central truths of
> computer science in general. Is there a countering argument for not
> doing this?
As a general point, you'll find that glusterfs always (almost always?) errs on 
the side of data consistency, even if it adversely affects performance. An NFS 
client can cache because it doesn't have to worry about HA - which has to be 
implemented with other tools. With recent changes in GlusterFS code, including 
further development of server-side code, it should be possible to create some 
type of client-side caching in the near future. There are also developments in 
fuse to think about, but mostly, it has to do with glusterfs' new server code. 
Previously, all the intelligence was in the client, so data consistency on the 
client was absolutely essential. Now, with "smarter" server-side translators, 
eg. self-heal and rebalancing, this is shifting. 3.3 was the first release with 
the shift in that direction, and more is coming. I know this doesn't help you 
*now*, but I wanted to give you an idea of why it is this way, and how it's 
changing going forward. 

-JM 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] Feedback on CLI message outputs

[Amar Tumballi]

Hi All,

We are in the process of improving our log messages, and message outputs 
through CLI. As a first step, we want to make sure our CLI outputs are 
both meaningful and script friendly (for people who want to use the CLI 
output in their scripts).


We are taking few baby steps in this regard in upstream (master branch). 
I recently sent a patch http://review.gluster.com/3753 which tries to 
separate out most of the error messages by outputting it to stderr 
instead of stdout.


In the meantime, I tried to keep it script friendly too. With that patch 
going in, the problem is, if any of you have written scripts which greps 
for specific pattern in the CLI output, will be broken when the next 
major release happens.


So, I want your feed back on getting it right by involving you all to 
make the messages more friendlier to both script writers, and also to 
admins who would understand the messages properly. (ie, meaning of the 
messages should be unambiguous).


To try to see the difference in CLI outputs, you need to checkout git 
repo, and build locally, as there are no releases available at the moment.



Thanks and Regards,
Amar
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users