Re: [Gluster-devel] Logging in a multi-brick daemon

[Dustin Black]

On Feb 15, 2017 5:39 PM, "Jeff Darcy"  wrote:

One of the issues that has come up with multiplexing is that all of the
bricks in a process end up sharing a single log file.  The reaction from
both of the people who have mentioned this is that we should find a way to
give each brick its own log even when they're in the same process, and make
sure gf_log etc. are able to direct messages to the correct one.  I can
think of ways to do this, but it doesn't seem optimal to me.  It will
certainly use up a lot of file descriptors.  I think it will use more
memory.  And then there's the issue of whether this would really be better
for debugging.  Often it's necessary to look at multiple brick logs while
trying to diagnose this problem, so it's actually kind of handy to have
them all in one file.  Which would you rather do?

(a) Weave together entries in multiple logs, either via a script or in your
head?

(b) Split or filter entries in a single log, according to which brick
they're from?


+1 for a single log file with tagging, combined with necessary grep-fu.
Plus I like the idea of an included script or other facility to aid said
grepping.


To me, (b) seems like a much more tractable problem.  I'd say that what we
need is not multiple logs, but *marking of entries* so that everything
pertaining to one brick can easily be found.  One way to do this would be
to modify volgen so that a brick ID (not name because that's a path and
hence too long) is appended/prepended to the name of every translator in
the brick.  Grep for that brick ID, and voila!  You now have all log
messages for that brick and no other.  A variant of this would be to leave
the names alone and modify gf_log so that it adds the brick ID
automagically (based on a thread-local variable similar to THIS).  Same
effect, other than making translator names longer, so I'd kind of prefer
this approach.  Before I start writing the code, does anybody else have any
opinions, preferences, or alternatives I haven't mentioned yet?

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Logging in a multi-brick daemon

[Shyam]

On 02/16/2017 05:27 AM, Rajesh Joseph wrote:

On Thu, Feb 16, 2017 at 9:46 AM, Ravishankar N  wrote:

On 02/16/2017 04:09 AM, Jeff Darcy wrote:


One of the issues that has come up with multiplexing is that all of the
bricks in a process end up sharing a single log file.  The reaction from
both of the people who have mentioned this is that we should find a way to
give each brick its own log even when they're in the same process, and make
sure gf_log etc. are able to direct messages to the correct one.  I can
think of ways to do this, but it doesn't seem optimal to me.  It will
certainly use up a lot of file descriptors.  I think it will use more
memory.  And then there's the issue of whether this would really be better
for debugging.  Often it's necessary to look at multiple brick logs while
trying to diagnose this problem, so it's actually kind of handy to have them
all in one file.  Which would you rather do?

(a) Weave together entries in multiple logs, either via a script or in
your head?

(b) Split or filter entries in a single log, according to which brick
they're from?

To me, (b) seems like a much more tractable problem.  I'd say that what we
need is not multiple logs, but *marking of entries* so that everything
pertaining to one brick can easily be found.  One way to do this would be to
modify volgen so that a brick ID (not name because that's a path and hence
too long) is appended/prepended to the name of every translator in the
brick.  Grep for that brick ID, and voila!  You now have all log messages
for that brick and no other.  A variant of this would be to leave the names
alone and modify gf_log so that it adds the brick ID automagically (based on
a thread-local variable similar to THIS).  Same effect, other than making
translator names longer, so I'd kind of prefer this approach.  Before I
start writing the code, does anybody else have any opinions, preferences, or
alternatives I haven't mentioned yet?

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel


My vote is for having separate log files per brick. Even in separate log
files
that we have today, I find it difficult to mentally ignore irrelevant
messages
in a single log file as I am sifting through it to look for errors that are
related to the problem at hand. Having entries from multiple bricks and then
grepping it would only make things harder. I cannot think of a case where
having
entries from all bricks in one file would particularly be beneficial for
debugging since what happens in one brick is independent of the other bricks
(at least until we move client xlators to server side and run them in the
brick process).
As for file descriptor count/memory usage, I think we should be okay
as it is not any worse than that in the non-multiplexed approach we have
today.

On a side-note, I think the problem is not having too many log files but
having
them in multiple nodes. Having a log-aggregation solution where all messages
are
logged to a single machine (but still in separate files) would make it
easier to
monitor/debug issues.
-Ravi



I believe the logs are not just from one volume but from all. In that
case merging them
into a single log file may not be great for debugging. Especially in
container use cases
there can be multiple volumes. Yes, with some tagging and scripting we
can separate
the logs and still live with it.


In container world, *I believe* centralized logging (using something 
like an ELK/EFK stack) would be the way to go, than collecting logs from 
each gluster (or application mount) container/node. In these situations 
we are going to get logs from different volumes anyway, or at best a 
filtered list from whichever stack is used for the centralized logging.


So, I would think (as being described) we would need to have enough 
identifiers in the log message, such that we can filter appropriately 
and that should take care of the debugging concern.


Of course building these scripts out from the beginning and possibly 
even shipping them with our RPMs may help a great deal, than having to 
roll one out when we get into troubleshooting or debugging a setup.




What about the log levels? Each volume can configure different log
levels. Will you carve
out a separate process in case log levels are changed for a volume?
How is this handled
here?

-Rajesh
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Logging in a multi-brick daemon

[Jeff Darcy]

> Debugging will involve getting far more/bigger files from customers
> unless we have a script (?) to grep out only those messages pertaining
> to the volume in question. IIUC, this would just be grepping for the
> volname and then determining which brick each message pertains to
> based on the brick id, correct?

Correct.  There would also be some possibly-interesting messages that
aren't specifically tied to any one brick, e.g. in protocol/server or
various parts of libglusterfs, so we'd probably always want those no
matter what brick(s) we're interested in.

> Would brick ids remain constant across add/remove brick operations? An
> easy way would probably be just to use the client xlator number as the
> brick id which would make it easy to map the brick to client
> connection.

Brick IDs should be constant across add/remove operations, which I
suppose means they'll need to be more globally unique than they are now
(client translator indices can clash across volumes).

> With several brick processes all writing to the same log file, can
> there be problems with interleaving messages?

AFAICT the "atom" of logging is a line, so there shouldn't be problems
of interleaving within a line ("foo" + "bar" won't become "fboaro").
However, when code tries to log multiple lines together - e.g. DHT
layouts or AFR self-heal info - that could end up being interleaved with
another brick doing the same.  They'd still be distinct according to
brick ID, but when looking at an unfiltered log it could look a bit
confusing.

> Logrotate might kick in faster as well causing us to lose debugging
> data if only a limited number of files are saved, as all those files
> would now hold less log data per volume. The logrotate config options
> would need to be changed to keep more files.

True.

> Having all messages for the bricks of the same volume in a single file
> would definitely be helpful. Still thinking through logging all
> messages for all bricks in a single file. :)

Something to keep in mind is that multiplexing everything into only one
process is only temporary.  Very soon, we'll be multiplexing into
multiple processes, with the number of processes proportional to the
number of cores in the system.  So, for a node with 200 bricks and 24
cores, we might have 24 processes each containing ~8 bricks.  In that
case, it would make sense to keep bricks for the same volume separate as
much as possible.  A process is a failure domain, and having multiple
related bricks in the same failure domain is undesirable (though
unavoidable in some cases).  The consequence for logging is that you'd
still have to look at multiple files.
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Logging in a multi-brick daemon

[Jeff Darcy]

> What about the log levels? Each volume can configure different log
> levels. Will you carve
> out a separate process in case log levels are changed for a volume?

I don't think we need to go that far, but you do raise a good point.
Log levels would need to be fetched from a brick-specific location
(probably similar to what I suggested for the brick ID) instead of
from the shared context.  Thanks for pointing that out.
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Logging in a multi-brick daemon

[Jeff Darcy]

> As for file descriptor count/memory usage, I think we should be okay
> as it is not any worse than that in the non-multiplexed approach we
> have today.

I don't think that's true.  Each logging context allocates a certain
amount of memory.  Let's call that X.  With N bricks in separate
processes, our total memory is N*X.  With N bricks in one process, each
using its own logging context, we're still at N*X.  However, with N
bricks in one process sharing a context, we'd be down to X.  Even more
importantly, with separate contexts we'd be touching N pages all the
time, increasing the amount of *physical* memory that is devoted to
logging.

With respect to file descriptors, having N file descriptors *in one
process* is more problematic than having one per process in N processes.
We already have approximately N times as many file descriptors per
process for client connections; adding another N for logging descriptors
isn't going to be helpful when N might be in the hundreds.

These might not be big enough effects to rule out the separate-log
approach, but they're not zero either.  Since reducing memory usage is a
primary goal of multiplexing, I don't think we should lightly forego
opportunities to do so.
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Logging in a multi-brick daemon

[Rajesh Joseph]

On Thu, Feb 16, 2017 at 9:46 AM, Ravishankar N  wrote:
> On 02/16/2017 04:09 AM, Jeff Darcy wrote:
>>
>> One of the issues that has come up with multiplexing is that all of the
>> bricks in a process end up sharing a single log file.  The reaction from
>> both of the people who have mentioned this is that we should find a way to
>> give each brick its own log even when they're in the same process, and make
>> sure gf_log etc. are able to direct messages to the correct one.  I can
>> think of ways to do this, but it doesn't seem optimal to me.  It will
>> certainly use up a lot of file descriptors.  I think it will use more
>> memory.  And then there's the issue of whether this would really be better
>> for debugging.  Often it's necessary to look at multiple brick logs while
>> trying to diagnose this problem, so it's actually kind of handy to have them
>> all in one file.  Which would you rather do?
>>
>> (a) Weave together entries in multiple logs, either via a script or in
>> your head?
>>
>> (b) Split or filter entries in a single log, according to which brick
>> they're from?
>>
>> To me, (b) seems like a much more tractable problem.  I'd say that what we
>> need is not multiple logs, but *marking of entries* so that everything
>> pertaining to one brick can easily be found.  One way to do this would be to
>> modify volgen so that a brick ID (not name because that's a path and hence
>> too long) is appended/prepended to the name of every translator in the
>> brick.  Grep for that brick ID, and voila!  You now have all log messages
>> for that brick and no other.  A variant of this would be to leave the names
>> alone and modify gf_log so that it adds the brick ID automagically (based on
>> a thread-local variable similar to THIS).  Same effect, other than making
>> translator names longer, so I'd kind of prefer this approach.  Before I
>> start writing the code, does anybody else have any opinions, preferences, or
>> alternatives I haven't mentioned yet?
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>
> My vote is for having separate log files per brick. Even in separate log
> files
> that we have today, I find it difficult to mentally ignore irrelevant
> messages
> in a single log file as I am sifting through it to look for errors that are
> related to the problem at hand. Having entries from multiple bricks and then
> grepping it would only make things harder. I cannot think of a case where
> having
> entries from all bricks in one file would particularly be beneficial for
> debugging since what happens in one brick is independent of the other bricks
> (at least until we move client xlators to server side and run them in the
> brick process).
> As for file descriptor count/memory usage, I think we should be okay
> as it is not any worse than that in the non-multiplexed approach we have
> today.
>
> On a side-note, I think the problem is not having too many log files but
> having
> them in multiple nodes. Having a log-aggregation solution where all messages
> are
> logged to a single machine (but still in separate files) would make it
> easier to
> monitor/debug issues.
> -Ravi
>

I believe the logs are not just from one volume but from all. In that
case merging them
into a single log file may not be great for debugging. Especially in
container use cases
there can be multiple volumes. Yes, with some tagging and scripting we
can separate
the logs and still live with it.

What about the log levels? Each volume can configure different log
levels. Will you carve
out a separate process in case log levels are changed for a volume?
How is this handled
here?

-Rajesh
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Logging in a multi-brick daemon

[Ravishankar N]

On 02/16/2017 04:09 AM, Jeff Darcy wrote:

One of the issues that has come up with multiplexing is that all of the bricks 
in a process end up sharing a single log file.  The reaction from both of the 
people who have mentioned this is that we should find a way to give each brick 
its own log even when they're in the same process, and make sure gf_log etc. 
are able to direct messages to the correct one.  I can think of ways to do 
this, but it doesn't seem optimal to me.  It will certainly use up a lot of 
file descriptors.  I think it will use more memory.  And then there's the issue 
of whether this would really be better for debugging.  Often it's necessary to 
look at multiple brick logs while trying to diagnose this problem, so it's 
actually kind of handy to have them all in one file.  Which would you rather do?

(a) Weave together entries in multiple logs, either via a script or in your 
head?

(b) Split or filter entries in a single log, according to which brick they're 
from?

To me, (b) seems like a much more tractable problem.  I'd say that what we need 
is not multiple logs, but *marking of entries* so that everything pertaining to 
one brick can easily be found.  One way to do this would be to modify volgen so 
that a brick ID (not name because that's a path and hence too long) is 
appended/prepended to the name of every translator in the brick.  Grep for that 
brick ID, and voila!  You now have all log messages for that brick and no 
other.  A variant of this would be to leave the names alone and modify gf_log 
so that it adds the brick ID automagically (based on a thread-local variable 
similar to THIS).  Same effect, other than making translator names longer, so 
I'd kind of prefer this approach.  Before I start writing the code, does 
anybody else have any opinions, preferences, or alternatives I haven't 
mentioned yet?

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel
My vote is for having separate log files per brick. Even in separate log 
files
that we have today, I find it difficult to mentally ignore irrelevant 
messages

in a single log file as I am sifting through it to look for errors that are
related to the problem at hand. Having entries from multiple bricks and then
grepping it would only make things harder. I cannot think of a case 
where having

entries from all bricks in one file would particularly be beneficial for
debugging since what happens in one brick is independent of the other 
bricks
(at least until we move client xlators to server side and run them in 
the brick process).

As for file descriptor count/memory usage, I think we should be okay
as it is not any worse than that in the non-multiplexed approach we have
today.

On a side-note, I think the problem is not having too many log files but 
having
them in multiple nodes. Having a log-aggregation solution where all 
messages are
logged to a single machine (but still in separate files) would make it 
easier to

monitor/debug issues.
-Ravi
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Logging in a multi-brick daemon

[Nithya Balachandran]

On 16 February 2017 at 07:30, Shyam  wrote:

> On 02/15/2017 08:51 PM, Atin Mukherjee wrote:
>
>>
>> On Thu, 16 Feb 2017 at 04:09, Jeff Darcy > > wrote:
>>
>> One of the issues that has come up with multiplexing is that all of
>> the bricks in a process end up sharing a single log file.  The
>> reaction from both of the people who have mentioned this is that we
>> should find a way to give each brick its own log even when they're
>> in the same process, and make sure gf_log etc. are able to direct
>> messages to the correct one.  I can think of ways to do this, but it
>> doesn't seem optimal to me.  It will certainly use up a lot of file
>> descriptors.  I think it will use more memory.  And then there's the
>> issue of whether this would really be better for debugging.  Often
>> it's necessary to look at multiple brick logs while trying to
>> diagnose this problem, so it's actually kind of handy to have them
>> all in one file.  Which would you rather do?
>>
>> (a) Weave together entries in multiple logs, either via a script or
>> in your head?
>>
>> (b) Split or filter entries in a single log, according to which
>> brick they're from?
>>
>> To me, (b) seems like a much more tractable problem.  I'd say that
>> what we need is not multiple logs, but *marking of entries* so that
>> everything pertaining to one brick can easily be found.  One way to
>> do this would be to modify volgen so that a brick ID (not name
>> because that's a path and hence too long) is appended/prepended to
>> the name of every translator in the brick.  Grep for that brick ID,
>> and voila!  You now have all log messages for that brick and no
>> other.  A variant of this would be to leave the names alone and
>> modify gf_log so that it adds the brick ID automagically (based on a
>> thread-local variable similar to THIS).  Same effect, other than
>> making translator names longer, so I'd kind of prefer this
>> approach.  Before I start writing the code, does anybody else have
>> any opinions, preferences, or alternatives I haven't mentioned yet?
>>
>


A few questions/thoughts here:

Debugging will involve getting far more/bigger files from customers unless
we have a script (?) to  grep out only those messages pertaining to the
volume in question. IIUC, this would just be grepping for the volname and
then determining which brick each message pertains to based on the brick
id, correct?

Would brick ids remain constant across add/remove brick operations? An easy
way would probably be just to use the client xlator number as the brick id
which would make it easy to map the brick to client connection.

With several brick processes all writing to the same log file, can there be
problems with interleaving messages?

Logrotate might kick in faster as well causing us to lose debugging data if
only a limited number of files are saved, as all those files would now hold
less log data per volume. The logrotate config options would need to be
changed to keep more files.

Having all messages for the bricks of the same volume in a single file
would definitely be helpful. Still thinking through logging all messages
for all bricks in a single file. :)



>
> (b) is better. Considering centralized logging or log file redirection
> etc, (a) becomes unnatural and unwieldy, (b) is better.
>
>
>>
>> I like this idea. +1
>>
>>
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org 
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>> --
>> - Atin (atinm)
>>
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Logging in a multi-brick daemon

[Shyam]

On 02/15/2017 08:51 PM, Atin Mukherjee wrote:


On Thu, 16 Feb 2017 at 04:09, Jeff Darcy mailto:jda...@redhat.com>> wrote:

One of the issues that has come up with multiplexing is that all of
the bricks in a process end up sharing a single log file.  The
reaction from both of the people who have mentioned this is that we
should find a way to give each brick its own log even when they're
in the same process, and make sure gf_log etc. are able to direct
messages to the correct one.  I can think of ways to do this, but it
doesn't seem optimal to me.  It will certainly use up a lot of file
descriptors.  I think it will use more memory.  And then there's the
issue of whether this would really be better for debugging.  Often
it's necessary to look at multiple brick logs while trying to
diagnose this problem, so it's actually kind of handy to have them
all in one file.  Which would you rather do?

(a) Weave together entries in multiple logs, either via a script or
in your head?

(b) Split or filter entries in a single log, according to which
brick they're from?

To me, (b) seems like a much more tractable problem.  I'd say that
what we need is not multiple logs, but *marking of entries* so that
everything pertaining to one brick can easily be found.  One way to
do this would be to modify volgen so that a brick ID (not name
because that's a path and hence too long) is appended/prepended to
the name of every translator in the brick.  Grep for that brick ID,
and voila!  You now have all log messages for that brick and no
other.  A variant of this would be to leave the names alone and
modify gf_log so that it adds the brick ID automagically (based on a
thread-local variable similar to THIS).  Same effect, other than
making translator names longer, so I'd kind of prefer this
approach.  Before I start writing the code, does anybody else have
any opinions, preferences, or alternatives I haven't mentioned yet?


(b) is better. Considering centralized logging or log file redirection 
etc, (a) becomes unnatural and unwieldy, (b) is better.





I like this idea. +1



___
Gluster-devel mailing list
Gluster-devel@gluster.org 
http://lists.gluster.org/mailman/listinfo/gluster-devel

--
- Atin (atinm)


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Logging in a multi-brick daemon

[Atin Mukherjee]

On Thu, 16 Feb 2017 at 04:09, Jeff Darcy  wrote:

> One of the issues that has come up with multiplexing is that all of the
> bricks in a process end up sharing a single log file.  The reaction from
> both of the people who have mentioned this is that we should find a way to
> give each brick its own log even when they're in the same process, and make
> sure gf_log etc. are able to direct messages to the correct one.  I can
> think of ways to do this, but it doesn't seem optimal to me.  It will
> certainly use up a lot of file descriptors.  I think it will use more
> memory.  And then there's the issue of whether this would really be better
> for debugging.  Often it's necessary to look at multiple brick logs while
> trying to diagnose this problem, so it's actually kind of handy to have
> them all in one file.  Which would you rather do?
>
> (a) Weave together entries in multiple logs, either via a script or in
> your head?
>
> (b) Split or filter entries in a single log, according to which brick
> they're from?
>
> To me, (b) seems like a much more tractable problem.  I'd say that what we
> need is not multiple logs, but *marking of entries* so that everything
> pertaining to one brick can easily be found.  One way to do this would be
> to modify volgen so that a brick ID (not name because that's a path and
> hence too long) is appended/prepended to the name of every translator in
> the brick.  Grep for that brick ID, and voila!  You now have all log
> messages for that brick and no other.  A variant of this would be to leave
> the names alone and modify gf_log so that it adds the brick ID
> automagically (based on a thread-local variable similar to THIS).  Same
> effect, other than making translator names longer, so I'd kind of prefer
> this approach.  Before I start writing the code, does anybody else have any
> opinions, preferences, or alternatives I haven't mentioned yet?


I like this idea. +1


>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>
-- 
- Atin (atinm)
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Logging in a multi-brick daemon

[Jeff Darcy]

One of the issues that has come up with multiplexing is that all of the bricks 
in a process end up sharing a single log file.  The reaction from both of the 
people who have mentioned this is that we should find a way to give each brick 
its own log even when they're in the same process, and make sure gf_log etc. 
are able to direct messages to the correct one.  I can think of ways to do 
this, but it doesn't seem optimal to me.  It will certainly use up a lot of 
file descriptors.  I think it will use more memory.  And then there's the issue 
of whether this would really be better for debugging.  Often it's necessary to 
look at multiple brick logs while trying to diagnose this problem, so it's 
actually kind of handy to have them all in one file.  Which would you rather do?

(a) Weave together entries in multiple logs, either via a script or in your 
head?

(b) Split or filter entries in a single log, according to which brick they're 
from?

To me, (b) seems like a much more tractable problem.  I'd say that what we need 
is not multiple logs, but *marking of entries* so that everything pertaining to 
one brick can easily be found.  One way to do this would be to modify volgen so 
that a brick ID (not name because that's a path and hence too long) is 
appended/prepended to the name of every translator in the brick.  Grep for that 
brick ID, and voila!  You now have all log messages for that brick and no 
other.  A variant of this would be to leave the names alone and modify gf_log 
so that it adds the brick ID automagically (based on a thread-local variable 
similar to THIS).  Same effect, other than making translator names longer, so 
I'd kind of prefer this approach.  Before I start writing the code, does 
anybody else have any opinions, preferences, or alternatives I haven't 
mentioned yet?

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel