Re: [Gluster-devel] Help needed in understanding GlusterFS logs and debugging elasticsearch failures

2015-12-17 Thread Sachidananda URS 
Hi,

I tried the same use case with pure DHT (1 & 2 nodes). I don't see any
problems.
However, if I try the same tests with distributed replicate, the indices go
into red.

If any additional details are needed than the logs attached in the earlier
mails please let me know.

-sac


On Mon, Dec 14, 2015 at 4:13 PM, Sachidananda URS  wrote:

> Hi,
>
>
> On Sat, Dec 12, 2015 at 2:35 AM, Vijay Bellur  wrote:
>
>>
>>
>> - Original Message -
>> > From: "Sachidananda URS" 
>> > To: "Gluster Devel" 
>> > Sent: Friday, December 11, 2015 10:26:04 AM
>> > Subject: [Gluster-devel] Help needed in understanding GlusterFS logs
>> and debugging elasticsearch failures
>> >
>> > Hi,
>> >
>> > I was trying to use GlusterFS as a backend filesystem for storing the
>> > elasticsearch indices on GlusterFS mount.
>> >
>> > The filesystem operations as far as I can understand is, lucene engine
>> > does a lot of renames on the index files. And multiple threads read
>> > from the same file concurrently.
>> >
>> > While writing index, elasticsearch/lucene complains of index corruption
>> and
>> > the
>> > health of the cluster goes to red, and all the operations on the index
>> fail
>> > hereafter.
>> >
>> > ===
>> >
>> > [2015-12-10 02:43:45,614][WARN ][index.engine ] [client-2]
>> > [logstash-2015.12.09][3] failed engine [merge failed]
>> > org.apache.lucene.index.MergePolicy$MergeException:
>> > org.apache.lucene.index.CorruptIndexException: checksum failed (hardware
>> > problem?) : expected=0 actual=6d811d06
>> >
>> (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/mnt/gluster2/rhs/nodes/0/indices/logstash-2015.12.09/3/index/_a7.cfs")
>> > [slice=_a7_Lucene50_0.doc]))
>> > at
>> >
>>  
>> org.elasticsearch.index.engine.InternalEngine$EngineMergeScheduler$1.doRun(InternalEngine.java:1233)
>> > at
>> >
>>  
>> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
>> > at
>> >
>>  
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>>  
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> > Caused by: org.apache.lucene.index.CorruptIndexException: checksum
>> failed
>> > (hardware problem?) : expected=0 actual=6d811d06
>> >
>> (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/mnt/gluster2/rhs/nodes/0/indices/logstash-2015.12.09/3/index/_a7.cfs")
>> > [slice=_a7_Lucene50_0.doc]))
>> >
>> > =
>> >
>> >
>> > Server logs does not have anything. The client logs is full of messages
>> like:
>> >
>> >
>> >
>> > [2015-12-03 18:44:17.882032] I [MSGID: 109066]
>> [dht-rename.c:1410:dht_rename]
>> > 0-esearch-dht: renaming
>> >
>> /rhs/nodes/0/indices/logstash-2015.12.03/1/translog/translog-61881676454442626.tlog
>> > (hash=esearch-replicate-0/cache=esearch-replicate-0) =>
>> > /rhs/nodes/0/indices/logstash-2015.12.03/1/translog/translog-311.ckp
>> > (hash=esearch-replicate-1/cache=)
>> > [2015-12-03 18:45:31.276316] I [MSGID: 109066]
>> [dht-rename.c:1410:dht_rename]
>> > 0-esearch-dht: renaming
>> >
>> /rhs/nodes/0/indices/logstash-2015.12.03/1/translog/translog-2384654015514619399.tlog
>> > (hash=esearch-replicate-0/cache=esearch-replicate-0) =>
>> > /rhs/nodes/0/indices/logstash-2015.12.03/1/translog/translog-312.ckp
>> > (hash=esearch-replicate-0/cache=)
>> > [2015-12-03 18:45:31.587660] I [MSGID: 109066]
>> [dht-rename.c:1410:dht_rename]
>> > 0-esearch-dht: renaming
>> >
>> /rhs/nodes/0/indices/logstash-2015.12.03/4/translog/translog-4957943728738197940.tlog
>> > (hash=esearch-replicate-0/cache=esearch-replicate-0) =>
>> > /rhs/nodes/0/indices/logstash-2015.12.03/4/translog/translog-312.ckp
>> > (hash=esearch-replicate-0/cache=)
>> > [2015-12-03 18:46:48.424605] I [MSGID: 109066]
>> [dht-rename.c:1410:dht_rename]
>> > 0-esearch-dht: renaming
>> >
>> /rhs/nodes/0/indices/logstash-2015.12.03/1/translog/translog-1731620600607498012.tlog
>> > (hash=esearch-replicate-1/cache=esearch-replicate-1) =>
>> > /rhs/nodes/0/indices/logstash-2015.12.03/1/translog/translog-313.ckp
>> > (hash=esearch-replicate-1/cache=)
>> > [2015-12-03 18:46:48.466558] I [MSGID: 109066]
>> [dht-rename.c:1410:dht_rename]
>> > 0-esearch-dht: renaming
>> >
>> /rhs/nodes/0/indices/logstash-2015.12.03/4/translog/translog-5214949393126318982.tlog
>> > (hash=esearch-replicate-1/cache=esearch-replicate-1) =>
>> > /rhs/nodes/0/indices/logstash-2015.12.03/4/translog/translog-313.ckp
>> > (hash=esearch-replicate-1/cache=)
>> > [2015-12-03 18:48:06.314138] I [MSGID: 109066]
>> [dht-rename.c:1410:dht_rename]
>> > 0-esearch-dht: renaming
>> >
>> /rhs/nodes/0/indices/logstash-2015.12.03/4/translog/translog-9110755229226773921.tlog
>> > (hash=esearch-replicate-0/cache=esearch-replicate-0) =>
>> >

Re: [Gluster-devel] Help needed in understanding GlusterFS logs and debugging elasticsearch failures

2015-12-17 Thread Vijay Bellur 

On 12/17/2015 05:09 AM, Sachidananda URS wrote:

Hi,

I tried the same use case with pure DHT (1 & 2 nodes). I don't see any
problems.
However, if I try the same tests with distributed replicate, the indices
go into red.

If any additional details are needed than the logs attached in the
earlier mails please let me know.



Can you please try with option "cluster.consistent-metadata" enabled on 
a distributed replicated volume?


Thanks,
Vijay

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Recognizing contributors and displaying other useful bits?

2015-12-17 Thread Kaleb KEITHLEY 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 12/17/2015 10:05 AM, Niels de Vos wrote:
> On Wed, Dec 16, 2015 at 10:05:04PM +0530, Ravishankar N wrote:
> 
>> But yes, if  we do want to expose this information via the mount 
>> point, then maybe /mount/.meta is a good place.
> 
> Indeed, .meta would be better :)

Can you be certain there isn't some application somewhere that uses
/mount/.meta ?

I would prefer that we not litter the namespace with magic filenames.
Any more than it's already littered.

I'd prefer an xattr over a file. And something that's sufficiently
random that it won't collide (high probability) with an application
file. E.g. a uuid created with `uuidgen -r`

- --

Kaleb

-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQEcBAEBCAAGBQJWctNJAAoJEDcw3UmJzK6LqBwH/AxqUtxvn8fWF/SqlLc4GekA
Ukpa/4i/VBjoLQLFLkS6HJw3e60ZaE5QnDWXyYPZPQXseQuxIC971rVI7JQgsOCA
HbrkNirkZ45LHLD1g7XO9+hJkObXF3aXAvsugmQByt4QZpVY3Rmz9gUjsH33WTCF
BFlOeYoGK75onWtK3haCYUJqEqlgXQmPXS0lCOJt+yXzznpF86G6ShKuf1GQvGzy
ugVQx8HRLsqY7Zvbf9n7ve8tLtCV22xR1uU3lUBqJFbVo9aum1rRX4H6QMb7M6qj
aFld1na0YeQOYG6HoRIdgQzwzbRzQZi1i3z8JJXRj8XQOdmA+XzvzP2JXOEPusA=
=V/id
-END PGP SIGNATURE-


0x89CCAE8B.asc
Description: application/pgp-keys
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Recognizing contributors and displaying other useful bits?

2015-12-17 Thread Niels de Vos 
On Thu, Dec 17, 2015 at 10:22:56AM -0500, Kaleb KEITHLEY wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
> On 12/17/2015 10:05 AM, Niels de Vos wrote:
> > On Wed, Dec 16, 2015 at 10:05:04PM +0530, Ravishankar N wrote:
> > 
> >> But yes, if  we do want to expose this information via the mount 
> >> point, then maybe /mount/.meta is a good place.
> > 
> > Indeed, .meta would be better :)
> 
> Can you be certain there isn't some application somewhere that uses
> /mount/.meta ?
> 
> I would prefer that we not litter the namespace with magic filenames.
> Any more than it's already littered.

That is why I was suggesting .glusterfs. There would not be many
applications that use this directory. However, .meta would be more
suitable because we already have the meta xlator:

  https://github.com/gluster/glusterfs-specs/blob/master/done/Features/meta.md

I'm thinking of extending this to include at least an AUTHORS file.

Niels


> 
> I'd prefer an xattr over a file. And something that's sufficiently
> random that it won't collide (high probability) with an application
> file. E.g. a uuid created with `uuidgen -r`
> 
> - --
> 
> Kaleb
> 
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v2
> 
> iQEcBAEBCAAGBQJWctNJAAoJEDcw3UmJzK6LqBwH/AxqUtxvn8fWF/SqlLc4GekA
> Ukpa/4i/VBjoLQLFLkS6HJw3e60ZaE5QnDWXyYPZPQXseQuxIC971rVI7JQgsOCA
> HbrkNirkZ45LHLD1g7XO9+hJkObXF3aXAvsugmQByt4QZpVY3Rmz9gUjsH33WTCF
> BFlOeYoGK75onWtK3haCYUJqEqlgXQmPXS0lCOJt+yXzznpF86G6ShKuf1GQvGzy
> ugVQx8HRLsqY7Zvbf9n7ve8tLtCV22xR1uU3lUBqJFbVo9aum1rRX4H6QMb7M6qj
> aFld1na0YeQOYG6HoRIdgQzwzbRzQZi1i3z8JJXRj8XQOdmA+XzvzP2JXOEPusA=
> =V/id
> -END PGP SIGNATURE-

> pub  2048R/89CCAE8B 2011-10-04 Kaleb S. KEITHLEY 
> sub  2048R/AFFECD68 2011-10-04

> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel



signature.asc
Description: PGP signature
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Recognizing contributors and displaying other useful bits?

2015-12-17 Thread Niels de Vos 
On Wed, Dec 16, 2015 at 10:05:04PM +0530, Ravishankar N wrote:
> On 12/16/2015 07:36 PM, Niels de Vos wrote:
> >Hi,
> >
> >Many GUI tools provide an "About" box that displays some information
> >about the project. Some applications (Wireshark for example) go an extra
> >step by including a list of all people that contributed patches. That is
> >quite a nice way for contributors to see the appreciation of their work.
> >
> >I think it would be really awesome if we could do something similar. We
> >have our reserved+hidden ".glusterfs" directory where all access is
> >denied. We could populate this directory with static compiled-in
> >contents. When accessed from a client, the licenses, versions, authors
> >and other bits could be displayed. A normal readdir should still not
> >list the directory at all (no change from current behaviour).
> >
> >Any thoughts?
> 
> I think It would be better to revive the who-wrote-glusterfs reports that
> Vijay used to send on the mailing list instead of statically adding
> contributor names to code.

Well, I would like to see an AUTHORS list automatically generated by
'make dist' or the such.

> We could have a `gluster --credits` or `gluster--help`  cli that points to
> gluster.org and maybe add a button thingy on the website which when clicked
> runs the who-wrote-glusterfs.sh  on   and displays the
> result. Things like version numbers and licenses are/should be covered in
> gluster--version.

I think it would be nice to be able to see some information through a
client mount point. Not everyone that uses Gluster has access to the
storage servers, and it can benefit them to see some info.

> But yes, if  we do want to expose this information via the mount point, then
> maybe /mount/.meta is a good place.

Indeed, .meta would be better :)

Thanks,
Niels

> 
> My 2 cents.
> Ravi
> 
> 
> 
> >Niels
> >
> >
> >___
> >Gluster-devel mailing list
> >Gluster-devel@gluster.org
> >http://www.gluster.org/mailman/listinfo/gluster-devel
> 
> 
> -- 
> > : Any porters out there should feel happier knowing that DEC is shipping >
> : me an AlphaPC that I intend to try getting linux running on: this will > :
> definitely help flush out some of the most flagrant unportable stuff. > :
> The Alpha is much more different from the i386 than the 68k stuff is, so > :
> it's likely to get most of the stuff fixed. > > It's posts like this that
> almost convince us non-believers that there > really is a god. (A follow-up
> by alov...@kerberos.demon.co.uk, Anthony Lovell, to Linus's remarks about
> porting)


signature.asc
Description: PGP signature
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Recognizing contributors and displaying other useful bits?

2015-12-17 Thread Niels de Vos 
On Wed, Dec 16, 2015 at 10:13:01PM +0530, Venky Shankar wrote:
> 
> 
> Ravishankar N wrote:
> >On 12/16/2015 07:36 PM, Niels de Vos wrote:
> >>Hi,
> >>
> >>Many GUI tools provide an "About" box that displays some information
> >>about the project. Some applications (Wireshark for example) go an extra
> >>step by including a list of all people that contributed patches. That is
> >>quite a nice way for contributors to see the appreciation of their work.
> >>
> >>I think it would be really awesome if we could do something similar. We
> >>have our reserved+hidden ".glusterfs" directory where all access is
> >>denied. We could populate this directory with static compiled-in
> >>contents. When accessed from a client, the licenses, versions, authors
> >>and other bits could be displayed. A normal readdir should still not
> >>list the directory at all (no change from current behaviour).
> >>
> >>Any thoughts?
> >
> >I think It would be better to revive the who-wrote-glusterfs reports
> >that Vijay used to send on the mailing list instead of statically adding
> >contributor names to code.
> >We could have a `gluster --credits` or `gluster--help`  cli that points
> >to gluster.org and maybe add a button thingy on the website which when
> >clicked runs the who-wrote-glusterfs.sh  on   and
> >displays the result. Things like version numbers and licenses are/should
> >be covered in gluster--version.
> 
> or a tshirt with contributors list on the back :)

That was a plan once, but I do not think this actually happened. Maybe
we should try to get that done for 3.8, or 4.0.

Niels

> 
> >
> >But yes, if  we do want to expose this information via the mount point,
> >then maybe /mount/.meta is a good place.
> 
> Ummm.. makes sense.
> 
> >
> >My 2 cents.
> >Ravi
> >
> >
> >
> >>Niels
> >>
> >>
> >>___
> >>Gluster-devel mailing list
> >>Gluster-devel@gluster.org
> >>http://www.gluster.org/mailman/listinfo/gluster-devel
> >
> >
> >--
> > > : Any porters out there should feel happier knowing that DEC is
> >shipping > : me an AlphaPC that I intend to try getting linux running
> >on: this will > : definitely help flush out some of the most flagrant
> >unportable stuff. > : The Alpha is much more different from the i386
> >than the 68k stuff is, so > : it's likely to get most of the stuff
> >fixed. > > It's posts like this that almost convince us non-believers
> >that there > really is a god. (A follow-up by
> >alov...@kerberos.demon.co.uk, Anthony Lovell, to Linus's remarks about
> >porting)
> >
> >___
> >Gluster-devel mailing list
> >Gluster-devel@gluster.org
> >http://www.gluster.org/mailman/listinfo/gluster-devel


signature.asc
Description: PGP signature
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Help needed in understanding GlusterFS logs and debugging elasticsearch failures

2015-12-17 Thread Sachidananda URS 
On Thu, Dec 17, 2015 at 4:03 PM, Vijay Bellur  wrote:

> On 12/17/2015 05:09 AM, Sachidananda URS wrote:
>
>> Hi,
>>
>> I tried the same use case with pure DHT (1 & 2 nodes). I don't see any
>> problems.
>> However, if I try the same tests with distributed replicate, the indices
>> go into red.
>>
>> If any additional details are needed than the logs attached in the
>> earlier mails please let me know.
>>
>>
> Can you please try with option "cluster.consistent-metadata" enabled on a
> distributed replicated volume?
>
>
I see the same errors with option `cluster.consistent-metadata' too. Will
work with Pranith, and see what the problem is.

-sac
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Gluster testing matrix

2015-12-17 Thread Deepak Shetty 
Where / How do you indicate whether the underlying GlusterFS is used/tested
using fusemount or libgfapi method or both ?

On Wed, Nov 25, 2015 at 5:39 PM, Raghavendra Talur 
wrote:

> Hi All,
>
> Here is a table representing the current state of Gluster testing.
>
> * Things in green are categories for which we have some kind of testing in
> place.
> * Things in red are the ones which don't have any tests.
> * Things in yellow are the ones which have no known tests or are not
> managed by Gluster community.
>
>
>
> Test Category/Sub-Category
>
>
>
>
>
> smoke source build + Posix complaince + Dbench
>
>
>
>
> functional tests/basic
>
>
>
>
> regression tests/bugs
>
>
>
>
> performance regression N/A
>
>
>
>
> integration Backends/FS Protocols Consumers/Cloud Environments Tools libgfapi
> bindings OS environment xfs smb qemu gdeploy java firewalld ext4 nfs
> openstack/cinder heketi python ufw btrfs swift openshift/docker/containers
> ruby selinux zfs
> aws
> go apparmor
>
> azure
>
>
>
>
> hadoop
>
>
> update major version upgrades minor version upgrades
>
>
>
> longevity a. memory leaks
> b. log accumulation
>
>
>
>
> distro packaging a. pkg build + smoke
> b. sysvinit/systemd
>
>
>
>
>
>
> I will send separate mails for each of the categories above highlighting
> plans for them.
>
> Use this thread to indicate addition/deletion/changes to this matrix.
> We will put this at gluster.org website once it is final.
>
> Thanks,
> Raghavendra Talur & MSV Bhat
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Help needed in understanding GlusterFS logs and debugging elasticsearch failures

2015-12-17 Thread Pranith Kumar Karampuri 



On 12/17/2015 04:03 PM, Vijay Bellur wrote:

On 12/17/2015 05:09 AM, Sachidananda URS wrote:

Hi,

I tried the same use case with pure DHT (1 & 2 nodes). I don't see any
problems.
However, if I try the same tests with distributed replicate, the indices
go into red.

If any additional details are needed than the logs attached in the
earlier mails please let me know.



Can you please try with option "cluster.consistent-metadata" enabled 
on a distributed replicated volume?
I am talking to Sac and will look at the machines with him. Will post 
with updates.


Pranith


Thanks,
Vijay

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel