Re: Elasticsearch-Hadoop repository plugin Cloudera Hadoop 2.0.0-cdh4.6.0
Thanks, Jinyuan (Jack) Zhou On Tue, Oct 14, 2014 at 1:36 PM, Costin Leau wrote: > You need the appropriate hadoop jar on your classpath otherwise > es-hadoop repository plugin cannot connect to HDFS. In the repo, > you'll find two versions with vanilla hadoop1 and hadoop2 - however if > you are using a certain distro, for best compatibility you should use > that distro client jars. > Please refer to your distro client configuration and use the light > version along with those jars. > > P.S. In the future, please create your own thread instead of > 'hijacking'/reviving an old one. > > Cheers, > > On Tue, Oct 14, 2014 at 10:00 PM, Jinyuan Zhou > wrote: > > My ES cluster nodes and Hadoop nodes are not collocated. Light version > does > > not works for me without putting enough correct versions of hadoop > related > > jars. Right now I don't want to create my jar as Brent did and I don't > want > > to install hadoop or copy jars on the es nodes either . Right now I can > wait > > until my hadoop cluster upgrade to a version that es-hadoop supports. Or > > there is a version that is compatible to the hadoop My ES nodes talk to . > > Thanks, > > > > > > On Monday, September 1, 2014 8:50:25 AM UTC-7, Mateusz Kaczynski wrote: > >> > >> (Much delayed) thank you Costin. > >> > >> Indeed, on Ubuntu, changing ES_CLASSPATH to include hadoop and > hadoop/lib > >> directories in /etc/default/elasticsearch (and exporting it in > >> /etc/init.d/elasticsearch) and installing light plugin version did work. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to elasticsearch+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/elasticsearch/ca1465d4-d617-4623-9de3-35bd69e6f8fb%40googlegroups.com > . > > > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/613YHEUAtuA/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAJogdmdihcht8e%3DDeYy0CTipOzazzT5KZgbkVbEyW80ZQEfVHA%40mail.gmail.com > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANBTPCGQFHAgz8SRzosoyJw8ixmqNJOxEh%3DUHa%3DMpP5XvfkWfA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Elasticsearch-Hadoop repository plugin Cloudera Hadoop 2.0.0-cdh4.6.0
You need the appropriate hadoop jar on your classpath otherwise es-hadoop repository plugin cannot connect to HDFS. In the repo, you'll find two versions with vanilla hadoop1 and hadoop2 - however if you are using a certain distro, for best compatibility you should use that distro client jars. Please refer to your distro client configuration and use the light version along with those jars. P.S. In the future, please create your own thread instead of 'hijacking'/reviving an old one. Cheers, On Tue, Oct 14, 2014 at 10:00 PM, Jinyuan Zhou wrote: > My ES cluster nodes and Hadoop nodes are not collocated. Light version does > not works for me without putting enough correct versions of hadoop related > jars. Right now I don't want to create my jar as Brent did and I don't want > to install hadoop or copy jars on the es nodes either . Right now I can wait > until my hadoop cluster upgrade to a version that es-hadoop supports. Or > there is a version that is compatible to the hadoop My ES nodes talk to . > Thanks, > > > On Monday, September 1, 2014 8:50:25 AM UTC-7, Mateusz Kaczynski wrote: >> >> (Much delayed) thank you Costin. >> >> Indeed, on Ubuntu, changing ES_CLASSPATH to include hadoop and hadoop/lib >> directories in /etc/default/elasticsearch (and exporting it in >> /etc/init.d/elasticsearch) and installing light plugin version did work. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/ca1465d4-d617-4623-9de3-35bd69e6f8fb%40googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJogdmdihcht8e%3DDeYy0CTipOzazzT5KZgbkVbEyW80ZQEfVHA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Elasticsearch-Hadoop repository plugin Cloudera Hadoop 2.0.0-cdh4.6.0
My ES cluster nodes and Hadoop nodes are not collocated. Light version does not works for me without putting enough correct versions of hadoop related jars. Right now I don't want to create my jar as Brent did and I don't want to install hadoop or copy jars on the es nodes either . Right now I can wait until my hadoop cluster upgrade to a version that es-hadoop supports. Or there is a version that is compatible to the hadoop My ES nodes talk to . Thanks, On Monday, September 1, 2014 8:50:25 AM UTC-7, Mateusz Kaczynski wrote: > > (Much delayed) thank you Costin. > > Indeed, on Ubuntu, changing ES_CLASSPATH to include hadoop and hadoop/lib > directories in /etc/default/elasticsearch (and exporting it in > /etc/init.d/elasticsearch) and installing light plugin version did work. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ca1465d4-d617-4623-9de3-35bd69e6f8fb%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Elasticsearch-Hadoop repository plugin Cloudera Hadoop 2.0.0-cdh4.6.0
(Much delayed) thank you Costin. Indeed, on Ubuntu, changing ES_CLASSPATH to include hadoop and hadoop/lib directories in /etc/default/elasticsearch (and exporting it in /etc/init.d/elasticsearch) and installing light plugin version did work. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f3dc7f6a-3dc0-4793-af8e-ea4390540204%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Elasticsearch-Hadoop repository plugin Cloudera Hadoop 2.0.0-cdh4.6.0
(Much delayed) thank you Costin.
Indeed, on Ubuntu, changing ES_CLASSPATH to include hadoop and hadoop/lib
directories in /etc/default/elasticsearch (and exporting it in
/etc/init.d/elasticsearch) and installing light plugin version did work.
On Thursday, 14 August 2014 20:59:39 UTC, Costin Leau wrote:
>
> Hi,
>
> The hdfs repository relies on vanilla Hadoop 2.2 since that's the official
> stable version of Yarn. Since you are using a
> different
> Hadoop version, use the 'light' version as explained in the docs - this
> contains only the repository-hdfs, without the
> Hadoop dependency
> (since you already have it installed).
>
> In other words, both the error that you see as well as the 2.0.1
> (regarding JobLocalizer) seems to be related to the
> differences between
> vanilla Hadoop 2.2 and the one you are using.
>
> Hope this helps,
>
> On 8/14/14 7:36 PM, Mateusz Kaczynski wrote:
> > I'm trying to get es-hadoop repository plugin working on our hadoop
> 2.0.0-cdh4.6.0 distribution and it seems like I'm
> > quite lost.
> >
> > I installed plugin's -hadoop2 version on the machines on our hadoop
> cluster (which also run our stage elasticsearch nodes).
> >
> > When attempting to create a repository on one of the datanodes with:
> >
> > curl -XPUT 1.0.0.1:9200/_snapshot/hdfs -d '{"type":"hdfs",
> "settings": {"uri": "hdfs://1.0.0.10:54310",
> > "path":"/es_backup"}}'
> >
> >
> > I end up with the logs being filled with the following error:
> > Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message contained an invalid tag (zero).
> > at
> com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
>
>
> > at
> com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
> > at
> org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.(RpcHeaderProtos.java:1398)
>
>
> > at
> org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.(RpcHeaderProtos.java:1362)
>
>
> > at
> org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1492)
>
>
> > at
> org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1487)
>
>
> > at
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
>
> > at
> com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
>
>
> > at
> com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
>
>
> > at
> com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
>
>
> > at
> com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
>
> > at
> org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
>
>
> > at
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:996)
> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
> >
> > Is it possible that this is caused by the incompatible hadoop versions
> (2.2 used by plugin with 2.0 being installed) and
> > it is necessary to get it build with downgraded version?
> >
> > Also, to build the jar, is it just
> >
> > gradle build -Pdistro=hadoopYarn ?
> >
> > Because release 2.0.1 does not quite work for me as it fails to find
> JobLocalizer.class.
> >
> > Regards,
> > Mateusz
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to
> > elasticsearc...@googlegroups.com elasticsearch+unsubscr...@googlegroups.com >.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/elasticsearch/5aff7e2a-eb3e-4bb8-8698-05fec6a67e87%40googlegroups.com
>
> > <
> https://groups.google.com/d/msgid/elasticsearch/5aff7e2a-eb3e-4bb8-8698-05fec6a67e87%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
>
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> Costin
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4ae0b343-72aa-459e-930e-559852c5d310%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Elasticsearch-Hadoop repository plugin Cloudera Hadoop 2.0.0-cdh4.6.0
Hi,
The hdfs repository relies on vanilla Hadoop 2.2 since that's the official stable version of Yarn. Since you are using a
different
Hadoop version, use the 'light' version as explained in the docs - this contains only the repository-hdfs, without the
Hadoop dependency
(since you already have it installed).
In other words, both the error that you see as well as the 2.0.1 (regarding JobLocalizer) seems to be related to the
differences between
vanilla Hadoop 2.2 and the one you are using.
Hope this helps,
On 8/14/14 7:36 PM, Mateusz Kaczynski wrote:
I'm trying to get es-hadoop repository plugin working on our hadoop
2.0.0-cdh4.6.0 distribution and it seems like I'm
quite lost.
I installed plugin's -hadoop2 version on the machines on our hadoop cluster
(which also run our stage elasticsearch nodes).
When attempting to create a repository on one of the datanodes with:
curl -XPUT 1.0.0.1:9200/_snapshot/hdfs -d '{"type":"hdfs", "settings": {"uri":
"hdfs://1.0.0.10:54310",
"path":"/es_backup"}}'
I end up with the logs being filled with the following error:
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message
contained an invalid tag (zero).
at
com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.(RpcHeaderProtos.java:1398)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.(RpcHeaderProtos.java:1362)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1492)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1487)
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
at
com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:996)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
Is it possible that this is caused by the incompatible hadoop versions (2.2
used by plugin with 2.0 being installed) and
it is necessary to get it build with downgraded version?
Also, to build the jar, is it just
gradle build -Pdistro=hadoopYarn ?
Because release 2.0.1 does not quite work for me as it fails to find
JobLocalizer.class.
Regards,
Mateusz
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
elasticsearch+unsubscr...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5aff7e2a-eb3e-4bb8-8698-05fec6a67e87%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/53ED232C.7080201%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Elasticsearch-Hadoop repository plugin Cloudera Hadoop 2.0.0-cdh4.6.0
I'm trying to get es-hadoop repository plugin working on our hadoop
2.0.0-cdh4.6.0 distribution and it seems like I'm quite lost.
I installed plugin's -hadoop2 version on the machines on our hadoop cluster
(which also run our stage elasticsearch nodes).
When attempting to create a repository on one of the datanodes with:
> curl -XPUT 1.0.0.1:9200/_snapshot/hdfs -d '{"type":"hdfs", "settings":
> {"uri": "hdfs://1.0.0.10:54310", "path":"/es_backup"}}'
I end up with the logs being filled with the following error:
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
message contained an invalid tag (zero).
at
com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.(RpcHeaderProtos.java:1398)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.(RpcHeaderProtos.java:1362)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1492)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1487)
at
com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
at
com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:996)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
Is it possible that this is caused by the incompatible hadoop versions (2.2
used by plugin with 2.0 being installed) and it is necessary to get it
build with downgraded version?
Also, to build the jar, is it just
> gradle build -Pdistro=hadoopYarn ?
Because release 2.0.1 does not quite work for me as it fails to find
JobLocalizer.class.
Regards,
Mateusz
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5aff7e2a-eb3e-4bb8-8698-05fec6a67e87%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
