If ES is not running, getting an exception is expected since one will get a
connectivity error.
As for logging, make sure you properly configure log4j for your Hadoop environment - it depends on what version you are
using and what libraries.
If the job is complete, you can always tests the results by querying ES for the
data that was just indexed.
On 4/28/14 9:20 AM, Chetana wrote:
I am using now ES 1.1.1 and as before es-hadoop 1.3.0.M3. But even with latest
ES I am unable to index.
I don't see any log pertaining to es-hadoop in any of the log file and also not
on console. Also there is no exception
while running the job and the job completes successfully
But if elasticsearch server is not running, the job throws an exception and
does not run
/hadoop-yarn/staging/user1/.staging/job_1398663730568_0001
14/04/28 11:18:27 ERROR security.UserGroupInformation:
PriviledgedActionException as:user1 (auth:SIMPLE)
cause:java.io.IOException: Out of nodes and retries; caught exception
Exception in thread "main" java.io.IOException: Out of nodes and retries;
caught exception
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:81)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:221)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:205)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:209)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:103)
Please someone suggest me how to troubleshoot this issue. After this excise I
need to index user log content to ES
Thanks,
On Thursday, April 24, 2014 3:38:38 PM UTC+5:30, Costin Leau wrote:
Try es-hadoop 1.3.0.M3 - if you enable logging as indicated in the docs
I've mentioned you should be seeing all the
activity - namely the connection being made, the data being transmitted,
the reply, etc....
On 4/24/14 12:21 PM, Chetana wrote:
> Yes, I am able to search but not able to index. I am using 1.3.0.M2. The
json file size is just 28bytes.
> I corrected port/host and some log setting. With all these chnages I am
to run the application, but the indexing is
> still not happening
> I dont see any error messages in log files.
> I am using a single node cluster and running both elastic search and
hadoop on the same system.
>
> On Thursday, April 24, 2014 10:30:22 AM UTC+5:30, Costin Leau wrote:
>
> Hi,
>
> If I understand correctly, you can read data from ES (through
es-hadoop) but you cannot write to it - am I correct? Can
> you confirm that you are using the latest es-hadoop, namely 1.3.0.M3?
> How big is the JSON file you are trying to index? Do you see any
activity in the console?
>
> There are various ways in which you can monitor activity - in ES you
can monitor the console or use Marvel [1], in
> ES-hadoop you can enable logging [3] and see how the job progresses.
Try starting with a small file to have a short
> feedback loop and once things get ironed out, try your actual desired
file.
>
>
> A few notes:
> - In general we recommend using the latest stable version of
Elasticsearch - 0.90.2 is quite old and unless you have a
> strong reason to stay on it, I highly recommend upgrading to 1.1.1
or, in the worst case scenario, ES 0.90.13.
> - you have repeating code - you set "es.nodes" and "es.port" twice
> - you are copying the local file to the destination filesystem
(presumably HDFS) which is okay but typically this is
> done outside the job launch
> - if you are using Hadoop MRv2, consider switching to MRv1. es-hadoop
supports both modes but the latter is easier to
> use and still the one the vendors recommend by default
> - since you are using Hadoop 2 from Hortonworks, you might want to
upgrade to their latest HDP (2.1) release.
>
> Hope this helps,
>
> [1]http://www.elasticsearch.org/overview/marvel/
<http://www.elasticsearch.org/overview/marvel/>
<http://www.elasticsearch.org/overview/marvel/
<http://www.elasticsearch.org/overview/marvel/>>
>
[2]http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/logging.html
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/logging.html>
>
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/logging.html
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/logging.html>>
>
> On 4/24/14 7:30 AM, Chetana wrote:
> > I am using elasticsearch-hadoop plugin
(https://github.com/elasticsearch/elasticsearch-hadoop
<https://github.com/elasticsearch/elasticsearch-hadoop>
<https://github.com/elasticsearch/elasticsearch-hadoop
<https://github.com/elasticsearch/elasticsearch-hadoop>>) and
> trying to index some
> > documents. Iam using elasticsearch version 0.90.2 and Hadoop
Hortonworks 2.2.0. The search functionality works fine, but
> > while indexing application hangs
> >https://github.com/elasticsearch/elasticsearch-hadoop
<https://github.com/elasticsearch/elasticsearch-hadoop>
<https://github.com/elasticsearch/elasticsearch-hadoop
<https://github.com/elasticsearch/elasticsearch-hadoop>>
> > The json file location is passed as a command line arguemnt and
below is the indexing code snippet
> > Configuration conf = new Configuration();
> > conf.setBoolean("mapred.map.tasks.speculative.execution", false);
> > conf.setBoolean("mapred.reduce.tasks.speculative.execution",
false);
> > conf.setInt("mapred.min.split.size",40);
> > conf.set("es.resource", "test/test");
> > conf.set("es.nodes", "localhost");
> > conf.set("es.port", "9200");
> > conf.set("es.input.json", "yes");
> > conf.set("es.nodes", "localhost");
> > conf.set("es.port", "9200");
> >
> > Job job = Job.getInstance(conf);
> > job.setMapperClass(Mapper.class);
> > job.setInputFormatClass(TextInputFormat.class);
> > job.setOutputFormatClass(EsOutputFormat.class);
> > job.setMapOutputKeyClass(LongWritable.class);
> > job.setMapOutputValueClass(Text.class);
> > Path jarPath = new Path(args[0]);
> > FileSystem fs = FileSystem.get(conf);
> > Path dst = new Path(fs.getHomeDirectory(), jarPath.getName());
> > fs.copyFromLocalFile(false, true, jarPath, dst);
> > FileInputFormat.setInputPaths(job, dst);
> >
> > job.waitForCompletion(true);
> > Am I missing anything, pls help
> >
> > --
> > You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it,
send an email to
> >[email protected] <javascript:>
<mailto:[email protected] <javascript:>
<javascript:>>.
> > To view this discussion on the web visit
>
>https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com>
>
<https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com>>
> >
<https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com?utm_medium=email&utm_source=footer>
>
<https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
>
> > For more options, visithttps://groups.google.com/d/optout
<http://groups.google.com/d/optout> <https://groups.google.com/d/optout
<https://groups.google.com/d/optout>>.
>
> --
> Costin
>
> --
> You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to
>[email protected] <javascript:>
<mailto:[email protected] <javascript:>>.
> To view this discussion on the web visit
>https://groups.google.com/d/msgid/elasticsearch/58142f74-93a0-4899-9ef6-1dec468b6c39%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/58142f74-93a0-4899-9ef6-1dec468b6c39%40googlegroups.com>
>
<https://groups.google.com/d/msgid/elasticsearch/58142f74-93a0-4899-9ef6-1dec468b6c39%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/58142f74-93a0-4899-9ef6-1dec468b6c39%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> For more options, visithttps://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected]
<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6397166a-c3f2-41a5-9ce7-50b09c237ad9%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/6397166a-c3f2-41a5-9ce7-50b09c237ad9%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/535DFDA8.80103%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
