Try es-hadoop 1.3.0.M3 - if you enable logging as indicated in the docs I've mentioned you should be seeing all the
activity - namely the connection being made, the data being transmitted, the reply, etc....
On 4/24/14 12:21 PM, Chetana wrote:
Yes, I am able to search but not able to index. I am using 1.3.0.M2. The json
file size is just 28bytes.
I corrected port/host and some log setting. With all these chnages I am to run
the application, but the indexing is
still not happening
I dont see any error messages in log files.
I am using a single node cluster and running both elastic search and hadoop on
the same system.
On Thursday, April 24, 2014 10:30:22 AM UTC+5:30, Costin Leau wrote:
Hi,
If I understand correctly, you can read data from ES (through es-hadoop)
but you cannot write to it - am I correct? Can
you confirm that you are using the latest es-hadoop, namely 1.3.0.M3?
How big is the JSON file you are trying to index? Do you see any activity
in the console?
There are various ways in which you can monitor activity - in ES you can
monitor the console or use Marvel [1], in
ES-hadoop you can enable logging [3] and see how the job progresses. Try
starting with a small file to have a short
feedback loop and once things get ironed out, try your actual desired file.
A few notes:
- In general we recommend using the latest stable version of Elasticsearch
- 0.90.2 is quite old and unless you have a
strong reason to stay on it, I highly recommend upgrading to 1.1.1 or, in
the worst case scenario, ES 0.90.13.
- you have repeating code - you set "es.nodes" and "es.port" twice
- you are copying the local file to the destination filesystem (presumably
HDFS) which is okay but typically this is
done outside the job launch
- if you are using Hadoop MRv2, consider switching to MRv1. es-hadoop
supports both modes but the latter is easier to
use and still the one the vendors recommend by default
- since you are using Hadoop 2 from Hortonworks, you might want to upgrade
to their latest HDP (2.1) release.
Hope this helps,
[1] http://www.elasticsearch.org/overview/marvel/
<http://www.elasticsearch.org/overview/marvel/>
[2]
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/logging.html
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/logging.html>
On 4/24/14 7:30 AM, Chetana wrote:
> I am using elasticsearch-hadoop plugin
(https://github.com/elasticsearch/elasticsearch-hadoop
<https://github.com/elasticsearch/elasticsearch-hadoop>) and
trying to index some
> documents. Iam using elasticsearch version 0.90.2 and Hadoop Hortonworks
2.2.0. The search functionality works fine, but
> while indexing application hangs
>https://github.com/elasticsearch/elasticsearch-hadoop
<https://github.com/elasticsearch/elasticsearch-hadoop>
> The json file location is passed as a command line arguemnt and below is
the indexing code snippet
> Configuration conf = new Configuration();
> conf.setBoolean("mapred.map.tasks.speculative.execution", false);
> conf.setBoolean("mapred.reduce.tasks.speculative.execution", false);
> conf.setInt("mapred.min.split.size",40);
> conf.set("es.resource", "test/test");
> conf.set("es.nodes", "localhost");
> conf.set("es.port", "9200");
> conf.set("es.input.json", "yes");
> conf.set("es.nodes", "localhost");
> conf.set("es.port", "9200");
>
> Job job = Job.getInstance(conf);
> job.setMapperClass(Mapper.class);
> job.setInputFormatClass(TextInputFormat.class);
> job.setOutputFormatClass(EsOutputFormat.class);
> job.setMapOutputKeyClass(LongWritable.class);
> job.setMapOutputValueClass(Text.class);
> Path jarPath = new Path(args[0]);
> FileSystem fs = FileSystem.get(conf);
> Path dst = new Path(fs.getHomeDirectory(), jarPath.getName());
> fs.copyFromLocalFile(false, true, jarPath, dst);
> FileInputFormat.setInputPaths(job, dst);
>
> job.waitForCompletion(true);
> Am I missing anything, pls help
>
> --
> You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to
>[email protected] <javascript:>
<mailto:[email protected] <javascript:>>.
> To view this discussion on the web visit
>https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com>
>
<https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> For more options, visithttps://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected]
<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/58142f74-93a0-4899-9ef6-1dec468b6c39%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/58142f74-93a0-4899-9ef6-1dec468b6c39%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5358E2A6.1020105%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
