Yes, I am able to search but not able to index. I am using 1.3.0.M2. The json file size is just 28bytes. I corrected port/host and some log setting. With all these chnages I am to run the application, but the indexing is still not happening I dont see any error messages in log files. I am using a single node cluster and running both elastic search and hadoop on the same system.
On Thursday, April 24, 2014 10:30:22 AM UTC+5:30, Costin Leau wrote: > Hi, > > If I understand correctly, you can read data from ES (through es-hadoop) > but you cannot write to it - am I correct? Can > you confirm that you are using the latest es-hadoop, namely 1.3.0.M3? > How big is the JSON file you are trying to index? Do you see any activity > in the console? > > There are various ways in which you can monitor activity - in ES you can > monitor the console or use Marvel [1], in > ES-hadoop you can enable logging [3] and see how the job progresses. Try > starting with a small file to have a short > feedback loop and once things get ironed out, try your actual desired > file. > > > A few notes: > - In general we recommend using the latest stable version of Elasticsearch > - 0.90.2 is quite old and unless you have a > strong reason to stay on it, I highly recommend upgrading to 1.1.1 or, in > the worst case scenario, ES 0.90.13. > - you have repeating code - you set "es.nodes" and "es.port" twice > - you are copying the local file to the destination filesystem (presumably > HDFS) which is okay but typically this is > done outside the job launch > - if you are using Hadoop MRv2, consider switching to MRv1. es-hadoop > supports both modes but the latter is easier to > use and still the one the vendors recommend by default > - since you are using Hadoop 2 from Hortonworks, you might want to upgrade > to their latest HDP (2.1) release. > > Hope this helps, > > [1] http://www.elasticsearch.org/overview/marvel/ > [2] > http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/logging.html > > > On 4/24/14 7:30 AM, Chetana wrote: > > I am using elasticsearch-hadoop plugin ( > https://github.com/elasticsearch/elasticsearch-hadoop) and trying to > index some > > documents. Iam using elasticsearch version 0.90.2 and Hadoop Hortonworks > 2.2.0. The search functionality works fine, but > > while indexing application hangs > > https://github.com/elasticsearch/elasticsearch-hadoop > > The json file location is passed as a command line arguemnt and below is > the indexing code snippet > > Configuration conf = new Configuration(); > > conf.setBoolean("mapred.map.tasks.speculative.execution", false); > > conf.setBoolean("mapred.reduce.tasks.speculative.execution", false); > > conf.setInt("mapred.min.split.size",40); > > conf.set("es.resource", "test/test"); > > conf.set("es.nodes", "localhost"); > > conf.set("es.port", "9200"); > > conf.set("es.input.json", "yes"); > > conf.set("es.nodes", "localhost"); > > conf.set("es.port", "9200"); > > > > Job job = Job.getInstance(conf); > > job.setMapperClass(Mapper.class); > > job.setInputFormatClass(TextInputFormat.class); > > job.setOutputFormatClass(EsOutputFormat.class); > > job.setMapOutputKeyClass(LongWritable.class); > > job.setMapOutputValueClass(Text.class); > > Path jarPath = new Path(args[0]); > > FileSystem fs = FileSystem.get(conf); > > Path dst = new Path(fs.getHomeDirectory(), jarPath.getName()); > > fs.copyFromLocalFile(false, true, jarPath, dst); > > FileInputFormat.setInputPaths(job, dst); > > > > job.waitForCompletion(true); > > Am I missing anything, pls help > > > > -- > > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to > > [email protected] <javascript:> <mailto: > [email protected] <javascript:>>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com > > > < > https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > > For more options, visit https://groups.google.com/d/optout. > > -- > Costin > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/58142f74-93a0-4899-9ef6-1dec468b6c39%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
