Hey, I've recently started using Elasticsearch for Spark (Scala application). I've added elasticsearch-spark_2.10 version 2.1.0.BUILD-SNAPSHOT to my Spark application pom file, and used org.apache.spark.rdd.RDD[String].saveJsonToEs() to send documents to Elasticsearch. When the documents are loaded to Elasticsearch, my metadata fields (e.g id, index, etc.) are being loaded as part of the _source field. Is there a way to exclude them from the _source? I've tried using the new "es.mapping.exclude" configuration property (added in this commit <https://github.com/elasticsearch/elasticsearch-hadoop/commit/aae4f0460a23bac9567ea2ad335c74245a1ba069> - that's why I needed to take the latest build rather than using version 2.1.0.Beta3), but it doesn't seem to have any affect (although I'm not sure it's even possible to exclude fields I'm using for mapping, e.g "es.mapping.id").
A code snippet (I'm using a single-node Elasticsearch cluster for testing purposes and running the Spark app from my desktop) : val conf = new SparkConf()... conf.set("es.index.auto.create", "false") conf.set("es.nodes.discovery", "false") conf.set("es.nodes", "XXX:9200") conf.set("es.update.script", "XXX") conf.set("es.update.script.params", "param1:events") conf.set("es.update.retry.on.conflict" , "2") conf.set("es.write.operation", "upsert") conf.set("es.input.json", "true") val documentsRdd = ... documentsRdd.saveJsonToEs("test/user", scala.collection.Map("es.mapping.id" -> "_id", "es.mapping.exclude" -> "_id")) The JSON looks like that : { "_id": "XXXX", "_type": "user", "_index": "test", "params": { "events": [ { ... } ] } Thanks! } -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8055055f-8787-492b-97f4-144b2a7f7fce%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.