I think you have something there. I have come up with this: curl -XPUT localhost:9200/_template/template_1 -d ' { "template" : "logstash-*", "order" : 0, "settings" : { "number_of_shards" : 15 }, "mappings" : { "dynamic_templates":[ {"apiservice_logstash":{ "match":"apiservice.logstash.@fields.parameters.*", "match_mapping_type":"dateOptionalTime", "mapping":{ "type":"string", "analyzer":"english" } } } ] } } '
However... When I try to post it, Elasticsearch throws: {"error":"ElasticsearchIllegalArgumentException[Malformed mappings section for type [dynamic_templates], should include an inner object describing the mapping]","status":400} i've tried a few things, but it doesn't seem to like my mappings block for some reason. Any idea why? On Friday, February 6, 2015 at 11:41:49 AM UTC, Itamar Syn-Hershko wrote: > > You mean something like dynamic templates? > http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-templates-to-avoid-rigorous-mappings > > -- > > Itamar Syn-Hershko > http://code972.com | @synhershko <https://twitter.com/synhershko> > Freelance Developer & Consultant > Lucene.NET committer and PMC member > > On Fri, Feb 6, 2015 at 1:39 PM, Paul Kavanagh <pkav...@shopkeep.com > <javascript:>> wrote: > >> Hi all, >> We're having a MapperParsingException problem with some field values when >> we get when we use the JSON Filter for Logstash to explode out a JSON >> document to Elasticsearch fields. >> >> In 99.9% of cases, certain of these fields are either blank, or contain >> dates in the format of yyyy-mm-dd. This allows ES to dynamically map this >> field to type dateOptionalTime. >> >> However, we occasionally see non-standard date formats in these fields, >> which our main service can handle fine, but which throws a >> MapperParsingException in Elasticsearch - such are here: >> >> >> >> [2015-02-06 10:46:50,679][WARN ][cluster.action.shard ] [logging- >> production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2] >> received shard failed for [logstash-2015.02.06][2], node[ >> GZpltBjAQUqGyp2B1SLz_g], [R], s[INITIALIZING], indexUUID [BEdTwj- >> QRuOZB713YAQwvA], reason [Failed to start shard, message [ >> RecoveryFailedException[[logstash-2015.02.06][2]: Recovery failed from [ >> logging-production-elasticsearch-ip-xxx-xxx-xxx-82][IALW-92RReiLffQjSL3I- >> g][logging-production-elasticsearch-ip-xxx-xxx-xxx-82][inet[ip-xxx-xxx- >> xxx-82.ec2.internal/xxx.xxx.xxx.82:9300]]{max_local_storage_nodes=1, >> aws_availability_zone=us-east-1e, aws_az=us-east-1e} into [logging- >> production-elasticsearch-ip-xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz_g][ >> logging-production-elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx.xxx.xxx. >> 148.ec2.internal/xxx.xxx.xxx.148:9300]]{max_local_storage_nodes=1, >> aws_availability_zone=us-east-1c, aws_az=us-east-1c}]; nested: >> RemoteTransportException[[logging-production-elasticsearch-ip-xxx-xxx-xxx >> -82][inet[/xxx.xxx.xxx.82:9300]][internal:index/shard/recovery/ >> start_recovery]]; nested: RecoveryEngineException[[logstash-2015.02.06][2 >> ] Phase[2] Execution failed]; nested: RemoteTransportException[[logging- >> production-elasticsearch-ip-xxx-xxx-xxx-148][inet[/xxx.xxx.xxx.148:9300 >> ]][internal:index/shard/recovery/translog_ops]]; nested: >> MapperParsingException[failed to parse [apiservice.logstash.@fields. >> parameters.start_time]]; nested: MapperParsingException[failed to parse >> date field [Feb 5 2015 12:00 AM], tried both date format [ >> dateOptionalTime], and timestamp number with locale []]; nested: >> IllegalArgumentException[Invalid format: "Feb 5 2015 12:00 AM"]; ]] >> >> 2015-02-06 10:46:53,685][WARN ][cluster.action.shard ] [logging- >> production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2] >> received shard failed for [logstash-2015.02.06][2], node[ >> GZpltBjAQUqGyp2B1SLz_g], [R], s[INITIALIZING], indexUUID [BEdTwj- >> QRuOZB713YAQwvA], reason [master [logging-production-elasticsearch-ip-xxx >> -xxx-xxx-148][GZpltBjAQUqGyp2B1SLz_g][logging-production-elasticsearch-ip >> -xxx-xxx-xxx-148][inet[ip-xxx-xxx-xxx-148.ec2.internal/xxx.xxx.xxx.148: >> 9300]]{max_local_storage_nodes=1, aws_availability_zone=us-east-1c, >> aws_az=us-east-1c} marked shard as initializing, but shard is marked as >> failed, resend shard failure] >> >> >> Our planned solution was to create a template for Logstash indices that >> will set these fields to string. But as the field above isn't the only >> culprit, and more may be added overtime, it makes more sense to create a >> template to map all fields under apiservice.logstash.@fields.parameters.* >> to be string. (We never need to query on user entered data, but it's great >> to have logged for debugging) >> >> Is it possible to do this with a template? I could not find a way to do >> this via the template documentation on the ES site. >> >> Any guidance would be great! >> >> Thanks, >> -Paul >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6ab11e08-873b-4610-9573-d8781327aeb2%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.