You mean something like dynamic templates? http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-templates-to-avoid-rigorous-mappings
-- Itamar Syn-Hershko http://code972.com | @synhershko <https://twitter.com/synhershko> Freelance Developer & Consultant Lucene.NET committer and PMC member On Fri, Feb 6, 2015 at 1:39 PM, Paul Kavanagh <pkavan...@shopkeep.com> wrote: > Hi all, > We're having a MapperParsingException problem with some field values when > we get when we use the JSON Filter for Logstash to explode out a JSON > document to Elasticsearch fields. > > In 99.9% of cases, certain of these fields are either blank, or contain > dates in the format of yyyy-mm-dd. This allows ES to dynamically map this > field to type dateOptionalTime. > > However, we occasionally see non-standard date formats in these fields, > which our main service can handle fine, but which throws a > MapperParsingException in Elasticsearch - such are here: > > > > [2015-02-06 10:46:50,679][WARN ][cluster.action.shard ] [logging- > production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2] > received shard failed for [logstash-2015.02.06][2], node[ > GZpltBjAQUqGyp2B1SLz_g], [R], s[INITIALIZING], indexUUID [BEdTwj- > QRuOZB713YAQwvA], reason [Failed to start shard, message [ > RecoveryFailedException[[logstash-2015.02.06][2]: Recovery failed from [ > logging-production-elasticsearch-ip-xxx-xxx-xxx-82][IALW-92RReiLffQjSL3I-g > ][logging-production-elasticsearch-ip-xxx-xxx-xxx-82][inet[ip-xxx-xxx-xxx- > 82.ec2.internal/xxx.xxx.xxx.82:9300]]{max_local_storage_nodes=1, > aws_availability_zone=us-east-1e, aws_az=us-east-1e} into [logging- > production-elasticsearch-ip-xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz_g][ > logging-production-elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx.xxx.xxx. > 148.ec2.internal/xxx.xxx.xxx.148:9300]]{max_local_storage_nodes=1, > aws_availability_zone=us-east-1c, aws_az=us-east-1c}]; nested: > RemoteTransportException[[logging-production-elasticsearch-ip-xxx-xxx-xxx- > 82][inet[/xxx.xxx.xxx.82:9300]][internal:index/shard/recovery/ > start_recovery]]; nested: RecoveryEngineException[[logstash-2015.02.06][2] > Phase[2] Execution failed]; nested: RemoteTransportException[[logging- > production-elasticsearch-ip-xxx-xxx-xxx-148][inet[/xxx.xxx.xxx.148:9300]][ > internal:index/shard/recovery/translog_ops]]; nested: > MapperParsingException[failed to parse [apiservice.logstash.@fields. > parameters.start_time]]; nested: MapperParsingException[failed to parse > date field [Feb 5 2015 12:00 AM], tried both date format [dateOptionalTime > ], and timestamp number with locale []]; nested: IllegalArgumentException[ > Invalid format: "Feb 5 2015 12:00 AM"]; ]] > > 2015-02-06 10:46:53,685][WARN ][cluster.action.shard ] [logging- > production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2] > received shard failed for [logstash-2015.02.06][2], node[ > GZpltBjAQUqGyp2B1SLz_g], [R], s[INITIALIZING], indexUUID [BEdTwj- > QRuOZB713YAQwvA], reason [master [logging-production-elasticsearch-ip-xxx- > xxx-xxx-148][GZpltBjAQUqGyp2B1SLz_g][logging-production-elasticsearch-ip- > xxx-xxx-xxx-148][inet[ip-xxx-xxx-xxx-148.ec2.internal/xxx.xxx.xxx.148:9300 > ]]{max_local_storage_nodes=1, aws_availability_zone=us-east-1c, aws_az=us- > east-1c} marked shard as initializing, but shard is marked as failed, > resend shard failure] > > > Our planned solution was to create a template for Logstash indices that > will set these fields to string. But as the field above isn't the only > culprit, and more may be added overtime, it makes more sense to create a > template to map all fields under apiservice.logstash.@fields.parameters.* > to be string. (We never need to query on user entered data, but it's great > to have logged for debugging) > > Is it possible to do this with a template? I could not find a way to do > this via the template documentation on the ES site. > > Any guidance would be great! > > Thanks, > -Paul > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZviZWbhJC83fB-3cm5qmcsuH-ScOo4x-ghS9BZ9t28HCA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
