A template wont help you here. I mean it's good to use and you should use them. But once the schema is defined you can't change it. This is no different then any database.
Your best bet here is to do a bit of data cleansing/normalizing. If you know that the field is date field and sometimes the date is different then you have to try to convert it proper date format before inserting. Especially if you are trying to push it all into one field. Even if you use wildcards in templates like suggested above, you would have to know that the date is different to have it pushed to another field. On Friday, 6 February 2015 06:41:49 UTC-5, Itamar Syn-Hershko wrote: > > You mean something like dynamic templates? > http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-templates-to-avoid-rigorous-mappings > > -- > > Itamar Syn-Hershko > http://code972.com | @synhershko <https://twitter.com/synhershko> > Freelance Developer & Consultant > Lucene.NET committer and PMC member > > On Fri, Feb 6, 2015 at 1:39 PM, Paul Kavanagh <pkav...@shopkeep.com > <javascript:>> wrote: > >> Hi all, >> We're having a MapperParsingException problem with some field values when >> we get when we use the JSON Filter for Logstash to explode out a JSON >> document to Elasticsearch fields. >> >> In 99.9% of cases, certain of these fields are either blank, or contain >> dates in the format of yyyy-mm-dd. This allows ES to dynamically map this >> field to type dateOptionalTime. >> >> However, we occasionally see non-standard date formats in these fields, >> which our main service can handle fine, but which throws a >> MapperParsingException in Elasticsearch - such are here: >> >> >> >> [2015-02-06 10:46:50,679][WARN ][cluster.action.shard ] [logging- >> production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2] >> received shard failed for [logstash-2015.02.06][2], node[ >> GZpltBjAQUqGyp2B1SLz_g], [R], s[INITIALIZING], indexUUID [BEdTwj- >> QRuOZB713YAQwvA], reason [Failed to start shard, message [ >> RecoveryFailedException[[logstash-2015.02.06][2]: Recovery failed from [ >> logging-production-elasticsearch-ip-xxx-xxx-xxx-82][IALW-92RReiLffQjSL3I- >> g][logging-production-elasticsearch-ip-xxx-xxx-xxx-82][inet[ip-xxx-xxx- >> xxx-82.ec2.internal/xxx.xxx.xxx.82:9300]]{max_local_storage_nodes=1, >> aws_availability_zone=us-east-1e, aws_az=us-east-1e} into [logging- >> production-elasticsearch-ip-xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz_g][ >> logging-production-elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx.xxx.xxx. >> 148.ec2.internal/xxx.xxx.xxx.148:9300]]{max_local_storage_nodes=1, >> aws_availability_zone=us-east-1c, aws_az=us-east-1c}]; nested: >> RemoteTransportException[[logging-production-elasticsearch-ip-xxx-xxx-xxx >> -82][inet[/xxx.xxx.xxx.82:9300]][internal:index/shard/recovery/ >> start_recovery]]; nested: RecoveryEngineException[[logstash-2015.02.06][2 >> ] Phase[2] Execution failed]; nested: RemoteTransportException[[logging- >> production-elasticsearch-ip-xxx-xxx-xxx-148][inet[/xxx.xxx.xxx.148:9300 >> ]][internal:index/shard/recovery/translog_ops]]; nested: >> MapperParsingException[failed to parse [apiservice.logstash.@fields. >> parameters.start_time]]; nested: MapperParsingException[failed to parse >> date field [Feb 5 2015 12:00 AM], tried both date format [ >> dateOptionalTime], and timestamp number with locale []]; nested: >> IllegalArgumentException[Invalid format: "Feb 5 2015 12:00 AM"]; ]] >> >> 2015-02-06 10:46:53,685][WARN ][cluster.action.shard ] [logging- >> production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2] >> received shard failed for [logstash-2015.02.06][2], node[ >> GZpltBjAQUqGyp2B1SLz_g], [R], s[INITIALIZING], indexUUID [BEdTwj- >> QRuOZB713YAQwvA], reason [master [logging-production-elasticsearch-ip-xxx >> -xxx-xxx-148][GZpltBjAQUqGyp2B1SLz_g][logging-production-elasticsearch-ip >> -xxx-xxx-xxx-148][inet[ip-xxx-xxx-xxx-148.ec2.internal/xxx.xxx.xxx.148: >> 9300]]{max_local_storage_nodes=1, aws_availability_zone=us-east-1c, >> aws_az=us-east-1c} marked shard as initializing, but shard is marked as >> failed, resend shard failure] >> >> >> Our planned solution was to create a template for Logstash indices that >> will set these fields to string. But as the field above isn't the only >> culprit, and more may be added overtime, it makes more sense to create a >> template to map all fields under apiservice.logstash.@fields.parameters.* >> to be string. (We never need to query on user entered data, but it's great >> to have logged for debugging) >> >> Is it possible to do this with a template? I could not find a way to do >> this via the template documentation on the ES site. >> >> Any guidance would be great! >> >> Thanks, >> -Paul >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/130d8b7e-6e8b-43e4-baf8-d17fca8bc7e6%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
