I think you have something there. I have come up with this:
curl -XPUT localhost:9200/_template/template_1 -d '
{
"template" : "logstash-*",
"order" : 0,
"settings" : {
"number_of_shards" : 15
},
"mappings" : {
"dynamic_templates":[
{"apiservice_logstash":{
"match":"apiservice.logstash.@fields.parameters.*",
"match_mapping_type":"dateOptionalTime",
"mapping":{
"type":"string",
"analyzer":"english"
}
}
}
]
}
}
'
However... When I try to post it, Elasticsearch throws:
{"error":"ElasticsearchIllegalArgumentException[Malformed mappings section
for type [dynamic_templates], should include an inner object describing the
mapping]","status":400}
i've tried a few things, but it doesn't seem to like my mappings block for
some reason.
Any idea why?
On Friday, February 6, 2015 at 11:41:49 AM UTC, Itamar Syn-Hershko wrote:
>
> You mean something like dynamic templates?
> http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-templates-to-avoid-rigorous-mappings
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko <https://twitter.com/synhershko>
> Freelance Developer & Consultant
> Lucene.NET committer and PMC member
>
> On Fri, Feb 6, 2015 at 1:39 PM, Paul Kavanagh <pkav...@shopkeep.com
> <javascript:>> wrote:
>
>> Hi all,
>> We're having a MapperParsingException problem with some field values when
>> we get when we use the JSON Filter for Logstash to explode out a JSON
>> document to Elasticsearch fields.
>>
>> In 99.9% of cases, certain of these fields are either blank, or contain
>> dates in the format of yyyy-mm-dd. This allows ES to dynamically map this
>> field to type dateOptionalTime.
>>
>> However, we occasionally see non-standard date formats in these fields,
>> which our main service can handle fine, but which throws a
>> MapperParsingException in Elasticsearch - such are here:
>>
>>
>>
>> [2015-02-06 10:46:50,679][WARN ][cluster.action.shard ] [logging-
>> production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
>> received shard failed for [logstash-2015.02.06][2], node[
>> GZpltBjAQUqGyp2B1SLz_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
>> QRuOZB713YAQwvA], reason [Failed to start shard, message [
>> RecoveryFailedException[[logstash-2015.02.06][2]: Recovery failed from [
>> logging-production-elasticsearch-ip-xxx-xxx-xxx-82][IALW-92RReiLffQjSL3I-
>> g][logging-production-elasticsearch-ip-xxx-xxx-xxx-82][inet[ip-xxx-xxx-
>> xxx-82.ec2.internal/xxx.xxx.xxx.82:9300]]{max_local_storage_nodes=1,
>> aws_availability_zone=us-east-1e, aws_az=us-east-1e} into [logging-
>> production-elasticsearch-ip-xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz_g][
>> logging-production-elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx.xxx.xxx.
>> 148.ec2.internal/xxx.xxx.xxx.148:9300]]{max_local_storage_nodes=1,
>> aws_availability_zone=us-east-1c, aws_az=us-east-1c}]; nested:
>> RemoteTransportException[[logging-production-elasticsearch-ip-xxx-xxx-xxx
>> -82][inet[/xxx.xxx.xxx.82:9300]][internal:index/shard/recovery/
>> start_recovery]]; nested: RecoveryEngineException[[logstash-2015.02.06][2
>> ] Phase[2] Execution failed]; nested: RemoteTransportException[[logging-
>> production-elasticsearch-ip-xxx-xxx-xxx-148][inet[/xxx.xxx.xxx.148:9300
>> ]][internal:index/shard/recovery/translog_ops]]; nested:
>> MapperParsingException[failed to parse [apiservice.logstash.@fields.
>> parameters.start_time]]; nested: MapperParsingException[failed to parse
>> date field [Feb 5 2015 12:00 AM], tried both date format [
>> dateOptionalTime], and timestamp number with locale []]; nested:
>> IllegalArgumentException[Invalid format: "Feb 5 2015 12:00 AM"]; ]]
>>
>> 2015-02-06 10:46:53,685][WARN ][cluster.action.shard ] [logging-
>> production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
>> received shard failed for [logstash-2015.02.06][2], node[
>> GZpltBjAQUqGyp2B1SLz_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
>> QRuOZB713YAQwvA], reason [master [logging-production-elasticsearch-ip-xxx
>> -xxx-xxx-148][GZpltBjAQUqGyp2B1SLz_g][logging-production-elasticsearch-ip
>> -xxx-xxx-xxx-148][inet[ip-xxx-xxx-xxx-148.ec2.internal/xxx.xxx.xxx.148:
>> 9300]]{max_local_storage_nodes=1, aws_availability_zone=us-east-1c,
>> aws_az=us-east-1c} marked shard as initializing, but shard is marked as
>> failed, resend shard failure]
>>
>>
>> Our planned solution was to create a template for Logstash indices that
>> will set these fields to string. But as the field above isn't the only
>> culprit, and more may be added overtime, it makes more sense to create a
>> template to map all fields under apiservice.logstash.@fields.parameters.*
>> to be string. (We never need to query on user entered data, but it's great
>> to have logged for debugging)
>>
>> Is it possible to do this with a template? I could not find a way to do
>> this via the template documentation on the ES site.
>>
>> Any guidance would be great!
>>
>> Thanks,
>> -Paul
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearc...@googlegroups.com <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4932d1e2-9194-431a-8458-afd423866b21%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
