_timestamp doesn't work in kibana even enabled in the mapping

[Xinli Shang]

I enabled _timestamp  in the mapping as below. I also queried the mapping
back and confirmed it is set like below.

"_timestamp" : { "enabled" : true,"store" : "yes"},

But in Kibana, the timefilter which relies on the _timestamp doesn't work.
In the _source, I don't see _timpstamp is automatically created for each
document.

Any idea where is wrong?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHsnNRgw8uUH6CHNmUZ6jyrb2XFxUnu4rUQojg1AXA1G2zXbbA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: performance issues

[AlexR]

How many docs do you expect your histogram will aggregate? Most of your 111M? 
If so with just one shard and one thread doing the work it is bound to be 
pretty slow.

Also have you tried moving your not missing filter out of the agg into the 
query filter and also just using > 0 instead of not missing. Also reducing 
precision of the timestamp could possible help

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f8e36514-6f2a-4283-9f75-312aab3a2fea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: performance issues

[msbreuer]

>
> Can you give examples of the documents and the queries you use?
>
> Docs look like this:

{
"duration" : "74",
"caller" : "128287",
"session_id" : "12312",
"id" : "901",
"position" : "1",
"parameters" : "",
"operation" : "export",
"timestamp" : "2014-01-15T14:17:06.245+01:00"
}

And this is the query:

{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"range": {
"timestamp": {
"ge": "2013-12-31T23:00:00.000Z",
"lt": "2014-12-31T23:00:00.000Z"
}
}
},
{
"term": {
"_type": "export-op"
}
}
]
}
}
}
},
"aggregations": {
"duration-not-empty": {
"filter": {
"bool": {
"must": [
{
"not": {
"filter": {
"missing": {
"field": "duration"
}
}
}
},
{
"range": {
"duration": {
"gt": "0"
}
}
}
]
}
},
"aggregations": {
"durations": {
"date_histogram": {
"field": "timestamp",
"interval": "1M"
},
"aggregations": {
"duration-stats": {
"extended_stats": {
"field": "duration"
}
}
}
}
}
}
},
"size": 0
}

The extended_stats seems to be expensive. Instead using min/max/avg will 
save some seconds. Also excluding _type from query has no effect, but i 
think thos restriction is applied to url, too.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/83e0e265-2fa1-46a8-ad5f-c64981919fd5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: default mappings for new indexes that match a pattern?

[Kevin Burton]

nevermind.  Looks like index templates are exactly what I need! 

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html

On Saturday, December 6, 2014 2:27:11 PM UTC-8, Kevin Burton wrote:
>
> I'm creating a new index, one per day.
>
> I'd like to set the default mappings used by wildcard.
>
> So this way the first time an index is created , the new mappings are used 
> based on the wildcard.
>
> I have some fields which are tokens/URLs that I don't want analyzed.
>
> So what I want to do is create a mapping saying that these aren't analyzed 
> and then when a new index is created, the default mapping is applied with 
> all my type settings.
>
> Is this possible to do by wildcard?  For example, "content*" or do I have 
> to have one per day? One per day wouldn't be super fun.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5a213c51-0d9f-4795-b80c-dee3289c4ad4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES Fails Initializing when Index is on NAS

[Yingkai Gao]

Okay.  I've got a super weird solution to this problem and I really think 
it does not make any sense.
After running jstack with the pid of ES, I found that the java process 
blocked at a native method lock0 as the following stack log shows.
"main" #1 prio=5 os_prio=0 tid=0x0238f000 nid=0x63ef runnable 
[0x4110e000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:90)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1067)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:1155)
at org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:169)
- locked <0xc10e2898> (a org.apache.lucene.store.NativeFSLock)
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:83)
at 
org.elasticsearch.node.internal.InternalNode.(InternalNode.java:157)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:70)
at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:203)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)

Then I just ran pstack with the pid.  While I got nothing usefully, the 
blocked process magically continued and successfully obtained the file lock 
and recovered index.
I believe cause of the problem is that the configuration of file locking on 
the NAS/NFS is inappropriate.  Because I don't have the privilege to 
re-configure it, I'm just going to use this solution but still need some 
explanation...

On Friday, December 5, 2014 4:41:19 PM UTC-5, Yingkai Gao wrote:
>
> I'm using ElasticSearch-1.4.0 on CentOS-5.6.  It works well if I set the 
> index path.data on local/NFS.  However, if I set the path.data on a NAS 
> folder, the node keep stuck after initializing and plugins loading.
>
> It looks like this link 
> 
>  
> a lot, but I'm using NAS.  The node did create the index directories on the 
> NAS path, but it just stopped there.
>
> The starting log of ElasticSearch is:
> [2014-12-05 16:36:12,745][INFO ][node ] [kyle] 
> version[1.4.0], pid[4819], build[bc94bd8/2014-11-05T14:26:12Z]
> [2014-12-05 16:36:12,747][INFO ][node ] [kyle] 
> initializing ...
> [2014-12-05 16:36:12,755][INFO ][plugins  ] [kyle] loaded 
> [], sites []
>
> Using command df, the file system information of the mounted NAS is:
> nas-2-25:/exports/volume02
>
> Anyone has any idea how to fix this problem.  I know it is not suggested 
> to use NAS for index, but I have to because the infrastructure problem of 
> our cluster.
>
> Thanks,
> Kyle
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/cb26e818-a0fa-44e3-8552-2532a951829e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: I Don't Know Why but My Hq Plugin Show Deleted Documents more than 0 ,and I m Not Deleted any Document yet via REST

[David Pilato]

Did you update some docs?

David

> Le 6 déc. 2014 à 23:21, Sandy  a écrit :
> 
> I Don't Know Why but My Hq Plugin Show Deleted Documents more than 0 ,and I m 
> Not Deleted any Document yet via REST.Any idea and what it means.I m new in 
> ElasticSearch.
> Thanx 
> Sandy  
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/2eaadf11-b426-4bf8-a121-30afe5b3005e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/D71CBED6-1FA0-4E9D-97F2-9AD7E82BE1F8%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Re: performance issues

[joergpra...@gmail.com]

Can you give examples of the documents and the queries you use?

Jörg
Am 06.12.2014 14:27 schrieb :

> While testing with large amount of data I am reaching the point of first
> performance issues. The initial situation as follows:
>
> - one ES node with 8GB heap assigned
> - one index with 110.000.000 documents
> - 78.000.000 docs assigned to single _type
> - histogram data and a sub-type of cardinality 20
> - histogram query using aggregation over sub-type runs fast (< 3 seconds)
> - histogram over whole index,_type but ignoring subtype run up to 50
> seconds (index is cold), on warm index the same query takes 10-12 seconds
> - there are currently no writes to index and index is optimized (this may
> change in future)
> - only one shard of size 30GB
> - one index per month
> - data for about 3-4 month into past
> - java 1.7u55 and es 1.4.1
>
> My requirements:
> - query should return in <3 seconds
> - one index per month (or probably week)
> - continuous adding new data to recent index
>
> Questions:
> 1. How to find out the bottleneck of this query?
> 2. What are the tuning options?
> 3. Over time there are serious heap issues: the heap grows up and many
> time is spent in parallel+full gc. After restarting the used heap is about
> 3GB and several GCs will hold it on this level. But over hours the usage
> grows up towards 8GB and full gc is not able to cleanup here. A restart is
> required. Why?
>
> regards,
> markus
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/fac725cd-d6f6-4fc2-b274-4af374695d82%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF%2BLuqo98a0HHOhxBEajdbUtw_3w_tx88WXf6QeD_ZM5A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

default mappings for new indexes that match a pattern?

[Kevin Burton]

I'm creating a new index, one per day.

I'd like to set the default mappings used by wildcard.

So this way the first time an index is created , the new mappings are used 
based on the wildcard.

I have some fields which are tokens/URLs that I don't want analyzed.

So what I want to do is create a mapping saying that these aren't analyzed 
and then when a new index is created, the default mapping is applied with 
all my type settings.

Is this possible to do by wildcard?  For example, "content*" or do I have 
to have one per day? One per day wouldn't be super fun.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4151b580-3c10-4c4a-a016-14d1c7d66da0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I Don't Know Why but My Hq Plugin Show Deleted Documents more than 0 ,and I m Not Deleted any Document yet via REST

[Sandy]

I Don't Know Why but My Hq Plugin Show Deleted Documents more than 0 ,and I 
m Not Deleted any Document yet via REST.Any idea and what it means.I m new 
in ElasticSearch.
Thanx 
Sandy  

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2eaadf11-b426-4bf8-a121-30afe5b3005e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregations with dates & scripts

[tog]

Hi Jun

Thanks a lot. That helped a lot. I also had to change:
   doc['end_date'].value by doc[\"end_date\"]

Cheers
Guillaume

On 6 December 2014 at 12:44, Jun Ohtani  wrote:

> Hi Guillaume,
>
> You should use doc[‘end_date’].value instead of doc[‘end_date’] .
> See :
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_document_fields
>
> And you should use 360 instead of 3600.
> Date type is a long number representing UTC milliseconds since the epoch.
>
> See :
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date
>
> I hope that those help you out.
>
>
> 
> Jun Ohtani
> joht...@gmail.com
> blog : http://blog.johtani.info
> twitter : http://twitter.com/johtani
>
> > 2014/12/04 7:47、Guillaume Alleon  のメール：
> >
> > Hi
> >
> > I have  a mapping which is having the following 2 properties:
> > "start_date": {
> > "format": "dateOptionalTime",
> > "type": "date"
> > },
> > "end_date": {
> > "format": "dateOptionalTime",
> > "type": "date"
> > },
> >
> > I want to create an histogram for durations with 1 hour intervals by
> doing  this:
> >
> > curl -XGET 'http://localhost:9200/myindex/travels/_search_search?pretty'
> -d '
> > {
> >
> > "aggs": {
> >
> > "duration": {
> >
> > "histogram": {
> >
> > "interval": 1,
> >
> > "script": "(doc['end_date']-doc['start_date'])/3600"
> >
> > }
> >
> > }
> >
> > }
> >
> > }'
> >
> > I got the following error:
> >
> > GroovyScriptExecutionException[MissingPropertyException[No such
> property: end_date for class: Script1]];
> }{[B4xwr8FCSnOLkrNAVhgHMg][gequest][2]:
> QueryPhaseExecutionException[[myindex][2]:
> query[ConstantScore(cache(_type:travels))],from[0],size[10]: Query Failed
> [Failed to execute main query]]; nested:
> GroovyScriptExecutionException[MissingPropertyException[No such property:
> end_date for class: Script1]]; }
> >
> > I have the same error for start_date.
> >
> >
> >
> > Any idea what I am doing wrong ?
> >
> >
> >
> > Cheers
> >
> > Guillaume
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to elasticsearch+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/6ca86563-5881-4004-af35-a258841ec326%40googlegroups.com
> .
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/EO8RWlNRJjA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/C3D0F8B9-CAA7-46A9-ACC0-505D6C0F22F8%40gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
PGP KeyID: 2048R/EA31CFC9  subkeys.pgp.net

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAE1e5yEfkKeghMRyahRA_C4c0u7k7pcsmuX2Bv3nQgxzujY5Hw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

On startup it takes a long time for Elasticsearch to write the first message to a log file

[Daniel Feinstein]

Hi,

I have ES version 1.4.1 installed on RedHat 6.5. The data is relatively 
small about 3.5Gb. The cluster contains two nodes sometimes it takes about 
30 mins to start the first node after shutdown. No log messages are written 
to the log during this time.

Is it a normal behavior? Is it possible to improve it?

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a8f0e002-671f-4fd0-ab06-87b070005fd7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES Fails Initializing when Index is on NAS

[Yingkai Gao]

I set the logger level to TRACE and get the following loggings.  It seems 
ES failed obtaining the node lock.  But I still have no idea how to solve 
it.

[2014-12-06 10:22:00,101][INFO ][node ] [kyle] 
version[1.4.0], pid[18889], build[bc94bd8/2014-11-05T14:26:12Z]
[2014-12-06 10:22:00,102][INFO ][node ] [kyle] 
initializing ...
[2014-12-06 10:22:00,103][DEBUG][node ] [kyle] using 
home [/bos/usr0/yingkaig/downloads/elasticsearch-debug], config 
[/bos/usr0/yingkaig/downloads/elasticsearch-debug/config], data 
[[/bos/tmp11/yingkaig/index.debug]], logs [/bos/usr0/yingkaig/temp/logs], 
work [/bos/usr0/yingkaig/downloads/elasticsearch-debug/work], plugins 
[/bos/usr0/yingkaig/downloads/elasticsearch-debug/plugins]
[2014-12-06 10:22:00,111][DEBUG][plugins  ] [kyle] 
[/bos/usr0/yingkaig/downloads/elasticsearch-debug/plugins] directory does 
not exist.
[2014-12-06 10:22:00,112][INFO ][plugins  ] [kyle] loaded 
[], sites []
[2014-12-06 10:22:00,152][DEBUG][common.compress.lzf  ] using encoder 
[VanillaChunkDecoder] and decoder[{}]
[2014-12-06 10:22:00,157][TRACE][env  ] [kyle] 
obtaining node lock on /bos/tmp11/yingkaig/index.debug/debug/nodes/0 ...


On Saturday, December 6, 2014 1:38:38 AM UTC-5, Mark Walkom wrote:
>
> Start the service, do a ps|grep java and grab the command that is called 
> then run it manually.
> You may find it returns more info you can use to resolve your issue.
>
> Otherwise you can turn up the logging levels in logging.yml.
>
> On 6 December 2014 at 08:41, Yingkai Gao > 
> wrote:
>
>> I'm using ElasticSearch-1.4.0 on CentOS-5.6.  It works well if I set the 
>> index path.data on local/NFS.  However, if I set the path.data on a NAS 
>> folder, the node keep stuck after initializing and plugins loading.
>>
>> It looks like this link 
>> 
>>  
>> a lot, but I'm using NAS.  The node did create the index directories on the 
>> NAS path, but it just stopped there.
>>
>> The starting log of ElasticSearch is:
>> [2014-12-05 16:36:12,745][INFO ][node ] [kyle] 
>> version[1.4.0], pid[4819], build[bc94bd8/2014-11-05T14:26:12Z]
>> [2014-12-05 16:36:12,747][INFO ][node ] [kyle] 
>> initializing ...
>> [2014-12-05 16:36:12,755][INFO ][plugins  ] [kyle] loaded 
>> [], sites []
>>
>> Using command df, the file system information of the mounted NAS is:
>> nas-2-25:/exports/volume02
>>
>> Anyone has any idea how to fix this problem.  I know it is not suggested 
>> to use NAS for index, but I have to because the infrastructure problem of 
>> our cluster.
>>
>> Thanks,
>> Kyle
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/9388166f-d250-4bb2-9f94-44f40dda338f%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/47a5a4d1-7bf9-4e89-ba95-1fd206d5da87%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: understaning terms syntax

[Nikolas Everett]

Also, its usually better to use a "match" query if you want to analyze the
query rather than "query_string".  Query string exposes a huge array of
syntax which is both useful and terribly dangerous.  Users can write
regexes and huge range queries and fuzzy queries that use much much more
cpu and ram than the term and phrase queries you expect them to write.

Nik

On Sat, Dec 6, 2014 at 10:18 AM, GX  wrote:

>
> Hi Jun,
>
> thanks for the reply will look into this
>
> GX
>
>
> On Saturday, December 6, 2014 3:53:10 PM UTC+2, Jun Ohtani wrote:
>>
>> Hi GX,
>>
>> “term" query does NOT analyzed your string.
>> See: http://www.elasticsearch.org/guide/en/elasticsearch/
>> reference/current/query-dsl-term-query.html#query-dsl-term-query
>>
>> On the other hand, “query_string” query analyze your string.
>>
>> If you want to know the difference, you can use _validate API.
>> See : http://www.elasticsearch.org/guide/en/elasticsearch/
>> reference/current/search-validate.html#search-validate
>>
>> Example commands are as follows:
>>
>> GET hoge/fuga/_validate/query?explain
>> {
>>"query": {
>> "term": {
>> "ac": "3A822F3B-3ECF-4463-98F86DF6DE28EC5C"
>> }
>> }
>> }
>>
>> GET  hoge/fuga/_validate/query?explain
>> {
>>   "query": {
>>  "query_string" : {
>>   "default_field" : "ac",
>>   "query" : "3A822F3B-3ECF-4463-98F86DF6DE28EC5C"
>>   }
>>   }
>> }
>>
>> I hope that those help you out.
>>
>>
>> 
>> Jun Ohtani
>> joh...@gmail.com
>> blog : http://blog.johtani.info
>> twitter : http://twitter.com/johtani
>>
>> > 2014/11/21 14:45、GX  のメール：
>> >
>> > Hi All
>> >
>> > Im having the following scenario (elasticsearch 1.0):
>> > the query
>> >"query": {
>> > "term": {
>> > "ac": "3A822F3B-3ECF-4463-98F86DF6DE28EC5C"
>> > }
>> > }
>> >
>> > yields no results
>> >
>> > but this works
>> >
>> > "query": {
>> >  "query_string" : {
>> >   "default_field" : "ac",
>> >   "query" : "3A822F3B-3ECF-4463-98F86DF6DE28EC5C"
>> >   }
>> >   }
>> >
>> > the problem is when I combine it with a "must_not" or "not" filter I
>> still get the same results
>> >
>> > what is the correct syntax I need?
>> >
>> > Thanks
>> >
>> > GX
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "elasticsearch" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to elasticsearc...@googlegroups.com.
>> > To view this discussion on the web visit https://groups.google.com/d/
>> msgid/elasticsearch/a1f4c507-f85a-4ebd-b71f-4962259abf5a%
>> 40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/9dfabbb3-ff0f-459c-bfbc-b69632076dda%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0rFhUPwAowxdi4Su_U%2Bvw1J6Ptr_r-EVe2f1jHqH-80w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: mapping definition help (copy_to, dates)

[Nikolas Everett]

Our you can always transform in you client application.  The advantage of
transform is that it is done _post_ source like copy_to.  Meaning is you
like the original format for disk space and highlighting purposes you
should use transform.  If you don't, transform in your app.

Nik

On Sat, Dec 6, 2014 at 10:04 AM, Jun Ohtani  wrote:

> Hi Jeff,
>
> elasticsearch doesn’t have “timestamp” type. elasticsearch have only date
> type.
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date
>
> And you should use “transform” instead of “copy_to”.
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-transform.html
>
> Example mapping using transform:
> {
>   "mappings": {
> "utilization": {
> "dynamic": "false",
>
> "transform" : {
> "script" : "if (!ctx._source['cst_observed'].empty &&
> !ctx._source['date_observed'].empty) ctx._source['my_ts'] =
> ctx._source['cst_observed'] + ' '+ ctx._source['date_observed']",
> "lang": "groovy"
> },
> "properties": {
> "cst_observed": {"type": "string"}
> ,"date_observed": {"type": "string"}
> ,"inbound_mbps": {"type": "float"}
> ,"inbound_percent": {"type": "float"}
> ,"ip_addr": {"type": "ip"}
> ,"outbound_mbps": {"type": "float"}
> ,"outbound_percent": {"type": "float"}
> ,"provider": {"type": "string"}
> ,"reporting_begin": {"type": "date"}
> ,"reporting_end": {"type": "date"}
> ,"speed": {"type": "float"}
> ,"my_ts":{"type": "date", "format": "HH:mm:ss mm/dd/"}
>   }
> }
>   }
> }
>
>
> That is only example.
> I hope that those help you out.
>
> 
> Jun Ohtani
> joht...@gmail.com
> blog : http://blog.johtani.info
> twitter : http://twitter.com/johtani
>
> > 2014/11/18 13:40、Jeff Fogarty  のメール：
> >
> > I'm looking to analyze network utilization data supplied by our vendor.
> One challenge is the observed data and time are in separate fields.
> >
> > I like to combine the fields date_observed and cst_observed into a
> date/time field.  Is this possible?
> >
> > Below defines that mapping I've come up with,  I get a few errors with
> the below
> >
> > No handler for type [timestamp] declared on fields
> > my_ts
> > cst_observed
> >
> > { "mappings": {
> > "utilization": {
> > "dynamic": "false", "properties": {
> > "cst_observed": {"type": "timestamp", "copy_to": "my_ts"}
> > ,"date_observed": {"type": "date", "copy_to": "my_ts"}
> > ,"inbound_mbps": {"type": "float"}
> > ,"inbound_percent": {"type": "float"}
> > ,"ip_addr": {"type": "ip"}
> > ,"outbound_mbps": {"type": "float"}
> > ,"outbound_percent": {"type": "float"}
> > ,"provider": {"type": "string"}
> > ,"reporting_begin": {"type": "date"}
> > ,"reporting_end": {"type": "date"}
> > ,"speed": {"type": "float"}
> > ,"my_ts":{"type": "timestamp", "format": "HH:mm:ss
> mm/dd/"}
> > }
> > }
> > }
> > }'
> >
> > Thanks,
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to elasticsearch+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/0109138a-4f04-4f66-9ab5-2f7f85fc9a77%40googlegroups.com
> .
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/B58F56BA-643C-4B75-98E3-35DB5282FE73%40gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2AY%2BKGoKxBFPSsayrGJ12VhLekFctmAvdCZo5SB95SdA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: understaning terms syntax

[GX]

Hi Jun, 

thanks for the reply will look into this 

GX


On Saturday, December 6, 2014 3:53:10 PM UTC+2, Jun Ohtani wrote:
>
> Hi GX, 
>
> “term" query does NOT analyzed your string. 
> See: 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-term-query.html#query-dsl-term-query
>  
>
> On the other hand, “query_string” query analyze your string. 
>
> If you want to know the difference, you can use _validate API. 
> See : 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-validate.html#search-validate
>  
>
> Example commands are as follows: 
>
> GET hoge/fuga/_validate/query?explain 
> { 
>"query": { 
> "term": { 
> "ac": "3A822F3B-3ECF-4463-98F86DF6DE28EC5C" 
> } 
> } 
> } 
>
> GET  hoge/fuga/_validate/query?explain 
> { 
>   "query": { 
>  "query_string" : { 
>   "default_field" : "ac", 
>   "query" : "3A822F3B-3ECF-4463-98F86DF6DE28EC5C" 
>   } 
>   } 
> } 
>
> I hope that those help you out. 
>
>
>  
> Jun Ohtani 
> joh...@gmail.com  
> blog : http://blog.johtani.info 
> twitter : http://twitter.com/johtani 
>
> > 2014/11/21 14:45、GX > のメール： 
> > 
> > Hi All 
> > 
> > Im having the following scenario (elasticsearch 1.0): 
> > the query 
> >"query": { 
> > "term": { 
> > "ac": "3A822F3B-3ECF-4463-98F86DF6DE28EC5C" 
> > } 
> > } 
> > 
> > yields no results 
> > 
> > but this works 
> > 
> > "query": { 
> >  "query_string" : { 
> >   "default_field" : "ac", 
> >   "query" : "3A822F3B-3ECF-4463-98F86DF6DE28EC5C" 
> >   } 
> >   } 
> > 
> > the problem is when I combine it with a "must_not" or "not" filter I 
> still get the same results 
> > 
> > what is the correct syntax I need? 
> > 
> > Thanks 
> > 
> > GX 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "elasticsearch" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to elasticsearc...@googlegroups.com . 
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/a1f4c507-f85a-4ebd-b71f-4962259abf5a%40googlegroups.com.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9dfabbb3-ff0f-459c-bfbc-b69632076dda%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: mapping definition help (copy_to, dates)

[Jun Ohtani]

Hi Jeff,

elasticsearch doesn’t have “timestamp” type. elasticsearch have only date type.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date

And you should use “transform” instead of “copy_to”.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-transform.html

Example mapping using transform:
{ 
  "mappings": {
"utilization": {
"dynamic": "false", 

"transform" : {
"script" : "if (!ctx._source['cst_observed'].empty && 
!ctx._source['date_observed'].empty) ctx._source['my_ts'] = 
ctx._source['cst_observed'] + ' '+ ctx._source['date_observed']",
"lang": "groovy"
},
"properties": {
"cst_observed": {"type": "string"}
,"date_observed": {"type": "string"}
,"inbound_mbps": {"type": "float"}
,"inbound_percent": {"type": "float"}
,"ip_addr": {"type": "ip"}
,"outbound_mbps": {"type": "float"}
,"outbound_percent": {"type": "float"}
,"provider": {"type": "string"}
,"reporting_begin": {"type": "date"}
,"reporting_end": {"type": "date"}
,"speed": {"type": "float"}
,"my_ts":{"type": "date", "format": "HH:mm:ss mm/dd/"}
  }
}
  }
}


That is only example. 
I hope that those help you out.


Jun Ohtani
joht...@gmail.com
blog : http://blog.johtani.info
twitter : http://twitter.com/johtani

> 2014/11/18 13:40、Jeff Fogarty  のメール：
> 
> I'm looking to analyze network utilization data supplied by our vendor.  One 
> challenge is the observed data and time are in separate fields.  
> 
> I like to combine the fields date_observed and cst_observed into a date/time 
> field.  Is this possible?
> 
> Below defines that mapping I've come up with,  I get a few errors with the 
> below
> 
> No handler for type [timestamp] declared on fields 
> my_ts
> cst_observed
> 
> { "mappings": {
> "utilization": {
> "dynamic": "false", "properties": {
> "cst_observed": {"type": "timestamp", "copy_to": "my_ts"}
> ,"date_observed": {"type": "date", "copy_to": "my_ts"}
> ,"inbound_mbps": {"type": "float"}
> ,"inbound_percent": {"type": "float"}
> ,"ip_addr": {"type": "ip"}
> ,"outbound_mbps": {"type": "float"}
> ,"outbound_percent": {"type": "float"}
> ,"provider": {"type": "string"}
> ,"reporting_begin": {"type": "date"}
> ,"reporting_end": {"type": "date"}
> ,"speed": {"type": "float"}
> ,"my_ts":{"type": "timestamp", "format": "HH:mm:ss 
> mm/dd/"}
> }
> }
> }
> }'
> 
> Thanks,
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/0109138a-4f04-4f66-9ab5-2f7f85fc9a77%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/B58F56BA-643C-4B75-98E3-35DB5282FE73%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: understaning terms syntax

[Jun Ohtani]

Hi GX,

“term" query does NOT analyzed your string.
See: 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-term-query.html#query-dsl-term-query

On the other hand, “query_string” query analyze your string.

If you want to know the difference, you can use _validate API.
See : 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-validate.html#search-validate

Example commands are as follows:

GET hoge/fuga/_validate/query?explain
{
   "query": {
"term": {
"ac": "3A822F3B-3ECF-4463-98F86DF6DE28EC5C"
}
}
}

GET  hoge/fuga/_validate/query?explain
{
  "query": {
 "query_string" : {
  "default_field" : "ac",
  "query" : "3A822F3B-3ECF-4463-98F86DF6DE28EC5C"
  }
  }
}

I hope that those help you out.



Jun Ohtani
joht...@gmail.com
blog : http://blog.johtani.info
twitter : http://twitter.com/johtani

> 2014/11/21 14:45、GX  のメール：
> 
> Hi All
> 
> Im having the following scenario (elasticsearch 1.0):
> the query 
>"query": {
> "term": {
> "ac": "3A822F3B-3ECF-4463-98F86DF6DE28EC5C"
> }
> }
> 
> yields no results
> 
> but this works
> 
> "query": {
>  "query_string" : {
>   "default_field" : "ac",
>   "query" : "3A822F3B-3ECF-4463-98F86DF6DE28EC5C"
>   }
>   }
> 
> the problem is when I combine it with a "must_not" or "not" filter I still 
> get the same results
> 
> what is the correct syntax I need?
> 
> Thanks
> 
> GX
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/a1f4c507-f85a-4ebd-b71f-4962259abf5a%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/634B3D51-39A8-469D-938A-A6F5BCDA2D67%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch saving integer field as String

[Jun Ohtani]

Hi Rashmi,

Do not you set “type” in Logstash configuration?

If you don’t set, logstash use “logs” for “index_type” in output elasticsearch.
See : 
https://github.com/elasticsearch/logstash/blob/1.4/lib/logstash/outputs/elasticsearch.rb#L300

Then you should change “logs” instead of “user” in your mappings in template.



Jun Ohtani
joht...@gmail.com
blog : http://blog.johtani.info
twitter : http://twitter.com/johtani

> 2014/11/25 3:39、Rashmi Gogia  のメール：
> 
> Hi,
> 
> Could some one help me figure out what is the issue with below Logstash 
> Configuration /mapping template . ES seems to ignore the mappings specified 
> in the template and stores everything as string.
> 
> I am using Logstash csv plugin to create index. My logstash conf file has csv 
> filter as below:
> 
> filter {
>   csv {
> columns => ["ESpeedRef","ProcessName",TotalMilliSec,AnyFloatNumber]
> separator => ","
>   }
> 
> output {
> elasticsearch{
> protocol => "http"
> host => "localhost"
> index => "test_index"
>   }
> }
> 
> 
> 
> Below json template is used to define field mappings. I use curl to send 
> template to ES.
> 
> {"template": "test_index",
> "settings": {
> "number_of_shards": 1
> },
> "mappings": {
> "user": {
> "properties": {
> "ESpeedRef": {"type" : "string"},
> "ProcessName": {"type" : "string"},
> "TotalMilliSec": {"type" : "integer"},
> "AnyFloatNumber": {"type" : "float"}
> }
> }
> }
> }
> 
> 
> When creating index, Elastic Search does apply this template. I can confirm 
> this my looking at below console stmts:
> 
> [2014-11-24 13:22:51,099][INFO ][cluster.metadata ] [Captain America] 
> [
> test_index] creating index, cause [auto(bulk api)], shards [1]/[1], mappings
> [user]
> [2014-11-24 13:22:51,146][INFO ][cluster.metadata ] [Captain America] 
> [
> test_index] update_mapping [logs] (dynamic)
> 
> 
> In Kibana i see the fields TotalMilliSec and AnyFloatNumber are saved as 
> string.. why??
> 
> Thanks
> Rashmi
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/568cb316-5946-4fbc-ad87-f65a413b990d%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/EE908F38-2DED-4624-A4CD-2E3DC9774C65%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

performance issues

[msbreuer]

While testing with large amount of data I am reaching the point of first 
performance issues. The initial situation as follows:

- one ES node with 8GB heap assigned
- one index with 110.000.000 documents
- 78.000.000 docs assigned to single _type
- histogram data and a sub-type of cardinality 20
- histogram query using aggregation over sub-type runs fast (< 3 seconds)
- histogram over whole index,_type but ignoring subtype run up to 50 
seconds (index is cold), on warm index the same query takes 10-12 seconds
- there are currently no writes to index and index is optimized (this may 
change in future)
- only one shard of size 30GB
- one index per month 
- data for about 3-4 month into past
- java 1.7u55 and es 1.4.1

My requirements:
- query should return in <3 seconds
- one index per month (or probably week)
- continuous adding new data to recent index

Questions:
1. How to find out the bottleneck of this query?
2. What are the tuning options? 
3. Over time there are serious heap issues: the heap grows up and many time 
is spent in parallel+full gc. After restarting the used heap is about 3GB 
and several GCs will hold it on this level. But over hours the usage grows 
up towards 8GB and full gc is not able to cleanup here. A restart is 
required. Why?

regards,
markus

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/fac725cd-d6f6-4fc2-b274-4af374695d82%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregations with dates & scripts

[Jun Ohtani]

Hi Guillaume,

You should use doc[‘end_date’].value instead of doc[‘end_date’] .
See : 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_document_fields

And you should use 360 instead of 3600. 
Date type is a long number representing UTC milliseconds since the epoch.

See : 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date

I hope that those help you out.



Jun Ohtani
joht...@gmail.com
blog : http://blog.johtani.info
twitter : http://twitter.com/johtani

> 2014/12/04 7:47、Guillaume Alleon  のメール：
> 
> Hi 
> 
> I have  a mapping which is having the following 2 properties:
> "start_date": {
> "format": "dateOptionalTime",
> "type": "date"
> },
> "end_date": {
> "format": "dateOptionalTime",
> "type": "date"
> },
> 
> I want to create an histogram for durations with 1 hour intervals by doing  
> this:
> 
> curl -XGET 'http://localhost:9200/myindex/travels/_search_search?pretty' -d '
> {
> 
> "aggs": {
> 
> "duration": {
> 
> "histogram": {
> 
> "interval": 1,
> 
> "script": "(doc['end_date']-doc['start_date'])/3600"
> 
> }
> 
> }
> 
> }
> 
> }'
> 
> I got the following error:
> 
> GroovyScriptExecutionException[MissingPropertyException[No such property: 
> end_date for class: Script1]]; }{[B4xwr8FCSnOLkrNAVhgHMg][gequest][2]: 
> QueryPhaseExecutionException[[myindex][2]: 
> query[ConstantScore(cache(_type:travels))],from[0],size[10]: Query Failed 
> [Failed to execute main query]]; nested: 
> GroovyScriptExecutionException[MissingPropertyException[No such property: 
> end_date for class: Script1]]; }
> 
> I have the same error for start_date.
> 
> 
> 
> Any idea what I am doing wrong ?
> 
> 
> 
> Cheers
> 
> Guillaume
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/6ca86563-5881-4004-af35-a258841ec326%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/C3D0F8B9-CAA7-46A9-ACC0-505D6C0F22F8%40gmail.com.
For more options, visit https://groups.google.com/d/optout.