Re: How many Java clients do I need?

[Andrew Gaydenko]

On Thursday, July 31, 2014 10:18:52 AM UTC+4, David Pilato wrote:
>
> Yeah. Only one client instance for the JVM.
>

Now I'm happy to be sure, thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/700c7505-5480-4059-94a0-0e0cfc29890f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How many Java clients do I need?

[David Pilato]

Yeah. Only one client instance for the JVM.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 31 juil. 2014 à 06:48, Andrew Gaydenko  a écrit :

As far as I understand, Java client instance is stateless, and it's methods are 
pure functions (I means "operating" methods rather those related to initial 
configuration just after instantiation). As a result, it is sufficient to have 
the only client for given cluster for given JVM. Is it true? Or - are there any 
benefits in, say, creating own client for every index?
-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d38de8b6-6ed9-4e3f-9dfc-c9dedd1293d4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5476E2C7-7804-4732-BAD3-FC91A3408406%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Re: How many Java clients do I need?

[Andrew Gaydenko]

I must to add the question is related to transport client, rather client 
node.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3267a8b5-f812-40f0-94bf-8258170ac75a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Recommendations needed for large ELK system design

[Mark Walkom]

1 - Looks ok, but why two replicas? You're chewing up disk for what reason?
Extra comments below.
2 - It's personal preference really and depends on how your end points send
to redis.
3 - 4GB for redis will cache quite a lot of data if you're only doing 50
events p/s (ie hours or even days based on what I've seen).
4 - No, spread it out to all the nodes. More on that below though.
5 - No it will handle that itself. Again, more on that below though.

Suggestions;
Set your indexes to (factors of) 6 shards, ie one per node, it spreads
query performance. I say "factors of" in that you can set it to 12 shards
per index to start and easily scale the node count and still spread the
load.
Split your stats and your log data into different indexes, it'll make
management and retention easier.
You can consider a master only node or (ideally) three that also handle
queries.
Preferably have an uneven number of master eligible nodes, whether you make
them VMs or physicals, that way you can ensure quorum is reached with
minimal fuss and stop split brain.
If you use VMs for master + query nodes then you might want to look at load
balancing the queries via an external service.

To give you an idea, we have a 27 node cluster - 3 masters that also handle
queries and 24 data nodes. Masters are 8GB with small disks, data nodes are
60GB (30 heap) and 512GB disk.
We're running with one replica and have 11TB of logging data. At a high
level we're running out of disk more than heap or CPU and we're very write
heavy, with an average of 1K events p/s and comparatively minimal reads.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 31 July 2014 01:35, Alex  wrote:

> Hello,
>
> We wish to set up an entire ELK system with the following features:
>
>- Input from Logstash shippers located on 400 Linux VMs. Only a
>handful of log sources on each VM.
>- Data retention for 30 days, which is roughly 2TB of data in indexed
>ES JSON form (not including replica shards)
>- Estimated input data rate of 50 messages per second at peak hours.
>Mostly short or medium length one-line messages but there will be Java
>traces and very large service responses (in the form of XML) to deal with
>too.
>- The entire system would be on our company LAN.
>- The stored data will be a mix of application logs (info, errors etc)
>and server stats (CPU, memory usage etc) and would mostly be accessed
>through Kibana.
>
> This is our current plan:
>
>- Have the LS shippers perform minimal parsing (but would do
>multiline). Have them point to two load-balanced servers containing Redis
>and LS indexers (which would do all parsing).
>- 2 replica shards for each index, which ramps the total data storage
>up to 6TB
>- ES cluster spread over 6 nodes. Each node is 1TB in size
>- LS indexers pointing to cluster.
>
> So I have a couple questions regarding the setup and would greatly
> appreciate the advice of someone with experience!
>
>1. Does the balance between the number of nodes, the number of replica
>shards, and storage size of each node seem about right? We use
>high-performance equipment and would expect minimal downtime.
>
>2. What is your recommendation for the system design of the LS
>indexers and Redis? I've seen various designs with each indexer assigned to
>a single Redis, or all indexers reading from all Redises.
>
>3. Leading from the previous question, what would your recommend data
>size for the Redis servers be?
>
>4. Not sure what to do about master/data nodes. Assuming all the nodes
>are on identical hardware would it be beneficial to have a node which is
>only a master which would only handle requests?
>
>5. Do we need to do any additional load balancing on the ES nodes?
>
> We are open to any and all suggestions. We have not yet committed to any
> particular design so can change if needed.
>
> Thank you for your time and responses,
> Alex
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/b0aee66a-35bb-4770-927b-d9c7e13ad9fc%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624ajES

How many Java clients do I need?

[Andrew Gaydenko]

As far as I understand, Java client instance is stateless, and it's methods 
are pure functions (I means "operating" methods rather those related to 
initial configuration just after instantiation). As a result, it is 
sufficient to have the only client for given cluster for given JVM. Is it 
true? Or - are there any benefits in, say, creating own client for every 
index?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d38de8b6-6ed9-4e3f-9dfc-c9dedd1293d4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Help needed understanding analyzer behavior

[Neko Escondido]

Hi Nikolas
Thank you very much for your feedback.  I was hoping to be able to search 
against the phone number field in normalized, original, number parts format.
If I modify the input into normalized format, then, search using 
original/number parts will not return the desired result... 
Or am I misunderstanding your suggestion?
Multi-field indexing is an option but that is to be avoided if possible (so 
that client executing query does not have to know all the possible field 
names a phone number field might be mapped)...
Once again, thank you very much for your feedback.  What I described above 
sounds possible using char filter/plugin?


On Wednesday, July 30, 2014 8:28:35 PM UTC-7, Nikolas Everett wrote:
>
> It's probably easier to do a char filter to remove all non digits. On the 
> other hand if you want to normalize numbers that sometimes contain area and 
> country code to numbers you'll probably want to do that outside of 
> elasticsearch or with a plugin. That gets difficult when you need to handle 
> non NANPA numbers. 
> On Jul 30, 2014 11:14 PM, "Neko Escondido"  > wrote:
>
>> Hello community,
>>
>> I'm having problem understanding how analyzer should work.  The result is 
>> different from what I expect.  :(
>>
>> I have created a custom analyzer to index phone number as below:
>>
>> "analysis" : {
>> "analyzer" : {
>> "phone" : {
>> "type": "custom",
>> "tokenizer":"phone_tokenizer",
>> "filter" : [ "phone_filter", "unique" ]
>> }
>> },
>>"tokenizer" : {
>> "phone_tokenizer" : {
>> "type" : "pattern",
>> "pattern":"\\s*[a-zA-Z]+\\s*"
>> 
>> }
>>},
>> "filter" : {
>>"phone_filter" : {
>> "type" : "word_delimiter",
>> "preserve_original" : 1,
>> "generate_number_parts" : 1,
>> "catenate_numbers" : 1
>>}
>> }
>>}
>>
>>
>>
>> The intention is to match:
>> Query Input: 
>>  111222, 111.222., 111-222-, or 111 222 , 
>> (111)222, 1-(111)-222-, etc. 
>> With records containing phone number such as:
>>  111.222., 111-222-, or 111 222 , (111)222, 
>> 1-(111)-222-, etc. 
>>
>> So with search input: (111)222 with queryType "matchPhraseQuery", I 
>> thought the query will return the records with phone number such as 
>> 111.222., 111-222-, etc. because input (111)222 would be 
>> analyzed into 111222, 111, and 222.
>> Given I have specified "catenate_numbers" in filter for my "phone" 
>> analyzer, I would expect that numbers the numbers that meets the following 
>> condition will be matched:
>> Match numbers that are indexed as ( 111 AND 222 ) OR 111222.
>> But result is no match.  
>>
>> Is my understanding incorrect?  With search input (111)222 using 
>> matchPhraseQuery, I thought it will match all numbers that has 111222 
>> as the concatenated value but it seems to match only with numbers whose 
>> number parts are 111 and 222... 
>>
>> Your feedback/help/input is greatly appreciated!!
>> Best regards
>>
>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/b0658b33-2efb-495a-8090-7cc12806a253%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2227f168-ec3d-4bad-95d0-09b2082f2c08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: "more like this" vs. mlt

[vineeth mohan]

Hello Peter ,

You have set these variable for the API and not the query , that is why its
working - min_term_freq=1 , min_doc_freq=1

Thanks
   Vineeth


On Thu, Jul 31, 2014 at 5:02 AM, Peter Li  wrote:

> I ran a query:
>
> curl -XGET
> "$url/ease/RadiologyResult/90642/_mlt?routing=07009409&mlt_fields=Observation.Value&min_term_freq=1&min_doc_freq=1&pretty"
>
> It worked and returned several documents. But if I ran this:
>
> curl -XGET "$url/ease/RadiologyResult/_search?routing=07009409&pretty" -d '
> {
>
> "query" : {
> "more_like_this" : {
> "fields": [
> "Observation.Value"
> ],
> "ids" : [ "90642" ]
>}
>}
> }'
>
> It returned nothing. Is there something I am missing ?
>
> Thanks in advance.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/ffa73983-ce04-4fd3-9786-ee09d3248d83%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5kGyJk%3DXn%2BNRj8bKxqOgTgQ3_U1p6D%3DA%3DEqih1u2v9eYQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch still scan all types in a index even if I specify a type

[panfei]

First, put some sample data:

curl -XPUT 'localhost:9200/testindex/action1/1?pretty' -d '
{
"title": "jumping tom",
"val": 101
}'

curl -XPUT 'localhost:9200/testindex/action2/1?pretty' -d '
{
"title": "jumping jerry",
"val": "test"
}'

as you can see, and the mapping is :

{
"action1" : {
"properties" : {
"val" : {
"type" : "long"
},
"title" : {
"type" : "string"
}
}
},
"action2" : {
"properties" : {
"val" : {
"type" : "string"
},
"title" : {
"type" : "string"
}
}
}
}

But when do a aggs action:

curl 'http://192.168.2.245:9200/testindex/action1/_search' -d '
{
"aggs": {
"vals": {
"terms": {
"field": "val"
}
}
}
}'


{
"took" : 37,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 4,
"failed" : 1,
"failures" : [
{
"index" : "testindex",
"shard" : 2,
"status" : 500,
"reason" : 
"RemoteTransportException[[a00][inet[/192.168.2.246:9300]][search/phase/query]];
nested: ElasticsearchException[java.lang.NumberFormatException: Invalid
shift value (84) in prefixCoded bytes (is encoded value really an INT?)];
nested: UncheckedExecutionException[java.lang.NumberFormatException:
Invalid shift value (84) in prefixCoded bytes (is encoded value really an
INT?)]; nested: NumberFormatException[Invalid shift value (84) in
prefixCoded bytes (is encoded value really an INT?)]; "
}
]
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [
]
},
"aggregations" : {
"vals" : {
"buckets" : [
]
}
}
}

The val field in action1 type is mapped to long, but it seems that ES still
scan the action2 type even if I specify the action1 type.

any advice to resolve this issue ? thanks.
-- 
不学习，不知道

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CA%2BJstLB_Md4w49%2BDW2O2OuLY6RBAR0DPz6rHLvb9WcLq8h3n6Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Help needed understanding analyzer behavior

[Nikolas Everett]

It's probably easier to do a char filter to remove all non digits. On the
other hand if you want to normalize numbers that sometimes contain area and
country code to numbers you'll probably want to do that outside of
elasticsearch or with a plugin. That gets difficult when you need to handle
non NANPA numbers.
On Jul 30, 2014 11:14 PM, "Neko Escondido"  wrote:

> Hello community,
>
> I'm having problem understanding how analyzer should work.  The result is
> different from what I expect.  :(
>
> I have created a custom analyzer to index phone number as below:
>
> "analysis" : {
>"analyzer" : {
>"phone" : {
>"type": "custom",
>"tokenizer":"phone_tokenizer",
>"filter" : [ "phone_filter", "unique" ]
>}
>},
>"tokenizer" : {
>"phone_tokenizer" : {
>"type" : "pattern",
>"pattern":"\\s*[a-zA-Z]+\\s*"
>
>}
>},
>"filter" : {
>   "phone_filter" : {
>"type" : "word_delimiter",
>"preserve_original" : 1,
>"generate_number_parts" : 1,
>"catenate_numbers" : 1
>   }
>}
>}
>
>
>
> The intention is to match:
> Query Input:
>  111222, 111.222., 111-222-, or 111 222 ,
> (111)222, 1-(111)-222-, etc.
> With records containing phone number such as:
>  111.222., 111-222-, or 111 222 , (111)222,
> 1-(111)-222-, etc.
>
> So with search input: (111)222 with queryType "matchPhraseQuery", I
> thought the query will return the records with phone number such as
> 111.222., 111-222-, etc. because input (111)222 would be
> analyzed into 111222, 111, and 222.
> Given I have specified "catenate_numbers" in filter for my "phone"
> analyzer, I would expect that numbers the numbers that meets the following
> condition will be matched:
> Match numbers that are indexed as ( 111 AND 222 ) OR 111222.
> But result is no match.
>
> Is my understanding incorrect?  With search input (111)222 using
> matchPhraseQuery, I thought it will match all numbers that has 111222
> as the concatenated value but it seems to match only with numbers whose
> number parts are 111 and 222...
>
> Your feedback/help/input is greatly appreciated!!
> Best regards
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/b0658b33-2efb-495a-8090-7cc12806a253%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1dLr4bOVsmeudfA29Pm12AH7VPAP9%2BPieHRGi7RyAZow%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Help needed understanding analyzer behavior

[Neko Escondido]

Hello community,

I'm having problem understanding how analyzer should work.  The result is 
different from what I expect.  :(

I have created a custom analyzer to index phone number as below:

"analysis" : {
   "analyzer" : {
   "phone" : {
   "type": "custom",
   "tokenizer":"phone_tokenizer",
   "filter" : [ "phone_filter", "unique" ]
   }
   },
   "tokenizer" : {
   "phone_tokenizer" : {
   "type" : "pattern",
   "pattern":"\\s*[a-zA-Z]+\\s*"
   
   }
   },
   "filter" : {
  "phone_filter" : {
   "type" : "word_delimiter",
   "preserve_original" : 1,
   "generate_number_parts" : 1,
   "catenate_numbers" : 1
  }
   }
   }



The intention is to match:
Query Input: 
 111222, 111.222., 111-222-, or 111 222 , (111)222, 
1-(111)-222-, etc. 
With records containing phone number such as:
 111.222., 111-222-, or 111 222 , (111)222, 
1-(111)-222-, etc. 

So with search input: (111)222 with queryType "matchPhraseQuery", I 
thought the query will return the records with phone number such as 
111.222., 111-222-, etc. because input (111)222 would be 
analyzed into 111222, 111, and 222.
Given I have specified "catenate_numbers" in filter for my "phone" 
analyzer, I would expect that numbers the numbers that meets the following 
condition will be matched:
Match numbers that are indexed as ( 111 AND 222 ) OR 111222.
But result is no match.  

Is my understanding incorrect?  With search input (111)222 using 
matchPhraseQuery, I thought it will match all numbers that has 111222 
as the concatenated value but it seems to match only with numbers whose 
number parts are 111 and 222... 

Your feedback/help/input is greatly appreciated!!
Best regards



-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b0658b33-2efb-495a-8090-7cc12806a253%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: The number of types a index can handle

[panfei]

Thanks for the information


2014-07-30 14:55 GMT+08:00 joergpra...@gmail.com :

> There is no limit in ES.
>
> Each type uses a certain amount of heap for caching ids and the mapping.
> You can create types / mappings until heap explodes. Each modification of a
> mapping is propagated through the cluster, which is not a cheap operation.
> You have to test by yourself if your design works.
>
> Jörg
>
>
> On Wed, Jul 30, 2014 at 7:18 AM, panfei  wrote:
>
>> index -> database
>> type -> table  (HERE)
>> doc -> row
>>
>> I want to know is there a limit on the number of types in a index and how
>> many is too many ?
>>
>>
>> --
>> 不学习，不知道
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CA%2BJstLBXOLeC12wLVmmtBKVyijf6FpyWo%3D4RRetEwA%2B6ajkEdg%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFEvGxxyopggm4i9_9OgSJ80w6OyaLdMRYjLc6jQ4y9xw%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
不学习，不知道

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CA%2BJstLD9yQjVss%3D7itx%3DO09njTeb%2BmfcC1v7ymGPxJ-nxpaxEg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Remote access through SSH

[Mark Walkom]

You can also curl from your local machine to the server, without having to
SSH to it - curl -XGET http://IPADDRESS:9200/

You don't need to provide SSH credentials for that transport client example.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 31 July 2014 10:35, Chia-Eng Chang  wrote:

> Thank you for the links. Yeah, I am new to ES. (and http rest)
> What I understand is that if I want to get the index documents on my SSH
> server, I can SSH log in the server.
> And then rest http get from localhost:9200.
>
> Could you explain more about  "use SSH directly for it"?
> I think what I want to do is close to this transport client example
> 
>
> But I have to provide ssh credential.
>
>
> On Wednesday, July 30, 2014 4:47:22 PM UTC-7, Mark Walkom wrote:
>
>> You may want to look at http://www.elasticsearch.
>> org/guide/en/elasticsearch/reference/current/search.html
>>
>> If you are just learning ES, then check out http://
>> exploringelasticsearch.com/
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 31 July 2014 09:35, Chia-Eng Chang  wrote:
>>
>>>  Thanks @Mark
>>> I have a public key on server and I know how to SSH to server then get
>>> the index from localhost:9200.
>>> But what I want to do is remotely obtain the index on the SSH server
>>> (which I know its public IP)
>>>
>>>
>>> On Wednesday, July 30, 2014 3:56:04 PM UTC-7, Mark Walkom wrote:
>>>
 You need to use SSH directly for it, curl won't work.

 ssh user@host -i ~/.ssh/id_rsa.pub

 Assuming you have a public key on the server.

 Regards,
 Mark Walkom

 Infrastructure Engineer
 Campaign Monitor
 email: ma...@campaignmonitor.com
 web: www.campaignmonitor.com


 On 31 July 2014 08:47, Chia-Eng Chang  wrote:

>  About the HTTP API, I wonder if I want to remote access a cluster on
> SSH server, what should I include in my http rest command:
>
> example as mapping:
>
> curl -XGET ' http://localhost:9200/ index /_mapping/ type '
>
> I  tried something like below but got failed:
>
> curl -XGET -u user_name: --key ~/.ssh/id_rsa --pubkey 
> ~/.ssh/id_rsa.pub  'xx.xxx.xxx.xxx:9200/index/_mapping/type'
>
> Is there anyone knows the solution?
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to elasticsearc...@googlegroups.com.
>
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/elasticsearch/31b4e835-8ebb-4dc7-bc2b-c8fa09414f12%40goo
> glegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/d6404d32-6b0f-4670-8626-f38b1284809d%
>>> 40googlegroups.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/3d1ad9da-e964-482b-89ac-75ad35b68227%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624a%2B4ZZeQvjz9sD1xgxqu_BP%3Db3WJt8PozqPw9hNpNteLw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Memory Explosion: Heap Dump in less than one minute

[Mark Walkom]

Unless you are attached to the stats you have in the marvel index for today
it might be easier to delete them than try to recover the unavailable
shards.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 31 July 2014 10:36, Tom Wilson  wrote:

> Upping to 1GB, memory usage seems to level off at 750MB, but there's a
> problem in there somewhere. I'm getting a failure message, and the marvel
> dashboard isn't able to fetch.
>
>
> C:\elasticsearch-1.1.1\bin>elasticsearch
> Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
> [2014-07-30 17:33:27,138][INFO ][node ] [Mondo]
> version[1.1.1], pid[10864], build[f1585f0/2014-04-16
> T14:27:12Z]
> [2014-07-30 17:33:27,139][INFO ][node ] [Mondo]
> initializing ...
> [2014-07-30 17:33:27,163][INFO ][plugins  ] [Mondo] loaded
> [ldap-river, marvel], sites [marvel]
> [2014-07-30 17:33:30,731][INFO ][node ] [Mondo]
> initialized
> [2014-07-30 17:33:30,731][INFO ][node ] [Mondo]
> starting ...
> [2014-07-30 17:33:31,027][INFO ][transport] [Mondo]
> bound_address {inet[/0.0.0.0:9300]}, publish_address
>  {inet[/192.168.0.6:9300]}
> [2014-07-30 17:33:34,202][INFO ][cluster.service  ] [Mondo]
> new_master [Mondo][liyNQAHAS0-8f-qDDqa5Rg][twilson-T
> HINK][inet[/192.168.0.6:9300]], reason: zen-disco-join (elected_as_master)
> [2014-07-30 17:33:34,239][INFO ][discovery] [Mondo]
> elasticsearch/liyNQAHAS0-8f-qDDqa5Rg
> [2014-07-30 17:33:34,600][INFO ][http ] [Mondo]
> bound_address {inet[/0.0.0.0:9200]}, publish_address
>  {inet[/192.168.0.6:9200]}
> [2014-07-30 17:33:35,799][INFO ][gateway  ] [Mondo]
> recovered [66] indices into cluster_state
> [2014-07-30 17:33:35,815][INFO ][node ] [Mondo] started
> [2014-07-30 17:33:39,823][DEBUG][action.search.type   ] [Mondo] All
> shards failed for phase: [query_fetch]
> [2014-07-30 17:33:39,830][DEBUG][action.search.type   ] [Mondo] All
> shards failed for phase: [query_fetch]
> [2014-07-30 17:33:39,837][DEBUG][action.search.type   ] [Mondo] All
> shards failed for phase: [query_fetch]
> [2014-07-30 17:33:39,838][DEBUG][action.search.type   ] [Mondo] All
> shards failed for phase: [query_fetch]
> [2014-07-30 17:33:43,973][DEBUG][action.search.type   ] [Mondo] All
> shards failed for phase: [query_fetch]
> [2014-07-30 17:33:44,212][DEBUG][action.search.type   ] [Mondo] All
> shards failed for phase: [query_fetch]
> [2014-07-30 17:33:44,357][DEBUG][action.search.type   ] [Mondo] All
> shards failed for phase: [query_fetch]
> [2014-07-30 17:33:44,501][DEBUG][action.search.type   ] [Mondo] All
> shards failed for phase: [query_fetch]
> [2014-07-30 17:33:53,294][DEBUG][action.search.type   ] [Mondo] All
> shards failed for phase: [query_fetch]
> [2014-07-30 17:33:53,309][DEBUG][action.search.type   ] [Mondo] All
> shards failed for phase: [query_fetch]
> [2014-07-30 17:33:53,310][DEBUG][action.search.type   ] [Mondo] All
> shards failed for phase: [query_fetch]
> [2014-07-30 17:33:53,310][DEBUG][action.search.type   ] [Mondo] All
> shards failed for phase: [query_fetch]
> [2014-07-30 17:34:03,281][DEBUG][action.search.type   ] [Mondo] All
> shards failed for phase: [query_fetch]
> [2014-07-30 17:34:03,283][DEBUG][action.search.type   ] [Mondo] All
> shards failed for phase: [query_fetch]
> [2014-07-30 17:34:03,286][DEBUG][action.search.type   ] [Mondo] All
> shards failed for phase: [query_fetch]
> [2014-07-30 17:34:45,662][ERROR][marvel.agent.exporter] [Mondo] create
> failure (index:[.marvel-2014.07.31] type: [no
> de_stats]): UnavailableShardsException[[.marvel-2014.07.31][0] [2]
> shardIt, [0] active : Timeout waiting for [1m], reque
> st: org.elasticsearch.action.bulk.BulkShardRequest@39b65640]
>
>
>
> On Wednesday, July 30, 2014 5:30:29 PM UTC-7, Mark Walkom wrote:
>
>> Up that to 1GB and see if it starts.
>> 512MB is pretty tiny, you're better off starting at 1/2GB if you can.
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 31 July 2014 10:28, Tom Wilson  wrote:
>>
>>>  JDK 1.7.0_51
>>>
>>> It has 512MB of heap, which was enough -- I've been running it like that
>>> for the past few months, and I only have two indexes and around 300-400
>>> documents. This is a development instance I'm running on my local machine.
>>> This only happened when I started it today.
>>>
>>> -tom
>>>
>>>
>>> On Wednesday, July 30, 2014 5:16:11 PM UTC-7, Mark Walkom wrote:
>>>
 What java version? How much heap have you allocated and how much RAM on
 the server?

 Basically you have too much data for the heap size, so increasing it
 will help.

 Regards,
 Mark Walkom

 Inf

Re: Memory Explosion: Heap Dump in less than one minute

[Tom Wilson]

Upping to 1GB, memory usage seems to level off at 750MB, but there's a 
problem in there somewhere. I'm getting a failure message, and the marvel 
dashboard isn't able to fetch.


C:\elasticsearch-1.1.1\bin>elasticsearch
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
[2014-07-30 17:33:27,138][INFO ][node ] [Mondo] 
version[1.1.1], pid[10864], build[f1585f0/2014-04-16
T14:27:12Z]
[2014-07-30 17:33:27,139][INFO ][node ] [Mondo] 
initializing ...
[2014-07-30 17:33:27,163][INFO ][plugins  ] [Mondo] loaded 
[ldap-river, marvel], sites [marvel]
[2014-07-30 17:33:30,731][INFO ][node ] [Mondo] 
initialized
[2014-07-30 17:33:30,731][INFO ][node ] [Mondo] 
starting ...
[2014-07-30 17:33:31,027][INFO ][transport] [Mondo] 
bound_address {inet[/0.0.0.0:9300]}, publish_address
 {inet[/192.168.0.6:9300]}
[2014-07-30 17:33:34,202][INFO ][cluster.service  ] [Mondo] 
new_master [Mondo][liyNQAHAS0-8f-qDDqa5Rg][twilson-T
HINK][inet[/192.168.0.6:9300]], reason: zen-disco-join (elected_as_master)
[2014-07-30 17:33:34,239][INFO ][discovery] [Mondo] 
elasticsearch/liyNQAHAS0-8f-qDDqa5Rg
[2014-07-30 17:33:34,600][INFO ][http ] [Mondo] 
bound_address {inet[/0.0.0.0:9200]}, publish_address
 {inet[/192.168.0.6:9200]}
[2014-07-30 17:33:35,799][INFO ][gateway  ] [Mondo] 
recovered [66] indices into cluster_state
[2014-07-30 17:33:35,815][INFO ][node ] [Mondo] started
[2014-07-30 17:33:39,823][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:39,830][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:39,837][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:39,838][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:43,973][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:44,212][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:44,357][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:44,501][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:53,294][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:53,309][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:53,310][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:53,310][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:34:03,281][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:34:03,283][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:34:03,286][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:34:45,662][ERROR][marvel.agent.exporter] [Mondo] create 
failure (index:[.marvel-2014.07.31] type: [no
de_stats]): UnavailableShardsException[[.marvel-2014.07.31][0] [2] shardIt, 
[0] active : Timeout waiting for [1m], reque
st: org.elasticsearch.action.bulk.BulkShardRequest@39b65640]



On Wednesday, July 30, 2014 5:30:29 PM UTC-7, Mark Walkom wrote:
>
> Up that to 1GB and see if it starts.
> 512MB is pretty tiny, you're better off starting at 1/2GB if you can.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>  
>
> On 31 July 2014 10:28, Tom Wilson > 
> wrote:
>
>> JDK 1.7.0_51
>>
>> It has 512MB of heap, which was enough -- I've been running it like that 
>> for the past few months, and I only have two indexes and around 300-400 
>> documents. This is a development instance I'm running on my local machine. 
>> This only happened when I started it today. 
>>
>> -tom
>>
>>
>> On Wednesday, July 30, 2014 5:16:11 PM UTC-7, Mark Walkom wrote:
>>
>>> What java version? How much heap have you allocated and how much RAM on 
>>> the server?
>>>
>>> Basically you have too much data for the heap size, so increasing it 
>>> will help.
>>>  
>>> Regards,
>>> Mark Walkom
>>>
>>> Infrastructure Engineer
>>> Campaign Monitor
>>> email: ma...@campaignmonitor.com
>>> web: www.campaignmonitor.com
>>>
>>>
>>> On 31 July 2014 10:11, Tom Wilson  wrote:
>>>
  Help! Elasticsearch was working fine, but now it's using up all its 
 heap space in the matter of a few minutes. I uninstalled the river and am 
 performing no queries. How do I diagnose the problem? 2-3 minutes after 
 starting, it runs out of heap space, and I'm no

Re: Remote access through SSH

[Chia-Eng Chang]

Thank you for the links. Yeah, I am new to ES. (and http rest)
What I understand is that if I want to get the index documents on my SSH 
server, I can SSH log in the server.
And then rest http get from localhost:9200.

Could you explain more about  "use SSH directly for it"? 
I think what I want to do is close to this transport client example 

 
But I have to provide ssh credential.


On Wednesday, July 30, 2014 4:47:22 PM UTC-7, Mark Walkom wrote:
>
> You may want to look at 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search.html
>
> If you are just learning ES, then check out 
> http://exploringelasticsearch.com/
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>  
>
> On 31 July 2014 09:35, Chia-Eng Chang > 
> wrote:
>
>> Thanks @Mark
>> I have a public key on server and I know how to SSH to server then get 
>> the index from localhost:9200.
>> But what I want to do is remotely obtain the index on the SSH server 
>> (which I know its public IP)
>>
>>
>> On Wednesday, July 30, 2014 3:56:04 PM UTC-7, Mark Walkom wrote:
>>
>>> You need to use SSH directly for it, curl won't work.
>>>
>>> ssh user@host -i ~/.ssh/id_rsa.pub
>>>
>>> Assuming you have a public key on the server.
>>>
>>> Regards,
>>> Mark Walkom
>>>
>>> Infrastructure Engineer
>>> Campaign Monitor
>>> email: ma...@campaignmonitor.com
>>> web: www.campaignmonitor.com
>>>
>>>
>>> On 31 July 2014 08:47, Chia-Eng Chang  wrote:
>>>
  About the HTTP API, I wonder if I want to remote access a cluster on 
 SSH server, what should I include in my http rest command:

 example as mapping:

 curl -XGET ' http://localhost:9200/ index /_mapping/ type ' 

 I  tried something like below but got failed:

 curl -XGET -u user_name: --key ~/.ssh/id_rsa --pubkey 
 ~/.ssh/id_rsa.pub  'xx.xxx.xxx.xxx:9200/index/_mapping/type'

 Is there anyone knows the solution?

 -- 
 You received this message because you are subscribed to the Google 
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to elasticsearc...@googlegroups.com.

 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/31b4e835-8ebb-4dc7-bc2b-c8fa09414f12%
 40googlegroups.com 
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/d6404d32-6b0f-4670-8626-f38b1284809d%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3d1ad9da-e964-482b-89ac-75ad35b68227%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Memory Explosion: Heap Dump in less than one minute

[Mark Walkom]

Up that to 1GB and see if it starts.
512MB is pretty tiny, you're better off starting at 1/2GB if you can.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 31 July 2014 10:28, Tom Wilson  wrote:

> JDK 1.7.0_51
>
> It has 512MB of heap, which was enough -- I've been running it like that
> for the past few months, and I only have two indexes and around 300-400
> documents. This is a development instance I'm running on my local machine.
> This only happened when I started it today.
>
> -tom
>
>
> On Wednesday, July 30, 2014 5:16:11 PM UTC-7, Mark Walkom wrote:
>
>> What java version? How much heap have you allocated and how much RAM on
>> the server?
>>
>> Basically you have too much data for the heap size, so increasing it will
>> help.
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 31 July 2014 10:11, Tom Wilson  wrote:
>>
>>>  Help! Elasticsearch was working fine, but now it's using up all its
>>> heap space in the matter of a few minutes. I uninstalled the river and am
>>> performing no queries. How do I diagnose the problem? 2-3 minutes after
>>> starting, it runs out of heap space, and I'm not sure how to find out why.
>>>
>>> Here is the profile of memory usage:
>>>
>>>
>>> 
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> And  here is the console output. You can see it takes less than a minute
>>> after starting to run out of memory. This isn't even enough time to examine
>>> the indices through marvel.
>>>
>>> C:\elasticsearch-1.1.1\bin>elasticsearch
>>> Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
>>> [2014-07-30 16:59:02,579][INFO ][node ] [Texas
>>> Twister] version[1.1.1], pid[8572], build[f1585f0/201
>>> 4-04-16T14:27:12Z]
>>> [2014-07-30 16:59:02,580][INFO ][node ] [Texas
>>> Twister] initializing ...
>>> [2014-07-30 16:59:02,600][INFO ][plugins  ] [Texas
>>> Twister] loaded [marvel], sites [marvel]
>>> [2014-07-30 16:59:06,437][INFO ][node ] [Texas
>>> Twister] initialized
>>> [2014-07-30 16:59:06,437][INFO ][node ] [Texas
>>> Twister] starting ...
>>> [2014-07-30 16:59:06,691][INFO ][transport] [Texas
>>> Twister] bound_address {inet[/0.0.0.0:9300]}, publish
>>> _address {inet[/192.168.0.6:9300]}
>>> [2014-07-30 16:59:09,862][INFO ][cluster.service  ] [Texas
>>> Twister] new_master [Texas Twister][ShQRhZRFQnuZMTRCu
>>> vY9XQ][twilson-THINK][inet[/192.168.0.6:9300]], reason: zen-disco-join
>>> (elected_as_master)
>>> [2014-07-30 16:59:09,902][INFO ][discovery] [Texas
>>> Twister] elasticsearch/ShQRhZRFQnuZMTRCuvY9XQ
>>> [2014-07-30 16:59:10,213][INFO ][http ] [Texas
>>> Twister] bound_address {inet[/0.0.0.0:9200]}, publish
>>> _address {inet[/192.168.0.6:9200]}
>>> [2014-07-30 16:59:11,631][INFO ][gateway  ] [Texas
>>> Twister] recovered [65] indices into cluster_state
>>> java.lang.OutOfMemoryError: Java heap space
>>> Dumping heap to java_pid8572.hprof ...
>>> Heap dump file created [814218130 bytes in 14.202 secs]
>>> Exception in thread "elasticsearch[Texas Twister][generic][T#2]"
>>> java.lang.OutOfMemoryError: Java heap space
>>> at java.lang.Class.getDeclaredFields0(Native Method)
>>> at java.lang.Class.privateGetDeclaredFields(Class.java:2397)
>>> at java.lang.Class.getDeclaredFields(Class.java:1806)
>>> at org.apache.lucene.util.RamUsageEstimator.
>>> shallowSizeOfInstance(RamUsageEstimator.java:388)
>>> at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.<
>>> init>(Lucene42DocValuesProducer.java:101)
>>> at org.apache.lucene.codecs.lucene42.Lucene42NormsFormat.
>>> normsProducer(Lucene42NormsFormat.java:75)
>>> at org.apache.lucene.index.SegmentCoreReaders.(
>>> SegmentCoreReaders.java:123)
>>> at org.apache.lucene.index.SegmentReader.(
>>> SegmentReader.java:96)
>>> at org.apache.lucene.index.ReadersAndUpdates.getReader(
>>> ReadersAndUpdates.java:141)
>>> at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(
>>> ReadersAndUpdates.java:235)
>>> at org.apache.lucene.index.StandardDirectoryReader.open(
>>> StandardDirectoryReader.java:101)
>>>  at org.apache.lucene.index.IndexWriter.getReader(
>>> IndexWriter.java:382)
>>> at org.apache.lucene.index.DirectoryReader.open(
>>> DirectoryReader.java:111)
>>> at org.apache.lucene.search.SearcherManager.(
>>> SearcherManager.java:89)
>>> at org.elasticsearch.index.engine.internal.InternalEngine.
>>> buildSearchManager(InternalEngine.java:1471)
>>> at org.elasticsearch.index.engine.i

Re: Memory Explosion: Heap Dump in less than one minute

[Tom Wilson]

JDK 1.7.0_51

It has 512MB of heap, which was enough -- I've been running it like that 
for the past few months, and I only have two indexes and around 300-400 
documents. This is a development instance I'm running on my local machine. 
This only happened when I started it today. 

-tom

On Wednesday, July 30, 2014 5:16:11 PM UTC-7, Mark Walkom wrote:
>
> What java version? How much heap have you allocated and how much RAM on 
> the server?
>
> Basically you have too much data for the heap size, so increasing it will 
> help.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
>
> On 31 July 2014 10:11, Tom Wilson > 
> wrote:
>
>> Help! Elasticsearch was working fine, but now it's using up all its heap 
>> space in the matter of a few minutes. I uninstalled the river and am 
>> performing no queries. How do I diagnose the problem? 2-3 minutes after 
>> starting, it runs out of heap space, and I'm not sure how to find out why.
>>
>> Here is the profile of memory usage:
>>
>>
>> 
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> And  here is the console output. You can see it takes less than a minute 
>> after starting to run out of memory. This isn't even enough time to examine 
>> the indices through marvel.
>>
>> C:\elasticsearch-1.1.1\bin>elasticsearch
>> Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
>> [2014-07-30 16:59:02,579][INFO ][node ] [Texas 
>> Twister] version[1.1.1], pid[8572], build[f1585f0/201
>> 4-04-16T14:27:12Z]
>> [2014-07-30 16:59:02,580][INFO ][node ] [Texas 
>> Twister] initializing ...
>> [2014-07-30 16:59:02,600][INFO ][plugins  ] [Texas 
>> Twister] loaded [marvel], sites [marvel]
>> [2014-07-30 16:59:06,437][INFO ][node ] [Texas 
>> Twister] initialized
>> [2014-07-30 16:59:06,437][INFO ][node ] [Texas 
>> Twister] starting ...
>> [2014-07-30 16:59:06,691][INFO ][transport] [Texas 
>> Twister] bound_address {inet[/0.0.0.0:9300]}, publish
>> _address {inet[/192.168.0.6:9300]}
>> [2014-07-30 16:59:09,862][INFO ][cluster.service  ] [Texas 
>> Twister] new_master [Texas Twister][ShQRhZRFQnuZMTRCu
>> vY9XQ][twilson-THINK][inet[/192.168.0.6:9300]], reason: zen-disco-join 
>> (elected_as_master)
>> [2014-07-30 16:59:09,902][INFO ][discovery] [Texas 
>> Twister] elasticsearch/ShQRhZRFQnuZMTRCuvY9XQ
>> [2014-07-30 16:59:10,213][INFO ][http ] [Texas 
>> Twister] bound_address {inet[/0.0.0.0:9200]}, publish
>> _address {inet[/192.168.0.6:9200]}
>> [2014-07-30 16:59:11,631][INFO ][gateway  ] [Texas 
>> Twister] recovered [65] indices into cluster_state
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid8572.hprof ...
>> Heap dump file created [814218130 bytes in 14.202 secs]
>> Exception in thread "elasticsearch[Texas Twister][generic][T#2]" 
>> java.lang.OutOfMemoryError: Java heap space
>> at java.lang.Class.getDeclaredFields0(Native Method)
>> at java.lang.Class.privateGetDeclaredFields(Class.java:2397)
>> at java.lang.Class.getDeclaredFields(Class.java:1806)
>> at 
>> org.apache.lucene.util.RamUsageEstimator.shallowSizeOfInstance(RamUsageEstimator.java:388)
>> at 
>> org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.(Lucene42DocValuesProducer.java:101)
>> at 
>> org.apache.lucene.codecs.lucene42.Lucene42NormsFormat.normsProducer(Lucene42NormsFormat.java:75)
>> at 
>> org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:123)
>> at 
>> org.apache.lucene.index.SegmentReader.(SegmentReader.java:96)
>> at 
>> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:141)
>> at 
>> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:235)
>> at 
>> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:101)
>> at 
>> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:382)
>> at 
>> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:111)
>> at 
>> org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)
>> at 
>> org.elasticsearch.index.engine.internal.InternalEngine.buildSearchManager(InternalEngine.java:1471)
>> at 
>> org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:279)
>> at 
>> org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard
>> .java:699)
>> at 
>> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:205)
>> at 
>> org.elasticsearch.index.gateway.IndexShardGatew

Re: Memory Explosion: Heap Dump in less than one minute

[Mark Walkom]

What java version? How much heap have you allocated and how much RAM on the
server?

Basically you have too much data for the heap size, so increasing it will
help.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 31 July 2014 10:11, Tom Wilson  wrote:

> Help! Elasticsearch was working fine, but now it's using up all its heap
> space in the matter of a few minutes. I uninstalled the river and am
> performing no queries. How do I diagnose the problem? 2-3 minutes after
> starting, it runs out of heap space, and I'm not sure how to find out why.
>
> Here is the profile of memory usage:
>
>
> 
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> And  here is the console output. You can see it takes less than a minute
> after starting to run out of memory. This isn't even enough time to examine
> the indices through marvel.
>
> C:\elasticsearch-1.1.1\bin>elasticsearch
> Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
> [2014-07-30 16:59:02,579][INFO ][node ] [Texas
> Twister] version[1.1.1], pid[8572], build[f1585f0/201
> 4-04-16T14:27:12Z]
> [2014-07-30 16:59:02,580][INFO ][node ] [Texas
> Twister] initializing ...
> [2014-07-30 16:59:02,600][INFO ][plugins  ] [Texas
> Twister] loaded [marvel], sites [marvel]
> [2014-07-30 16:59:06,437][INFO ][node ] [Texas
> Twister] initialized
> [2014-07-30 16:59:06,437][INFO ][node ] [Texas
> Twister] starting ...
> [2014-07-30 16:59:06,691][INFO ][transport] [Texas
> Twister] bound_address {inet[/0.0.0.0:9300]}, publish
> _address {inet[/192.168.0.6:9300]}
> [2014-07-30 16:59:09,862][INFO ][cluster.service  ] [Texas
> Twister] new_master [Texas Twister][ShQRhZRFQnuZMTRCu
> vY9XQ][twilson-THINK][inet[/192.168.0.6:9300]], reason: zen-disco-join
> (elected_as_master)
> [2014-07-30 16:59:09,902][INFO ][discovery] [Texas
> Twister] elasticsearch/ShQRhZRFQnuZMTRCuvY9XQ
> [2014-07-30 16:59:10,213][INFO ][http ] [Texas
> Twister] bound_address {inet[/0.0.0.0:9200]}, publish
> _address {inet[/192.168.0.6:9200]}
> [2014-07-30 16:59:11,631][INFO ][gateway  ] [Texas
> Twister] recovered [65] indices into cluster_state
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid8572.hprof ...
> Heap dump file created [814218130 bytes in 14.202 secs]
> Exception in thread "elasticsearch[Texas Twister][generic][T#2]"
> java.lang.OutOfMemoryError: Java heap space
> at java.lang.Class.getDeclaredFields0(Native Method)
> at java.lang.Class.privateGetDeclaredFields(Class.java:2397)
> at java.lang.Class.getDeclaredFields(Class.java:1806)
> at
> org.apache.lucene.util.RamUsageEstimator.shallowSizeOfInstance(RamUsageEstimator.java:388)
> at
> org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.(Lucene42DocValuesProducer.java:101)
> at
> org.apache.lucene.codecs.lucene42.Lucene42NormsFormat.normsProducer(Lucene42NormsFormat.java:75)
> at
> org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:123)
> at
> org.apache.lucene.index.SegmentReader.(SegmentReader.java:96)
> at
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:141)
> at
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:235)
> at
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:101)
> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:382)
> at
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:111)
> at
> org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)
> at
> org.elasticsearch.index.engine.internal.InternalEngine.buildSearchManager(InternalEngine.java:1471)
> at
> org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:279)
> at
> org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard
> .java:699)
> at
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:205)
> at
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:197)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegrou

Memory Explosion: Heap Dump in less than one minute

[Tom Wilson]

Help! Elasticsearch was working fine, but now it's using up all its heap 
space in the matter of a few minutes. I uninstalled the river and am 
performing no queries. How do I diagnose the problem? 2-3 minutes after 
starting, it runs out of heap space, and I'm not sure how to find out why.

Here is the profile of memory usage:




















And  here is the console output. You can see it takes less than a minute 
after starting to run out of memory. This isn't even enough time to examine 
the indices through marvel.

C:\elasticsearch-1.1.1\bin>elasticsearch
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
[2014-07-30 16:59:02,579][INFO ][node ] [Texas Twister] 
version[1.1.1], pid[8572], build[f1585f0/201
4-04-16T14:27:12Z]
[2014-07-30 16:59:02,580][INFO ][node ] [Texas Twister] 
initializing ...
[2014-07-30 16:59:02,600][INFO ][plugins  ] [Texas Twister] 
loaded [marvel], sites [marvel]
[2014-07-30 16:59:06,437][INFO ][node ] [Texas Twister] 
initialized
[2014-07-30 16:59:06,437][INFO ][node ] [Texas Twister] 
starting ...
[2014-07-30 16:59:06,691][INFO ][transport] [Texas Twister] 
bound_address {inet[/0.0.0.0:9300]}, publish
_address {inet[/192.168.0.6:9300]}
[2014-07-30 16:59:09,862][INFO ][cluster.service  ] [Texas Twister] 
new_master [Texas Twister][ShQRhZRFQnuZMTRCu
vY9XQ][twilson-THINK][inet[/192.168.0.6:9300]], reason: zen-disco-join 
(elected_as_master)
[2014-07-30 16:59:09,902][INFO ][discovery] [Texas Twister] 
elasticsearch/ShQRhZRFQnuZMTRCuvY9XQ
[2014-07-30 16:59:10,213][INFO ][http ] [Texas Twister] 
bound_address {inet[/0.0.0.0:9200]}, publish
_address {inet[/192.168.0.6:9200]}
[2014-07-30 16:59:11,631][INFO ][gateway  ] [Texas Twister] 
recovered [65] indices into cluster_state
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid8572.hprof ...
Heap dump file created [814218130 bytes in 14.202 secs]
Exception in thread "elasticsearch[Texas Twister][generic][T#2]" 
java.lang.OutOfMemoryError: Java heap space
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2397)
at java.lang.Class.getDeclaredFields(Class.java:1806)
at 
org.apache.lucene.util.RamUsageEstimator.shallowSizeOfInstance(RamUsageEstimator.java:388)
at 
org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.(Lucene42DocValuesProducer.java:101)
at 
org.apache.lucene.codecs.lucene42.Lucene42NormsFormat.normsProducer(Lucene42NormsFormat.java:75)
at 
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:123)
at 
org.apache.lucene.index.SegmentReader.(SegmentReader.java:96)
at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:141)
at 
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:235)
at 
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:101)
at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:382)
at 
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:111)
at 
org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)
at 
org.elasticsearch.index.engine.internal.InternalEngine.buildSearchManager(InternalEngine.java:1471)
at 
org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:279)
at 
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard
.java:699)
at 
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:205)
at 
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:197)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/21e9cc63-0c5d-4ea0-96a2-78d817b89236%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Configuration Brain Wobbles

[Christopher Ambler]

Yes, I'm aware of the multi-DC issue ;)

So yeah, this morning I dove into index shard allocation, and did just that.

3 zones (using node.zone as my tag) and then I set things to have 1 primary 
and 2 replications and then set it such that each zone should play along 
with this.

This worked as advertised. One zone gets the primary and the other two 
zones get the two replications. Each zone as 2 nodes and I have 5 shards, 
so one node gets two and the other gets three.

Head shows me this and it all makes sense.

We're now doing about 30 searches per second, and I'm still seeing, about 
every 5 or 6 seconds, a single "slow query" in the 600ms to 900ms range. 
All other queries are sub-50ms.

I need to find out why I'm seeing these consistent 600ms+ queries and 
eliminate them if I can.

Everything else looks good.

Oh, and on the unallocated shards issue, I tracked that down to having had 
more replications when I built the index and then taking my replications 
down. I had shards that had nowhere to go. So I just removed those indexes 
(they were old) and everything is green.

So the consistent 600ms slow queries is my only issue now.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/99286c26-9816-4174-8358-a44438b448aa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Remote access through SSH

[Mark Walkom]

You may want to look at
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search.html

If you are just learning ES, then check out
http://exploringelasticsearch.com/

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 31 July 2014 09:35, Chia-Eng Chang  wrote:

> Thanks @Mark
> I have a public key on server and I know how to SSH to server then get the
> index from localhost:9200.
> But what I want to do is remotely obtain the index on the SSH server
> (which I know its public IP)
>
>
> On Wednesday, July 30, 2014 3:56:04 PM UTC-7, Mark Walkom wrote:
>
>> You need to use SSH directly for it, curl won't work.
>>
>> ssh user@host -i ~/.ssh/id_rsa.pub
>>
>> Assuming you have a public key on the server.
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 31 July 2014 08:47, Chia-Eng Chang  wrote:
>>
>>>  About the HTTP API, I wonder if I want to remote access a cluster on
>>> SSH server, what should I include in my http rest command:
>>>
>>> example as mapping:
>>>
>>> curl -XGET ' http://localhost:9200/ index /_mapping/ type '
>>>
>>> I  tried something like below but got failed:
>>>
>>> curl -XGET -u user_name: --key ~/.ssh/id_rsa --pubkey ~/.ssh/id_rsa.pub 
>>>  'xx.xxx.xxx.xxx:9200/index/_mapping/type'
>>>
>>> Is there anyone knows the solution?
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>>
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/31b4e835-8ebb-4dc7-bc2b-c8fa09414f12%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/d6404d32-6b0f-4670-8626-f38b1284809d%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624Z%2B3wOyb%2BEQFyQvrfiGA-9e%2BwEc1M7AA1QcfM0CbJCdCg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Remote access through SSH

[Chia-Eng Chang]

Thanks @Mark
I have a public key on server and I know how to SSH to server then get the 
index from localhost:9200.
But what I want to do is remotely obtain the index on the SSH server (which 
I know its public IP)


On Wednesday, July 30, 2014 3:56:04 PM UTC-7, Mark Walkom wrote:
>
> You need to use SSH directly for it, curl won't work.
>
> ssh user@host -i ~/.ssh/id_rsa.pub
>
> Assuming you have a public key on the server.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
>
> On 31 July 2014 08:47, Chia-Eng Chang > 
> wrote:
>
>> About the HTTP API, I wonder if I want to remote access a cluster on SSH 
>> server, what should I include in my http rest command:
>>
>> example as mapping:
>>
>> curl -XGET ' http://localhost:9200/ index /_mapping/ type ' 
>>
>> I  tried something like below but got failed:
>>
>> curl -XGET -u user_name: --key ~/.ssh/id_rsa --pubkey ~/.ssh/id_rsa.pub  
>> 'xx.xxx.xxx.xxx:9200/index/_mapping/type'
>>
>> Is there anyone knows the solution?
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/31b4e835-8ebb-4dc7-bc2b-c8fa09414f12%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d6404d32-6b0f-4670-8626-f38b1284809d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

"more like this" vs. mlt

[Peter Li]

I ran a query:

curl -XGET 
"$url/ease/RadiologyResult/90642/_mlt?routing=07009409&mlt_fields=Observation.Value&min_term_freq=1&min_doc_freq=1&pretty"

It worked and returned several documents. But if I ran this:

curl -XGET "$url/ease/RadiologyResult/_search?routing=07009409&pretty" -d '
{

"query" : {
"more_like_this" : {
"fields": [
"Observation.Value"
],
"ids" : [ "90642" ]
   }
   }
}'

It returned nothing. Is there something I am missing ?

Thanks in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ffa73983-ce04-4fd3-9786-ee09d3248d83%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)

[Mark Walkom]

I've seen this as well Ivan, and have also had a few people on IRC comment
on the same thing - shards that are local are not simply being initialised,
but being reallocated elsewhere.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 31 July 2014 03:54, Ivan Brusic  wrote:

> The idea is that the cluster should be delayed when a cluster rebalance
> occurs, but even with these settings, I often find that shards are moved
> immediately.
>
> Are you using the default stores throttling settings? I found them to be
> quite low.
>
> Cheers,
>
> Ivan
>
>
> On Wed, Jul 30, 2014 at 6:02 AM, Andrew Davidoff 
> wrote:
>
>>
>>
>> On Tuesday, July 29, 2014 3:27:13 PM UTC-4, Ivan Brusic wrote:
>>>
>>> Have you changed your gateway settings? http://www.
>>> elasticsearch.org/guide/en/elasticsearch/reference/
>>> current/modules-gateway.html#recover-after
>>>
>>> It still remains a bit of black magic to me. Sometimes it works,
>>> sometimes it does not.
>>>
>>
>> Ivan,
>>
>> I have read over that documentation several times and I don't understand
>> how it would help me. By which I mean I honestly don't understand how it
>> would help me - not that I am unwilling to try it. Those settings read like
>> they control when recovery would begin, but my problem isn't that recovery
>> is starting when I don't want it to, it's that when I start it (by setting
>> shard allocation about to "all") shards that I'd think would just stick
>> with the node they were previously on, get assigned to other nodes, then
>> ultimately get rebalanced back to the original node.
>>
>> At this point I am finding that for quick restarts, just doing them with
>> no additional prep work allows me to recover in ~30m, vs ~24h. So for now
>> I'm just going to do that. Whatever I am doing wrong here just isn't at all
>> clear to me.
>>
>> Thanks for your advice. If I have misunderstood the settings you pointed
>> me at and you think you can help me understand, I'd be grateful for more
>> information.
>>
>> Andy
>>
>>
>>
>>> Cheers,
>>>
>>> Ivan
>>>
>>>
>>> On Mon, Jul 28, 2014 at 1:52 PM, Andrew Davidoff 
>>> wrote:
>>>
  I guess I'll ask about this once more for now. This happened again
 today. I set allocation to new_primaries, restarted a node, set allocation
 back to all, and the cluster is assigning across all nodes the shards that
 were on the restarted node, and when it's done, which will probably take a
 day, it'll likely rebalance by moving them back to the original node. I
 have to assume I'm doing something wrong here. Am I?

 Thanks for any advice.
 Andy

 --
 You received this message because you are subscribed to the Google
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3%
 40googlegroups.com
 
 .

 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/b8cf076f-2494-4b0e-b6c3-5a21d1c9f9a1%40googlegroups.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBnsTUiXYZVyUgnuU6k7nN8NJiwh2UK_JPTOhEVa8BCRA%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624ZRRCJNbVBcaaJ_0dsSGou_xhx5g8wDGvdeZKV6XQ3qsA%40mail.gmail.com.
For more options, visit https://groups.go

Re: Configuration Brain Wobbles

[Mark Walkom]

Standard response to this is ES is not built for multi DC clustering, but
as long as you are aware you are of that then it's fine.

Have you looked at
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-allocation.html
?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 31 July 2014 08:31, Christopher Ambler  wrote:

> I have a cluster with six nodes. The nodes are in different data centers,
> but I don't think that matters, as the connectivity is beefy and thick. I
> have turned multicast off and unicast on. Each node knows about all the
> others explicitly. When I bring up a visualization of the cluster using
> the "head" plugin, I see them all. This appears to work as it should. My
> cluster looks like this:
>
> DEV-02 (development data center)
> MESA-01 (mesa data center)
> MESA-02 (mesa data center)
> MESA-03 (mesa data center)
> BUCK-01 (buck data center)
> BUCK-02 (buck data center)
>
> I have each node configured for 5 shards.
> I have each node set to be master true and data true
>
> I do ALL of my document addition using MESA-01 and I can do queries on ANY
> node and get a result, so that's working. But I notice two things and have
> one requirement I can't figure out:
>
> 1. Most queries come in sub-30ms. But every now and again I get a query
> that is longer. I set my slow query log to complain over 100ms and I see
> that maybe one query out of 15 or so takes 800ms to 1200ms. This is on any
> node.
>
> 2. I have unassigned shards. I presume this is bad, yes? How do I get them
> to allocate? When I stop and start the service on any of the nodes, the
> shards are shuffled around, but rarely are the unassigned shards put on a
> node. Why? How do I resolve this?
>
> And my requirement - Is there a way to say, "Look, Elasticsearch, I don't
> want you shuffling shards around here and there, I'd like EVERY node to
> have a COMPLETE replica of the data, and you just keep it up to date. That
> way, you see, a query on a buck data center node won't have to ask a mesa
> data center for a document if it doesn't have it."
>
> Solving #2 is important, but solving my requirement is somewhat critical.
> I think fixing these two things will take care of issue #1.
>
> At least it'll get me configured right so if #1 is still there, I can
> diagnose from a position of not wondering if misconfiguration is my problem.
>
> Help?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/33e7db26-ed5e-4c9e-abe5-fd656a73e978%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624ZRF-uK7-dhuOSVqNwyQ13t6E3fGWkc4PpjvP_e2F9Z4A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Remote access through SSH

[Mark Walkom]

You need to use SSH directly for it, curl won't work.

ssh user@host -i ~/.ssh/id_rsa.pub

Assuming you have a public key on the server.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 31 July 2014 08:47, Chia-Eng Chang  wrote:

> About the HTTP API, I wonder if I want to remote access a cluster on SSH
> server, what should I include in my http rest command:
>
> example as mapping:
>
> curl -XGET ' http://localhost:9200/ index /_mapping/ type '
>
> I  tried something like below but got failed:
>
> curl -XGET -u user_name: --key ~/.ssh/id_rsa --pubkey ~/.ssh/id_rsa.pub  
> 'xx.xxx.xxx.xxx:9200/index/_mapping/type'
>
> Is there anyone knows the solution?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/31b4e835-8ebb-4dc7-bc2b-c8fa09414f12%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624YMcO4oP74SUAPwQFDpa8cZsVqNwMWGNg%3DcJGU2qkwQSw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Remote access through SSH

[Chia-Eng Chang]

About the HTTP API, I wonder if I want to remote access a cluster on SSH 
server, what should I include in my http rest command:

example as mapping:

curl -XGET ' http://localhost:9200/ index /_mapping/ type ' 

I  tried something like below but got failed:

curl -XGET -u user_name: --key ~/.ssh/id_rsa --pubkey ~/.ssh/id_rsa.pub  
'xx.xxx.xxx.xxx:9200/index/_mapping/type'

Is there anyone knows the solution?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/31b4e835-8ebb-4dc7-bc2b-c8fa09414f12%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Using ES for gmail-like application

[Maxim Kramarenko]

I am going to use ES to implement gmail-like app, distributed over multiple 
cluster nodes. Questions are:

1) Messages will be compressed, so I need to store binaries in index, not 
plain text. Is it possible to store binary data with minimal overhead, 
without base64 encoding ? 

2) I need to show message list ordered by date with paging. How to 
implement this on clustered ES ? As I understand, if we ask first 10 
records and don't specify routing key, ES will return 10 items from each 
server, not global first 10. How to get page 2,3,..,1000 for clustered ES ?

3) It seems like using Lucene+GlusterFS can solve both problems, any ES 
benefits for this case ? 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/26cd0bb4-56bc-45f1-9c35-01b9a30c7ea2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Configuration Brain Wobbles

[Christopher Ambler]

I have a cluster with six nodes. The nodes are in different data centers, 
but I don't think that matters, as the connectivity is beefy and thick. I 
have turned multicast off and unicast on. Each node knows about all the 
others explicitly. When I bring up a visualization of the cluster using the 
"head" plugin, I see them all. This appears to work as it should. My 
cluster looks like this:

DEV-02 (development data center)
MESA-01 (mesa data center)
MESA-02 (mesa data center)
MESA-03 (mesa data center)
BUCK-01 (buck data center)
BUCK-02 (buck data center)

I have each node configured for 5 shards.
I have each node set to be master true and data true

I do ALL of my document addition using MESA-01 and I can do queries on ANY 
node and get a result, so that's working. But I notice two things and have 
one requirement I can't figure out:

1. Most queries come in sub-30ms. But every now and again I get a query 
that is longer. I set my slow query log to complain over 100ms and I see 
that maybe one query out of 15 or so takes 800ms to 1200ms. This is on any 
node.

2. I have unassigned shards. I presume this is bad, yes? How do I get them 
to allocate? When I stop and start the service on any of the nodes, the 
shards are shuffled around, but rarely are the unassigned shards put on a 
node. Why? How do I resolve this?

And my requirement - Is there a way to say, "Look, Elasticsearch, I don't 
want you shuffling shards around here and there, I'd like EVERY node to 
have a COMPLETE replica of the data, and you just keep it up to date. That 
way, you see, a query on a buck data center node won't have to ask a mesa 
data center for a document if it doesn't have it."

Solving #2 is important, but solving my requirement is somewhat critical. I 
think fixing these two things will take care of issue #1.

At least it'll get me configured right so if #1 is still there, I can 
diagnose from a position of not wondering if misconfiguration is my problem.

Help?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/33e7db26-ed5e-4c9e-abe5-fd656a73e978%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to know if my curator instance is running fine ?

[Aaron Mildenstein]

Sorry this never got responded to.  Unless your indices are hourly, and in 
a format that curator recognizes, it will not delete anything.

What are your index names, or your naming schema?

--Aaron

On Thursday, May 15, 2014 8:49:00 AM UTC-5, Guillaume boufflers wrote:
>
> Hello buds !
>
> I've installed Elasticsearch, Logstash and Kibana lately ..
> I've been playing with those tools for a couple of days now and it's been 
> so cool, great tool ! 
> Now that i've got data in my elasticsearch base, i'd like to remove the 
> indexes that are older than an hour.
> I made some searches on the web and i found the curator tool. seems great.
> I did follow the instructions on the github webpage.
> This is the command i ran :
>
> curator --host 192.168.1.12 --port 9200 -T hours -d 1
>
>
> And Here is the output :
>
> 2014-05-15T15:34:44.412 INFOmain:333  Job starting
> ...
>
> 2014-05-15T15:34:44.413 INFO   _new_conn:180  Starting new 
> HTTP connection (1): 192.168.1.12
>
> 2014-05-15T15:34:44.415 INFO log_request_success:49   GET http://
> 192.168.1.12:9200/ [status:200 request:0.002s]
>
> 2014-05-15T15:34:44.416 INFOmain:359  Deleting 
> indices older than 1 hours...
>
> 2014-05-15T15:34:44.418 INFO log_request_success:49   GET http://
> 192.168.1.12:9200/logstash-*/_settings?expand_wildcards=closed 
> [status:200 request:0.001s]
>
> 2014-05-15T15:34:44.418 INFO  index_loop:309  DELETE 
> index operations completed.
>
> 2014-05-15T15:34:44.419 INFOmain:379  Done in 0:00
> :00.010193.
>
> Seems pretty good in fact but the problem (or not) is that i don't see any 
> difference in my Kibana monitor, i'm still seeing graph from this morning,
>
> i was just wondering if this was a normal behavior or if i did something 
> wrong with the command i ran.
>
> Is there a way to concretely see if my curator instance is doing the job 
> just fine ?
>
> Thank you for reading and helping.
> Regards, Guillaume.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9d09a722-f49a-4ca9-824d-cc68ee65d59b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ip type and support for a port?

[joergpra...@gmail.com]

Can you give an example what you mean by IP ports?

Transport protocols like TCP has ports, but IP (Internet addresses) is used
to address hosts on a network.

Jörg



On Wed, Jul 30, 2014 at 11:02 PM, Chris Neal 
wrote:

> Hi all,
>
> I'm trying to use the ip type in ES, but my IPs also have ports.  That
> doesn't seem to be supported, which was a bit of a surprise!
>
> Does anyone know of a way to do this?  Or does it sound like a good
> feature to add support for to this type?
>
> Thanks!
> Chris
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAND3DpgEWdxZcaFe1wmSWtjRfXqxvUC0vbq8bfE0mC7DusS_%2Bw%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFKah6pkCWgKZYASM%3DfxyBq8waPQjyCBHj2aZaAeyiRwQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to use Curator to manage old data and avoid running out of storage space?

[Aaron Mildenstein]

Hi David,

Backing up indices to a repository is a great way to conserve space in your 
cluster.

Curator provides a helper script called es_repo_mgr that will aid in 
creation of a repository.  There is more information about snapshot 
creation here: modules-snapshots.html 


However, it should be noted that Curator is only for *taking* snapshots—it 
cannot restore them.  This functionality was omitted because restoring is 
not typically a daily occurrence, like the other procedures Curator does. 
Fortunately, restoring indices is a relatively simple thing to do with the 
API (modules-snapshots.html 

).

--Aaron

On Wednesday, July 30, 2014 12:10:36 PM UTC-5, David Reagan wrote:
>
> I've been implementing an ELK stack for the past year or so. I had thought 
> that we would have plenty of space, but recently added a log source that 
> increased the number of log entries a day by around 30x. That prompted me 
> to start looking into ways of managing ES's data storage in order to keep 
> from running out of space. Which led me to Curator and Snapshots.
>
> If I am reading the documentation[1] for both systems correctly, I think I 
> can do the following:
>
>- Create a repository for old data.
>- Use a cron job and Curator to automatically take snapshots of data 
>older than a certain time period (say, 6 months).
>   - Then have Curator delete the data older than that time period.
>   - The result would be that all data older than the time period 
>   would be stored in the repository. The data would be compressed (what 
> kind 
>   of compression?)
>- When I have need for data older than the time period, I could use 
>Curator to restore it to the ES cluster, or even a different ES cluster. 
>   - After that I could do what I needed, before deleting it again.
>
>
> I'd test all this myself, but I don't have the resources for a decent test 
> environment yet. :( Still working on that. 
>
> Am I missing anything? Are there better ways to keep from running out of 
> storage space? Any general advice related to this kind of thing?
>
> Thanks in advance!
>
> [1] 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
> https://github.com/elasticsearch/curator/wiki
>
> http://www.elasticsearch.org/blog/elasticsearch-curator-version-1-1-0-released/
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d7a6924b-cd74-4678-99dc-930c26d95529%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ip type and support for a port?

[Chris Neal]

Hi all,

I'm trying to use the ip type in ES, but my IPs also have ports.  That
doesn't seem to be supported, which was a bit of a surprise!

Does anyone know of a way to do this?  Or does it sound like a good feature
to add support for to this type?

Thanks!
Chris

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAND3DpgEWdxZcaFe1wmSWtjRfXqxvUC0vbq8bfE0mC7DusS_%2Bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Unique data by referencing 2 fields

[Cameron Barker]

Hi all,

Would it be possible to get unique items from an elastic search database 
that reference 2 fields for uniqueness, all while using only elastic search 
or a plug in?

*I.E.*
*Initial Data:*
{
 provider: "tumblr"
 text: "I need to get this."
}
{
 provider: "twitter"
 text: "I need to get this."
}
{
 provider: "tumblr"
 text: "I need to get this."
}

*Return:*
{
 provider: "twitter"
 text: "I need to get this."
}
{
 provider: "tumblr"
 text: "I need to get this."
}

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e67f33df-643e-4840-a056-271b5db217ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

is there a query that can return a combined parent child

[Stephen Ward]

I've got my rivers working my parent child mapping done.

I've written some "has_child" querys   but I'm a noob to ES is there any 
way to "join" the data . i.e. aggs and bucketing the children ?
if so does anyone have an example.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/692366f0-11eb-42ae-bf17-cce2788ea586%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: "Failed to configure logging" error on start up.

[Peter Li]

Did more experiments. If I used a "real" scripts directory, instead of a 
symbolic link,
then no error message. But does this means that I will have to drop the 
same script
into all my server's config/scripts directory ? It would be nice to use 
symbolic links
for this.

Any suggestions ?

On Wednesday, July 30, 2014 1:36:24 PM UTC-5, Peter Li wrote:
>
> I have a setup with multiple servers.
> The file tree for each is like the following:
>
> /data/
>configs/
>  elastic-1.yml
>  logging-1.yml
>  scripts/
>(empty)
>elastic-core/   (from distribution)
>  bin/...
>  config/...
>  lib/...
>  logs/...
>elastic-1/
>   bin --> ../elastic-core/bin
>   config/
>  elasticsearch.yml --> ../../configs/elastic-1.yml
>  logging.yml --> ../../configs/logging-1.yml
>  scripts --> ../../configs/scripts
>   data/...
>   lib --> ../elastic-core/lib
>   logs/...
>
> In the elastic-1.yml, I have:
>
>path.conf=/data/elastic-1/config
>
> When I start the node without the config/scripts symbolic link:
>
> /data/elastic-1/bin/elasticsearch 
> -Des.config=/data/elastic-1/config/elasticsearch.yml
>
> It runs fine. But if I have the scripts link/directory, it complains of:
>
> Failed to configure logging...
> org.elasticsearch.ElasticsearchException: Failed to load logging 
> configuration
> at 
> org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:117)
> at 
> org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:81)
> at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstrap.java:94)
> at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:178)
> at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
> Caused by: java.nio.file.FileSystemException: /data/elastic-1/config: 
> Unknown error 1912615013
> at sun.nio.fs.UnixException.translateToIOException(Unknown Source)
> at sun.nio.fs.UnixException.asIOException(Unknown Source)
> at 
> sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.readNextEntry(Unknown 
> Source)
> at sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.hasNext(Unknown 
> Source)
> at java.nio.file.FileTreeWalker.walk(Unknown Source)
> at java.nio.file.FileTreeWalker.walk(Unknown Source)
> at java.nio.file.Files.walkFileTree(Unknown Source)
> at 
> org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:107)
> ... 4 more
> log4j:WARN No appenders could be found for logger (common.jna).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for 
> more info.
>
> Any guesses as to why it is complaining ?
>
> Thanks in advance.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5a010109-7f5e-43a4-b0b6-1ee59957fb8d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ORA-01882: timezone region not found

[joergpra...@gmail.com]

Ups, I mean of course, this is *not*  ES related ...

Jörg


On Wed, Jul 30, 2014 at 7:59 PM, joergpra...@gmail.com <
joergpra...@gmail.com> wrote:

> This is ES related, but, what Oracle JDBC version is this and what Oracle
> Database Server version?
>
> Jörg
>
>
> On Wed, Jul 30, 2014 at 3:59 PM, George DRAGU 
> wrote:
>
>> Hello,
>>
>> Is it any possibility to specify a parameter value to java command line
>> behind the JDBC River?
>> I think at a -Duser.timezone=Europe/Istanbul, for exemple.
>> When I try to create a JDBC River for an Oracle database (with jprante
>> plugin) I catch this error.
>>
>> Thanks
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/a75e8e67-01f8-4b20-a2e9-86caba59e5aa%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFSnRensOgxnZ73HKTNocSwfPJ%3DdbT4S14TzQfctogYtg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

"Failed to configure logging" error on start up.

[Peter Li]

I have a setup with multiple servers.
The file tree for each is like the following:

/data/
   configs/
 elastic-1.yml
 logging-1.yml
 scripts/
   (empty)
   elastic-core/   (from distribution)
 bin/...
 config/...
 lib/...
 logs/...
   elastic-1/
  bin --> ../elastic-core/bin
  config/
 elasticsearch.yml --> ../../configs/elastic-1.yml
 logging.yml --> ../../configs/logging-1.yml
 scripts --> ../../configs/scripts
  data/...
  lib --> ../elastic-core/lib
  logs/...

In the elastic-1.yml, I have:

   path.conf=/data/elastic-1/config

When I start the node without the config/scripts symbolic link:

/data/elastic-1/bin/elasticsearch 
-Des.config=/data/elastic-1/config/elasticsearch.yml

It runs fine. But if I have the scripts link/directory, it complains of:

Failed to configure logging...
org.elasticsearch.ElasticsearchException: Failed to load logging 
configuration
at 
org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:117)
at 
org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:81)
at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstrap.java:94)
at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:178)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: java.nio.file.FileSystemException: /data/elastic-1/config: 
Unknown error 1912615013
at sun.nio.fs.UnixException.translateToIOException(Unknown Source)
at sun.nio.fs.UnixException.asIOException(Unknown Source)
at 
sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.readNextEntry(Unknown 
Source)
at sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.hasNext(Unknown 
Source)
at java.nio.file.FileTreeWalker.walk(Unknown Source)
at java.nio.file.FileTreeWalker.walk(Unknown Source)
at java.nio.file.Files.walkFileTree(Unknown Source)
at 
org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:107)
... 4 more
log4j:WARN No appenders could be found for logger (common.jna).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for 
more info.

Any guesses as to why it is complaining ?

Thanks in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c4afc564-eae2-4826-8675-3f968bd7133b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch always uses the default mapping

[David Pilato]

You did specify the type. But you sent the put mapping request in the wrong 
index.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

> Le 30 juil. 2014 à 16:08, Amira BELDJILALI  a écrit :
> 
> ah, yes, i didn't specify the type, thank you so much for your help
> 
> 
>> On 30 July 2014 16:03, David Pilato  wrote:
>> You applied a mapping to index _river and type newindex.
>> 
>> This is not what I said. You need to apply your mapping to newindex index 
>> and newindex type.
>> 
>> Basically something like:
>> PUT /newindex/
>> 
>> PUT /newindex/newindex/_mapping
>> {
>>  
>>   "newindex" : {
>>"properties": {
>>"marques": {"type": "string"},
>>"ventes": {"type": "long"},
>>"mois":{"type": "date"}
>>  }
>>  
>>  
>>  }
>> }
>> 
>> 
>> 
>> -- 
>> David Pilato | Technical Advocate | Elasticsearch.com
>> @dadoonet | @elasticsearchfr
>> 
>> 
>>> Le 30 juillet 2014 à 15:58:02, Amirah (beldjilal...@gmail.com) a écrit:
>>> 
>>> I don't really see the problem, i selected my newindex ( it exsists in my 
>>> mapping with my types) 
>>> 
>>> PUT /newindex/
>>> 
>>> PUT /_river/newindex/_mapping
>>> {
>>>  
>>>   "newindex" : {
>>>"properties": {
>>>"marques": {"type": "string"},
>>>"ventes": {"type": "long"},
>>>"mois":{"type": "date"}
>>>  }
>>>  
>>>  
>>>  }
>>> }
>>> PUT /_river/newindex/_meta
>>> {
>>>   "type": "csv",
>>>   "csv_file": {
>>> "folder": "E:/elasticsearch-1.2.2/tmp",
>>> "poll": "5m",
>>>  "first_line_is_header":"true",
>>> "field_separator": ";",
>>> "escape_character": " ",
>>> "quote_character": "'"
>>>   }
>>> ,
>>> 
>>> "index" : {
>>>"index" : "newindex",
>>>"type" : "newindex",
>>>"bulk_size" : 100,
>>>"bulk_threshold" : 10
>>>}
>>>  
>>>
>>> }
>>> 
>>> In addition, when i create the newindex it showed {
>>>"_index": "_river",
>>>"_type": "newindex",
>>>"_id": "_meta",
>>>"_version": 1,
>>>"created": true
>>> }
>>> newindex exists in _river with correct types
>>> 
>>> 
>>> 
>>> 
 On Wednesday, July 30, 2014 2:06:50 PM UTC+2, David Pilato wrote:
 That's the problem.
 
 A River creates documents in another index than _river.
 
 If I look at the river documentation, you can set it using:
 
 "index" : {
 "index" : "my_csv_data",
 "type" : "csv_type",
 "bulk_size" : 100,
 "bulk_threshold" : 10
 }
 
 So basically, you need to define your aping for index my_csv_data and type 
 csv_type
 
 See https://github.com/AgileWorksOrg/elasticsearch-river-csv#full-request
 
 
 -- 
 David Pilato | Technical Advocate | Elasticsearch.com
 @dadoonet | @elasticsearchfr
 
 
> Le 30 juillet 2014 à 13:54:53, Amirah (beldji...@gmail.com) a écrit:
> 
> there is a missing part ( copy paste error) /_river/
> So, yes i use this 
> PUT /_river/newindex/_mapping
> {
>  "newindex" : {
>  
>  "properties": {
>"MyStringValue": {"type": "string"},
>"MyLongValue": {"type": "long"},
>"MyDateValue":{"type": "date"}
>  }
>  }
>  }
>
> }
> 
> to create the mapping, my variables ( type names ) are : MyStringValue 
> and its type is String, MyLongValue and its type is long, MyDateValue and 
> its type is date.
> It creates the mapping ( i already checked using GET _mapping ) it exists 
> with my real types ! but once i use my csv file, it changes to string 
> everywhere!
> 
> 
> 
>> On Wednesday, July 30, 2014 12:34:25 PM UTC+2, David Pilato wrote:
>> This looks strange to me
>> 
>> PUT /newindex/_mapping
>>  {
>>   "newindex" : {
>>   
>>   "properties": {
>> "MyStringValue": {"type": "string"},
>> "MyLongValue": {"type": "long"},
>> "MyDateValue":{"type": "date"}
>>   }
>>   }
>>   }
>>  
>> }
>> 
>> What is your type name?
>> 
>> -- 
>> David Pilato | Technical Advocate | Elasticsearch.com
>> @dadoonet | @elasticsearchfr
>> 
>> 
>>> Le 30 juillet 2014 à 11:14:42, Amirah (beldji...@gmail.com) a écrit:
>>> 
>>> Thanks for the answer,
>>> Am creating and defining my mapping ( and index) as following :
>>> 
>>> 
>>> PUT /newindex/
>>> 
>>> PUT /newindex/_mapping
>>>  {
>>>   "newindex" : {
>>>   
>>>   "properties": {
>>> "MyStringValue": {"type": "string"},
>>> "MyLongValue": {"type": "long"},
>>> "MyDateValue":{"type": "date"}
>>>   }
>>>   }
>>>   }
>>>  
>>> }
>>> PUT /_river/newindex/_meta
>>> {
>>>"type": "csv",
>>>"csv_file": {
>>>  "folder": "E:/elasticsearch-1.2.2/tmp",
>>>  "poll": "5m",
>>>   "first_lin

Re: [java api] Trying to use groovy instead of mvel

[David Pilato]

It is!

The issue you run into is "just" a Java dependency issue. Clients don't need 
for example to have Groovy. That's the reason it's marked as optional 
dependency.

Best.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

> Le 30 juil. 2014 à 17:55, "Laurent T."  a écrit :
> 
> Just FYI, if anyone else runs into the same troubles, Groovy seems to be 
> provided on a real cluster and it's in version 2.3.2.
> 
> 
>> On Wednesday, July 30, 2014 1:19:17 PM UTC+2, Laurent T. wrote:
>> Ok well, anyway i think you may want to update the docs about this cause i 
>> think i won't be the only one facing this :)
>> Thanks again to both of you.
>> 
>>> On Wednesday, July 30, 2014 12:30:09 PM UTC+2, David Pilato wrote:
>>> Ha! Right!  Thanks Jörg!
>>> 
>>> I forgot that I run the same "issue" recently. I should add more memory to 
>>> my brain cluster :)
>>> 
>>> -- 
>>> David Pilato | Technical Advocate | Elasticsearch.com
>>> @dadoonet | @elasticsearchfr
>>> 
>>> 
 Le 30 juillet 2014 à 12:08:58, joerg...@gmail.com (joerg...@gmail.com) a 
 écrit:
 
 The ES team decided to postpone groovy as default to Elasticsearch 1.4 
 version.
 
 In 1.3, mvel is still the default, so authors have some time to rewrite 
 their scripts if they prefer to. So I think it is ok to not include groovy 
 jar by default, and make this optional to those who want to switch over to 
 groovy.
 
 Jörg
 
 
> On Wed, Jul 30, 2014 at 11:53 AM, Laurent T.  wrote:
> Nice catch Jörg, that indeed did the trick.
> 
> @David Shouldn't groovy be bundled in the ES jar if it's the new default ?
> Will it be provided by ES when i run on a live cluster ?
> 
> Thanks!
> 
>> On Wednesday, July 30, 2014 11:41:23 AM UTC+2, Jörg Prante wrote:
>> You should try to add groovy jar to your classpath. It is not in the 
>> dependencies in Maven's pom.xml.
>> 
>> Example:
>> 
>> 
>> org.codehaus.groovy
>> groovy-all
>> 2.3.5
>> jar
>> 
>> 
>> Jörg
>> 
>> 
>> 
>>> On Wed, Jul 30, 2014 at 10:46 AM, Laurent T.  wrote:
>>> Hi David,
>>> 
>>> I tried, as you suggested, to activate dynamic scripting and to force 
>>> groovy as a default_lang but the results stay unchanged.
> --
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/0c285769-7872-41cd-9efe-6c97ed8d3df4%40googlegroups.com.
> 
> For more options, visit https://groups.google.com/d/optout.
 
 --
 You received this message because you are subscribed to the Google Groups 
 "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG9mwEK_65v084icNeLvtk2S6eEwk4cPu2sZNXCPDgZ0Q%40mail.gmail.com.
 For more options, visit https://groups.google.com/d/optout.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/6f8752e8-3b2f-40f1-b274-6a94558cf7ec%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/03648E8D-AB08-4603-B014-78B273A947F5%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Re: ORA-01882: timezone region not found

[joergpra...@gmail.com]

This is ES related, but, what Oracle JDBC version is this and what Oracle
Database Server version?

Jörg


On Wed, Jul 30, 2014 at 3:59 PM, George DRAGU 
wrote:

> Hello,
>
> Is it any possibility to specify a parameter value to java command line
> behind the JDBC River?
> I think at a -Duser.timezone=Europe/Istanbul, for exemple.
> When I try to create a JDBC River for an Oracle database (with jprante
> plugin) I catch this error.
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/a75e8e67-01f8-4b20-a2e9-86caba59e5aa%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHqCoSbbw_OidTVbyr7AB14R07Ag%2Bg1x6dBU2uUteV2Sg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)

[Ivan Brusic]

The idea is that the cluster should be delayed when a cluster rebalance
occurs, but even with these settings, I often find that shards are moved
immediately.

Are you using the default stores throttling settings? I found them to be
quite low.

Cheers,

Ivan


On Wed, Jul 30, 2014 at 6:02 AM, Andrew Davidoff  wrote:

>
>
> On Tuesday, July 29, 2014 3:27:13 PM UTC-4, Ivan Brusic wrote:
>>
>> Have you changed your gateway settings? http://www.
>> elasticsearch.org/guide/en/elasticsearch/reference/
>> current/modules-gateway.html#recover-after
>>
>> It still remains a bit of black magic to me. Sometimes it works,
>> sometimes it does not.
>>
>
> Ivan,
>
> I have read over that documentation several times and I don't understand
> how it would help me. By which I mean I honestly don't understand how it
> would help me - not that I am unwilling to try it. Those settings read like
> they control when recovery would begin, but my problem isn't that recovery
> is starting when I don't want it to, it's that when I start it (by setting
> shard allocation about to "all") shards that I'd think would just stick
> with the node they were previously on, get assigned to other nodes, then
> ultimately get rebalanced back to the original node.
>
> At this point I am finding that for quick restarts, just doing them with
> no additional prep work allows me to recover in ~30m, vs ~24h. So for now
> I'm just going to do that. Whatever I am doing wrong here just isn't at all
> clear to me.
>
> Thanks for your advice. If I have misunderstood the settings you pointed
> me at and you think you can help me understand, I'd be grateful for more
> information.
>
> Andy
>
>
>
>> Cheers,
>>
>> Ivan
>>
>>
>> On Mon, Jul 28, 2014 at 1:52 PM, Andrew Davidoff 
>> wrote:
>>
>>> I guess I'll ask about this once more for now. This happened again
>>> today. I set allocation to new_primaries, restarted a node, set allocation
>>> back to all, and the cluster is assigning across all nodes the shards that
>>> were on the restarted node, and when it's done, which will probably take a
>>> day, it'll likely rebalance by moving them back to the original node. I
>>> have to assume I'm doing something wrong here. Am I?
>>>
>>> Thanks for any advice.
>>> Andy
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3%
>>> 40googlegroups.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/b8cf076f-2494-4b0e-b6c3-5a21d1c9f9a1%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBnsTUiXYZVyUgnuU6k7nN8NJiwh2UK_JPTOhEVa8BCRA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Logging every query

[Alejandro de la Viña]

've got an enviroment set on Dev that should keep a log with every query 
ran, but it's not writing anything. I'm using the slow-log feature for 
it... 

These are my thresholds on the elasticsearch.yml:

http://pastebin.com/raw.php?i=qfwnruhD

And this is my whole logging.yml:

http://pastebin.com/raw.php?i=aXg8xHNE

I'm using ElasticSearch 1.3.1 in this enviroment.

However, the file is not moving forward:

-rw-r--r-- 1 root staff   0 Jul 29 13:23 
elastictest_index_search_slowlog.log

Elastic is running as root as well. Any thoughts on what I am missing?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4f9847b6-4cee-4df8-afef-272368a0d488%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: log index creation API requests

[Ivan Brusic]

The logging.xml file will only control which logging statements get
outputed, not the amount of information it may contain.

The log line in question does not have the source ip, which is long gone by
the time the service gets the request.

https://github.com/elasticsearch/elasticsearch/blob/a848b658e89884b0bb3a40c8de1d912412890fa8/src/main/java/org/elasticsearch/cluster/metadata/MetaDataCreateIndexService.java#L412

-- 
Ivan


On Wed, Jul 30, 2014 at 5:15 AM,  wrote:

> Hi - any tips for how I should configure the logging.yml file to give me
> more verbose output, including source ip address if possible, to give more
> info when an index is created?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/2b423a9b-1e60-4e53-87b0-4681be643683%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBadWwueKO%2Bd_A-NdUiE0Cf3r32tY4TUha%2BrgzdpeobFA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

How to use Curator to manage old data and avoid running out of storage space?

[David Reagan]

I've been implementing an ELK stack for the past year or so. I had thought 
that we would have plenty of space, but recently added a log source that 
increased the number of log entries a day by around 30x. That prompted me 
to start looking into ways of managing ES's data storage in order to keep 
from running out of space. Which led me to Curator and Snapshots.

If I am reading the documentation[1] for both systems correctly, I think I 
can do the following:

   - Create a repository for old data.
   - Use a cron job and Curator to automatically take snapshots of data 
   older than a certain time period (say, 6 months).
  - Then have Curator delete the data older than that time period.
  - The result would be that all data older than the time period would 
  be stored in the repository. The data would be compressed (what kind of 
  compression?)
   - When I have need for data older than the time period, I could use 
   Curator to restore it to the ES cluster, or even a different ES cluster. 
  - After that I could do what I needed, before deleting it again.
   

I'd test all this myself, but I don't have the resources for a decent test 
environment yet. :( Still working on that. 

Am I missing anything? Are there better ways to keep from running out of 
storage space? Any general advice related to this kind of thing?

Thanks in advance!

[1] 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
https://github.com/elasticsearch/curator/wiki
http://www.elasticsearch.org/blog/elasticsearch-curator-version-1-1-0-released/

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8957c038-f6d7-47d9-8225-5a975454aa54%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Java transport client, which hosts to add?

[Andrew Gaydenko]

On Wednesday, July 30, 2014 8:13:28 PM UTC+4, Ivan Brusic wrote:
>
> You should as many nodes as possible. If you 
> enable client.transport.sniff, then the transport client will ask the nodes 
> it does connect to about the other nodes in the cluster, which means you 
> can potentially only need to specific a single node (not ideal in case that 
> node is down).
>

Ivan, I see, thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/83dfce66-b3a8-4222-9ca4-93cbfe064a63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Geo distance filter exceptions

[Madhavan Ramachandran]

Nope.. it did not work..got exception as 
QueryParsingException[[offlocations_geo] failed to find geo_point field 
[city.location

Regards
Madhavan.TR
On Wednesday, July 30, 2014 10:08:45 AM UTC-5, Joffrey Hercule wrote:
>
> Hi !
> Use  "query".
>
> ex :
> {
>   "query" : {
> "filtered" : {
> "query" : {
> "match_all" : {}
> },
> "filter" : {
> "geo_distance" : {
> "distance" : "50km",
> "city.location" : {
> "lat" : 43.4,
> "lon" : 5.4
> }
> }
> }
> }
>   }
> }
>
> Le mardi 29 juillet 2014 22:30:12 UTC+2, Madhavan Ramachandran a écrit :
>>
>> Hi Team,
>>
>> I am trying to find a solution for the below
>>
>>1. Geo boundary based search.. My index have property for lat and lon 
>>as double.. not as a geopoint.. Here is my mapping for my index..
>>
>> How do i use the lon and lat from the below mapping for geo distance 
>> filter/geo distance range filter ?
>>
>> Name Type Format Store?
>> data string
>>
>> getpath string
>>
>> id double
>>
>> Region string
>>
>> Submarket string
>>
>> addr1 string
>>
>> addr2 string
>>
>> city string
>>
>> citymarket string
>>
>> country string
>>
>> countryid long
>>
>> cultureid long
>>
>> data string
>>
>> details string
>>
>> fax string
>>
>> id string
>>
>> language string
>>
>> lat double
>>
>> lon double
>>
>> When I search for the documents.. i got the below exception..
>>
>> Query : 
>> {
>> "filter": {
>> "geo_distance" : {
>> "distance" : "300km",
>> "location" : {
>> "lat" : 45,
>> "lon" : -122
>> }
>> }
>>
>> }
>> }
>>
>> Exception: 
>> "error": "SearchPhaseExecutionException[Failed to execute phase [query], 
>> all shards failed; shardFailures 
>> {[o3f66HetT3OSpVw895w0nA][offlocations][4]: 
>> SearchParseException[[offlocations][4]: from[-1],size[-1]: Parse Failure 
>> [Failed to parse source [{\n\"filter\": {\n \"geo_distance\" : {\n 
>> \"distance\" : \"300km\",\n \"location\" : {\n \"lat\" : 45,\n \"lon\" : 
>> -122\n }\n } \n }\n}]]]; nested:
>>
>> I tried with removing the location, which i dont have in my mapping.. 
>>
>> {"filter": { "geo_distance" : { "distance" : "300km",
>> "lat" 
>> : 45, "lon" : -122   }  }}
>>
>> I got the exception as lon is not a geo_point field..
>>
>> ElasticsearchIllegalArgumentException[the character '-' is not a valid 
>> geohash character]; }]"
>>
>> If i remove the "-" infront of lon.. then the exception says : 
>>
>> QueryParsingException[[offlocations] field [lon] is not a geo_point 
>> field];
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/11cd8b18-7057-4190-a3d0-463bd59b7f5a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Java transport client, which hosts to add?

[Ivan Brusic]

You should as many nodes as possible. If you enable client.transport.sniff,
then the transport client will ask the nodes it does connect to about the
other nodes in the cluster, which means you can potentially only need to
specific a single node (not ideal in case that node is down).

-- 
Ivan


On Wed, Jul 30, 2014 at 8:36 AM, Andrew Gaydenko 
wrote:

> The client has addTransportAddress(). So, I can add all cluster nodes. Is
> it intended way? Or - what are those considerations must be taken into
> account while adding hosts?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/13d2a061-5418-40bd-9233-72c432aa7643%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQC7i_wT_r8kiXYSSee92S%3DJUtp3uROZ1wgO8znQ26c7Uw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: slow filter execution

[Kireet Reddy]

Thanks for the detailed reply. 

I am a bit confused about and vs bool filter execution. I read this post 
 on 
the elasticsearch blog. From that, I thought the bool filter would work by 
basically creating a bitset for the entire segment(s) being examined. If 
the filter value changes every time, will this still be cheaper than an AND 
filter that will just examine the matching docs? My segments can be very 
big and this query for example on matched one document.

There is no match_all query filter, There is a "match" query filter on a 
field named "all". :)

Based on your feedback, I moved all filters, including the query filter, 
into the bool filter. However it didn't change things: the query takes an 
order of magnitude slower with the range filter, unless I set execution to 
fielddata. I am using 1.2.2, I tried the strategy anyways and it didn't 
make a difference.

{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"terms": {
"source_id": ["s1", "s2", "s3"]
}
},
{
"query": {
"match": {
"all": {
"query": "foo"
}
}
}
},
{
"range": {
"published": {
"to": 1406064191883
}
}
}
]
}
}
}
},
"sort": [
{
"crawlDate": {
"order": "desc"
}
}
]
}

On Wednesday, July 30, 2014 4:30:10 AM UTC-7, Clinton Gormley wrote:
>
> Don't use the `and` filter - use the `bool` filter instead.  They have 
> different execution modes and the `bool` filter works best with bitset 
> filters (but also knows how to handle non-bitset filters like geo etc).  
>
> Just remove the `and`, `or` and `not` filters from your DSL vocabulary.
>
> Also, not sure why you are ANDing with a match_all filter - that doesn't 
> make much sense.
>
> Depending on which version of ES you're using, you may be encountering a 
> bug in the filtered query which ended up always running the query first, 
> instead of the filter. This was fixed in v1.2.0 
> https://github.com/elasticsearch/elasticsearch/issues/6247 .  If you are 
> on an earlier version you can force filter-first execution manually by 
> specifying a "strategy" of "random_access_100".  See 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-filtered-query.html#_filter_strategy
>
> In summary, (and taking your less granular datetime clause into account) 
> your query would be better written as:
>
> GET /_search
> {
>   "query": {
> "filtered": {
>   "strategy": "random_access_100",   pre 1.2 only
>   "filter": {
> "bool": {
>   "must": [
> {
>   "terms": {
> "source_id": [ "s1", "s2", "s3" ]
>   }
> },
> {
>   "range": {
> "published": {
>   "gte": "now-1d/d"   coarse grained, cached
> }
>   }
> },
> {
>   "range": {
> "published": {
>   "gte": "now-30m"  fine grained, not cached, 
> could use fielddata too
> },
> "_cache": false
>   }
> }
>   ]
> }
>   }
> }
>   }
> }
>
>
>
>
>
> On 30 July 2014 10:55, David Pilato > wrote:
>
>> May be a stupid question: why did you put that filter inside a query and 
>> not within the same filter you have at the end?
>>
>>
>> For my test case it's the same every time. In the "real" query it will 
>>> change every time, but I planned to not cache this filter and have a less 
>>> granular date filter in the bool filter that would be cached. However while 
>>> debugging I noticed slowness with the date range filters even while testing 
>>> with the same value repeatedly.
>>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discuss

Re: [java api] Trying to use groovy instead of mvel

[Laurent T.]

Just FYI, if anyone else runs into the same troubles, Groovy seems to be 
provided on a real cluster and it's in version 2.3.2.


On Wednesday, July 30, 2014 1:19:17 PM UTC+2, Laurent T. wrote:
>
> Ok well, anyway i think you may want to update the docs about this cause i 
> think i won't be the only one facing this :)
> Thanks again to both of you.
>
> On Wednesday, July 30, 2014 12:30:09 PM UTC+2, David Pilato wrote:
>>
>> Ha! Right!  Thanks Jörg!
>>
>> I forgot that I run the same "issue" recently. I should add more memory 
>> to my brain cluster :)
>>
>> -- 
>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
>> @dadoonet  | @elasticsearchfr 
>> 
>>
>>
>> Le 30 juillet 2014 à 12:08:58, joerg...@gmail.com (joerg...@gmail.com) a 
>> écrit:
>>
>> The ES team decided to postpone groovy as default to Elasticsearch 1.4 
>> version. 
>>
>> In 1.3, mvel is still the default, so authors have some time to rewrite 
>> their scripts if they prefer to. So I think it is ok to not include groovy 
>> jar by default, and make this optional to those who want to switch over to 
>> groovy.
>>
>> Jörg
>>  
>>
>> On Wed, Jul 30, 2014 at 11:53 AM, Laurent T.  wrote:
>>
>>> Nice catch Jörg, that indeed did the trick. 
>>>
>>> @David Shouldn't groovy be bundled in the ES jar if it's the new default 
>>> ?
>>> Will it be provided by ES when i run on a live cluster ?
>>>
>>> Thanks!
>>>  
>>> On Wednesday, July 30, 2014 11:41:23 AM UTC+2, Jörg Prante wrote:
>>>
  You should try to add groovy jar to your classpath. It is not in the 
 dependencies in Maven's pom.xml. 

 Example:

  
 org.codehaus.groovy
 groovy-all
 2.3.5
 jar
 
  
 Jörg

  

 On Wed, Jul 30, 2014 at 10:46 AM, Laurent T.  
 wrote:

> Hi David, 
>
> I tried, as you suggested, to activate dynamic scripting and to force 
> groovy as a default_lang but the results stay unchanged.
>  
 --
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/0c285769-7872-41cd-9efe-6c97ed8d3df4%40googlegroups.com
>>>  
>>> .
>>>  
>>>
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>  
>>  
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG9mwEK_65v084icNeLvtk2S6eEwk4cPu2sZNXCPDgZ0Q%40mail.gmail.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6f8752e8-3b2f-40f1-b274-6a94558cf7ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Java transport client, which hosts to add?

[Andrew Gaydenko]

The client has addTransportAddress(). So, I can add all cluster nodes. Is 
it intended way? Or - what are those considerations must be taken into 
account while adding hosts?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/13d2a061-5418-40bd-9233-72c432aa7643%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Recommendations needed for large ELK system design

[Alex]

Hello,

We wish to set up an entire ELK system with the following features:

   - Input from Logstash shippers located on 400 Linux VMs. Only a handful 
   of log sources on each VM.
   - Data retention for 30 days, which is roughly 2TB of data in indexed ES 
   JSON form (not including replica shards)
   - Estimated input data rate of 50 messages per second at peak hours. 
   Mostly short or medium length one-line messages but there will be Java 
   traces and very large service responses (in the form of XML) to deal with 
   too.
   - The entire system would be on our company LAN.
   - The stored data will be a mix of application logs (info, errors etc) 
   and server stats (CPU, memory usage etc) and would mostly be accessed 
   through Kibana.

This is our current plan:

   - Have the LS shippers perform minimal parsing (but would do multiline). 
   Have them point to two load-balanced servers containing Redis and LS 
   indexers (which would do all parsing).
   - 2 replica shards for each index, which ramps the total data storage up 
   to 6TB
   - ES cluster spread over 6 nodes. Each node is 1TB in size
   - LS indexers pointing to cluster.

So I have a couple questions regarding the setup and would greatly 
appreciate the advice of someone with experience!

   1. Does the balance between the number of nodes, the number of replica 
   shards, and storage size of each node seem about right? We use 
   high-performance equipment and would expect minimal downtime.
   
   2. What is your recommendation for the system design of the LS indexers 
   and Redis? I've seen various designs with each indexer assigned to a single 
   Redis, or all indexers reading from all Redises.
   
   3. Leading from the previous question, what would your recommend data 
   size for the Redis servers be?
   
   4. Not sure what to do about master/data nodes. Assuming all the nodes 
   are on identical hardware would it be beneficial to have a node which is 
   only a master which would only handle requests?
   
   5. Do we need to do any additional load balancing on the ES nodes?

We are open to any and all suggestions. We have not yet committed to any 
particular design so can change if needed.

Thank you for your time and responses,
Alex

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b0aee66a-35bb-4770-927b-d9c7e13ad9fc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

best way to parse deeply nested aggregations using client API

[birjupat]

I am using Java client API to get aggregations back. Following is the 
structure which I am dealing with.


aggregations
   top_models
   buckets
   key : "BMW"
  doc_count : 3
  top_models
   buckets
   key : "X5"
   doc_count : 2
  top_hits
   source
  model : X5
  color : Red
  source
 model:X5
color : White
   key : "X3"
   doc_count : 1
  top_hits
   source
  model : X3
  color : Red
  key : "Mercedes"
  doc_count : 2
  top_models
   buckets
   key : "Benz"
   doc_count : 1
   top_hits
   source
  model : Benz
  color : Red
  
  key : "ML"
  doc_count : 1
  top_hits
   source
  model : ML
  color : Black

I am trying following (toy) code to retrieve all the results. 

def getAggregations(aggres: Option[Aggregations]): Option[Iterable[Any]] = {

aggres map { agg =>

  val aggS = agg.asMap().asScala

  aggS map {

case (name, termAgg: Terms) => getBuckets(Option(termAgg.getBuckets
()))

case (name, topHits: TopHits) =>

  val tHits = Option(topHits.getHits())

  tHits map { th => getTopHits(th.asScala)
}

case (h, a: InternalAvg) => println(h + "=>" + a.getValue());

  }



}

  }



  def getBuckets(buckets: Option[java.util.Collection[Bucket]]) = {

buckets map { bks =>

  val bksS = bks.asScala

  bksS map { b =>

println("Bucket Key =>" + b.getKey())

println("Doc count =>" + b.getDocCount())

getAggregations(Option(b.getAggregations())

  }

}

 }

I need to populate final result to this class 

  
case class FinalResponse(bucketName: String, count: Long, children: List[
FinalResponse])

With nested relationship between Aggregations and Buckets it's becoming 
convoluted to retrieve all aggregation results. how do you approach this?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b47ef1bd-9a15-4d37-94f2-95b687dfc0d4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Geo distance filter exceptions

[Joffrey Hercule]

Hi !
Use  "query".

ex :
{
  "query" : {
"filtered" : {
"query" : {
"match_all" : {}
},
"filter" : {
"geo_distance" : {
"distance" : "50km",
"city.location" : {
"lat" : 43.4,
"lon" : 5.4
}
}
}
}
  }
}

Le mardi 29 juillet 2014 22:30:12 UTC+2, Madhavan Ramachandran a écrit :
>
> Hi Team,
>
> I am trying to find a solution for the below
>
>1. Geo boundary based search.. My index have property for lat and lon 
>as double.. not as a geopoint.. Here is my mapping for my index..
>
> How do i use the lon and lat from the below mapping for geo distance 
> filter/geo distance range filter ?
>
> Name Type Format Store?
> data string
>
> getpath string
>
> id double
>
> Region string
>
> Submarket string
>
> addr1 string
>
> addr2 string
>
> city string
>
> citymarket string
>
> country string
>
> countryid long
>
> cultureid long
>
> data string
>
> details string
>
> fax string
>
> id string
>
> language string
>
> lat double
>
> lon double
>
> When I search for the documents.. i got the below exception..
>
> Query : 
> {
> "filter": {
> "geo_distance" : {
> "distance" : "300km",
> "location" : {
> "lat" : 45,
> "lon" : -122
> }
> }
>
> }
> }
>
> Exception: 
> "error": "SearchPhaseExecutionException[Failed to execute phase [query], 
> all shards failed; shardFailures 
> {[o3f66HetT3OSpVw895w0nA][offlocations][4]: 
> SearchParseException[[offlocations][4]: from[-1],size[-1]: Parse Failure 
> [Failed to parse source [{\n\"filter\": {\n \"geo_distance\" : {\n 
> \"distance\" : \"300km\",\n \"location\" : {\n \"lat\" : 45,\n \"lon\" : 
> -122\n }\n } \n }\n}]]]; nested:
>
> I tried with removing the location, which i dont have in my mapping.. 
>
> {"filter": { "geo_distance" : { "distance" : "300km",
> "lat" 
> : 45, "lon" : -122   }  }}
>
> I got the exception as lon is not a geo_point field..
>
> ElasticsearchIllegalArgumentException[the character '-' is not a valid 
> geohash character]; }]"
>
> If i remove the "-" infront of lon.. then the exception says : 
>
> QueryParsingException[[offlocations] field [lon] is not a geo_point field];
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/567759b4-21bb-489f-b08f-36ba982a30c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Logging of percolator reverse queries

[Arkadiy Rudin]

Just checking if anybody knows the answer...

On Monday, July 28, 2014 4:14:59 PM UTC-4, Arkadiy Rudin wrote:
>
> Looks like the percolator queries are not getting recorded in any of 
> existing slow query logs.
> Is it something that I am missing in configuration or logging for 
> percolator is not available? 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/51fa2b59-d54e-4f04-8cb0-af5ab0dc2189%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Geo distance filter exceptions

[Madhavan Ramachandran]

Hi 

I have updated the mapping for my index.. added a column with geopoint..

locationgeo_point
when i search without geo filter for data.. i can able to see the location 
information.

{
"_index": "offlocations_geo",
"_type": "officelocations",
"_id": "21",
"_score": 1,
"_source": {
"countryid": 232,
"lat": 32.8641213476658,
"lon": -96.8086239695549,
"id": 21,
"cultureid": 1033,
"country": "US",
"id": "21-10",
"title": "8343 Douglas Avenue",
"getpath": "8343 Douglas Avenue",
"topic": "",
"details": "Dallas, TX 75225 
",
"addr1": "Suite 100",
"addr2": "",
"city": "Dallas",
"province": "TX",
"zip": "75225",
"Region": "Americas",
"market": "",
"phone": "",
"fax": "+",
"language": "en-us",
"location": {
"lat": 32.8641213476658,
"lon": -96.8086239695549
}
}
}


When i add the geofilter query .. again getting exception..
{
"filtered" : {
"query" : {
"match_all" : {}
},
"filter" : {
"geo_distance" : {
"distance" : "10km",

"location": {
"lat": 41.0453822,
"lon": -73.5474359
}

}
}
}
}

exception info:

nested: SearchParseException[[offlocations_geo][2]: from[-1],size[-1]: 
Parse Failure [No parser for element [filtered]]]; }]",
"status": 400

Regards
Madhavan.TR

On Wednesday, July 30, 2014 8:48:22 AM UTC-5, Madhavan Ramachandran wrote:
>
> Thanks for your response. 
>
> I am using Nest dll (.Net) to index the data in ES (in windows as a 
> service).  How to add the geopoint to my index columns? 
>
> Regards
> Madhavan.TR
>
>
> On Tuesday, July 29, 2014 3:35:56 PM UTC-5, David Pilato wrote:
>>
>> No you can't out of the box. If you want to use built in geo filters you 
>> need geo points.
>>
>> That said, you can use scripts to do it based on lat,lon fields but it 
>> will be slower.
>>
>> My 2 cents
>>
>> --
>> David ;-)
>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>>
>>
>> Le 29 juil. 2014 à 22:30, Madhavan Ramachandran  a 
>> écrit :
>>
>> Hi Team,
>>
>> I am trying to find a solution for the below
>>
>>1. Geo boundary based search.. My index have property for lat and lon 
>>as double.. not as a geopoint.. Here is my mapping for my index..
>>
>> How do i use the lon and lat from the below mapping for geo distance 
>> filter/geo distance range filter ?
>>
>> Name Type Format Store?
>> data string
>>
>> getpath string
>>
>> id double
>>
>> Region string
>>
>> Submarket string
>>
>> addr1 string
>>
>> addr2 string
>>
>> city string
>>
>> citymarket string
>>
>> country string
>>
>> countryid long
>>
>> cultureid long
>>
>> data string
>>
>> details string
>>
>> fax string
>>
>> id string
>>
>> language string
>>
>> lat double
>>
>> lon double
>>
>> When I search for the documents.. i got the below exception..
>>
>> Query : 
>> {
>> "filter": {
>> "geo_distance" : {
>> "distance" : "300km",
>> "location" : {
>> "lat" : 45,
>> "lon" : -122
>> }
>> }
>>
>> }
>> }
>>
>> Exception: 
>> "error": "SearchPhaseExecutionException[Failed to execute phase [query], 
>> all shards failed; shardFailures 
>> {[o3f66HetT3OSpVw895w0nA][offlocations][4]: 
>> SearchParseException[[offlocations][4]: from[-1],size[-1]: Parse Failure 
>> [Failed to parse source [{\n\"filter\": {\n \"geo_distance\" : {\n 
>> \"distance\" : \"300km\",\n \"location\" : {\n \"lat\" : 45,\n \"lon\" : 
>> -122\n }\n } \n }\n}]]]; nested:
>>
>> I tried with removing the location, which i dont have in my mapping.. 
>>
>> {"filter": { "geo_distance" : { "distance" : "300km",
>> "lat" 
>> : 45, "lon" : -122   }  }}
>>
>> I got the exception as lon is not a geo_point field..
>>
>> ElasticsearchIllegalArgumentException[the character '-' is not a valid 
>> geohash character]; }]"
>>
>> If i remove the "-" infront of lon.. then the exception says : 
>>
>> QueryParsingException[[offlocations] field [lon] is not a geo_point 
>> field];
>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/85658110-e7ae-42c5-8ca9-13f10c7f3820%40g

Re: Elasticsearch always uses the default mapping

[Amirah]

ah, yes, i didn't specify the type, thank you so much for your help

On Wednesday, July 30, 2014 4:04:18 PM UTC+2, David Pilato wrote:
>
> You applied a mapping to index _river and type newindex.
>
> This is not what I said. You need to apply your mapping to newindex index 
> and newindex type.
>
> Basically something like:
> PUT /newindex/
>
> PUT /newindex/newindex/_mapping
> {
>  
>   "newindex" : {
>"properties": {
>"marques": {"type": "string"},
>"ventes": {"type": "long"},
>"mois":{"type": "date"}
>  }
>  
>  
>  }
> }
>
>
>
> -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
> @dadoonet  | @elasticsearchfr 
> 
>
>
> Le 30 juillet 2014 à 15:58:02, Amirah (beldji...@gmail.com ) 
> a écrit:
>
> I don't really see the problem, i selected my newindex ( it exsists in my 
> mapping with my types)  
>
>   PUT /newindex/
>  
>  PUT /_river/newindex/_mapping
>  {
>   
>"newindex" : {
> "properties": {
> "marques": {"type": "string"},
> "ventes": {"type": "long"},
> "mois":{"type": "date"}
>   }
>   
>   
>   }
>  } 
>  PUT /_river/newindex/_meta
>  {
>"type": "csv",
>"csv_file": {
>  "folder": "E:/elasticsearch-1.2.2/tmp",
>  "poll": "5m",
>   "first_line_is_header":"true",
>  "field_separator": ";",
>  "escape_character": " ",
>  "quote_character": "'"
>}
>  ,
>  
>  "index" : {
> "index" : "newindex",
> "type" : "newindex",
> "bulk_size" : 100,
> "bulk_threshold" : 10
> }
>   
> 
>  }
>  
> In addition, when i create the newindex it showed {
>"_index": "_river",
>"_type": "newindex",
>"_id": "_meta",
>"_version": 1,
>"created": true
> }
> newindex exists in _river with correct types
>
>
>
>
> On Wednesday, July 30, 2014 2:06:50 PM UTC+2, David Pilato wrote: 
>
>  That's the problem.
>  
>  A River creates documents in another index than _river.
>  
>  If I look at the river documentation, you can set it using:
>  
>  "index" : {
> "index" : "my_csv_data",
> "type" : "csv_type",
> "bulk_size" : 100,
> "bulk_threshold" : 10
> }
>  
> So basically, you need to define your aping for index my_csv_data and type 
> csv_type
>
> See https://github.com/AgileWorksOrg/elasticsearch-river-csv#full-request
>
>
>  -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com* 
>  @dadoonet  | @elasticsearchfr 
> 
>  
>
> Le 30 juillet 2014 à 13:54:53, Amirah (beldji...@gmail.com) a écrit:
>
>  there is a missing part ( copy paste error) /_river/
> So, yes i use this  
>  PUT /_river/newindex/_mapping
>  {
>   "newindex" : {
>   
>   "properties": {
> "MyStringValue": {"type": "string"},
> "MyLongValue": {"type": "long"},
> "MyDateValue":{"type": "date"}
>   }
>   }
>   }
> 
>  }
>  
> to create the mapping, my variables ( type names ) are : *MyStringValue* 
> and its type is *String*, *MyLongValue* and its type is *long*, 
> *MyDateValue* and its type is *date*. 
> It creates the mapping ( i already checked using GET _mapping ) it exists 
> with my real types ! but once i use my csv file, it changes to string 
> everywhere!
>
>
>
> On Wednesday, July 30, 2014 12:34:25 PM UTC+2, David Pilato wrote: 
>
>  This looks strange to me
>
> PUT /newindex/_mapping
>  {
>   "newindex" : {
>   
>   "properties": {
> "MyStringValue": {"type": "string"},
> "MyLongValue": {"type": "long"},
> "MyDateValue":{"type": "date"}
>   }
>   }
>   }
>  
> }
>  
> What is your type name?
>  
>  -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com* 
>  @dadoonet  | @elasticsearchfr 
> 
>  
>
> Le 30 juillet 2014 à 11:14:42, Amirah (beldji...@gmail.com) a écrit:
>
>  Thanks for the answer, 
> Am creating and defining my mapping ( and index) as following :
>
>   
> PUT /newindex/
>
> PUT /newindex/_mapping
>  {
>   "newindex" : {
>   
>   "properties": {
> "MyStringValue": {"type": "string"},
> "MyLongValue": {"type": "long"},
> "MyDateValue":{"type": "date"}
>   }
>   }
>   }
>  
> }
> PUT /_river/newindex/_meta
> {
>"type": "csv",
>"csv_file": {
>  "folder": "E:/elasticsearch-1.2.2/tmp",
>  "poll": "5m",
>   "first_line_is_header":"true",
>  "field_separator": ";",
>  "escape_character": " ",
>  "quote_character": "'"
>}
> }
>   
> and am always getting string in all types ( except date as i put the right 
> format in my csv file *( 2014-07-24T13:58:25.000Z)*
>
> any idea ?
>
>
> On Wednesday, July 30, 2014 10:47:13 AM UTC+2, David Pilato wrote: 
>
>  I think you are doing something wrong.
>  If you defined a mapping it should not be overwritten by the CSV river 
> as fa

Re: Elasticsearch always uses the default mapping

[Amira BELDJILALI]

ah, yes, i didn't specify the type, thank you so much for your help


On 30 July 2014 16:03, David Pilato  wrote:

> You applied a mapping to index _river and type newindex.
>
> This is not what I said. You need to apply your mapping to newindex index
> and newindex type.
>
> Basically something like:
> PUT /newindex/
>
> PUT /newindex/newindex/_mapping
> {
>
>   "newindex" : {
>"properties": {
>"marques": {"type": "string"},
>"ventes": {"type": "long"},
>"mois":{"type": "date"}
>  }
>
>
>  }
> }
>
>
>
>  --
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
> @dadoonet  | @elasticsearchfr
> 
>
>
> Le 30 juillet 2014 à 15:58:02, Amirah (beldjilal...@gmail.com) a écrit:
>
> I don't really see the problem, i selected my newindex ( it exsists in my
> mapping with my types)
>
>   PUT /newindex/
>
>  PUT /_river/newindex/_mapping
>  {
>
>"newindex" : {
> "properties": {
> "marques": {"type": "string"},
> "ventes": {"type": "long"},
> "mois":{"type": "date"}
>   }
>
>
>   }
>  }
>  PUT /_river/newindex/_meta
>  {
>"type": "csv",
>"csv_file": {
>  "folder": "E:/elasticsearch-1.2.2/tmp",
>  "poll": "5m",
>   "first_line_is_header":"true",
>  "field_separator": ";",
>  "escape_character": " ",
>  "quote_character": "'"
>}
>  ,
>
>  "index" : {
> "index" : "newindex",
> "type" : "newindex",
> "bulk_size" : 100,
> "bulk_threshold" : 10
> }
>
>
>  }
>
> In addition, when i create the newindex it showed {
>"_index": "_river",
>"_type": "newindex",
>"_id": "_meta",
>"_version": 1,
>"created": true
> }
> newindex exists in _river with correct types
>
>
>
>
> On Wednesday, July 30, 2014 2:06:50 PM UTC+2, David Pilato wrote:
>>
>>  That's the problem.
>>
>>  A River creates documents in another index than _river.
>>
>>  If I look at the river documentation, you can set it using:
>>
>>  "index" : {
>> "index" : "my_csv_data",
>> "type" : "csv_type",
>> "bulk_size" : 100,
>> "bulk_threshold" : 10
>> }
>>
>> So basically, you need to define your aping for index my_csv_data and
>> type csv_type
>>
>> See https://github.com/AgileWorksOrg/elasticsearch-river-csv#full-request
>>
>>
>>  --
>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
>>  @dadoonet  | @elasticsearchfr
>> 
>>
>>
>> Le 30 juillet 2014 à 13:54:53, Amirah (beldji...@gmail.com) a écrit:
>>
>>  there is a missing part ( copy paste error) /_river/
>> So, yes i use this
>>  PUT /_river/newindex/_mapping
>>  {
>>   "newindex" : {
>>
>>   "properties": {
>> "MyStringValue": {"type": "string"},
>> "MyLongValue": {"type": "long"},
>> "MyDateValue":{"type": "date"}
>>   }
>>   }
>>   }
>>
>>  }
>>
>> to create the mapping, my variables ( type names ) are : *MyStringValue*
>> and its type is *String*, *MyLongValue* and its type is *long*,
>> *MyDateValue* and its type is *date*.
>> It creates the mapping ( i already checked using GET _mapping ) it exists
>> with my real types ! but once i use my csv file, it changes to string
>> everywhere!
>>
>>
>>
>> On Wednesday, July 30, 2014 12:34:25 PM UTC+2, David Pilato wrote:
>>>
>>>  This looks strange to me
>>>
>>> PUT /newindex/_mapping
>>>  {
>>>   "newindex" : {
>>>
>>>   "properties": {
>>> "MyStringValue": {"type": "string"},
>>> "MyLongValue": {"type": "long"},
>>> "MyDateValue":{"type": "date"}
>>>   }
>>>   }
>>>   }
>>>
>>> }
>>>
>>> What is your type name?
>>>
>>>  --
>>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
>>>  @dadoonet  | @elasticsearchfr
>>> 
>>>
>>>
>>> Le 30 juillet 2014 à 11:14:42, Amirah (beldji...@gmail.com) a écrit:
>>>
>>>  Thanks for the answer,
>>> Am creating and defining my mapping ( and index) as following :
>>>
>>>
>>> PUT /newindex/
>>>
>>> PUT /newindex/_mapping
>>>  {
>>>   "newindex" : {
>>>
>>>   "properties": {
>>> "MyStringValue": {"type": "string"},
>>> "MyLongValue": {"type": "long"},
>>> "MyDateValue":{"type": "date"}
>>>   }
>>>   }
>>>   }
>>>
>>> }
>>> PUT /_river/newindex/_meta
>>> {
>>>"type": "csv",
>>>"csv_file": {
>>>  "folder": "E:/elasticsearch-1.2.2/tmp",
>>>  "poll": "5m",
>>>   "first_line_is_header":"true",
>>>  "field_separator": ";",
>>>  "escape_character": " ",
>>>  "quote_character": "'"
>>>}
>>> }
>>>
>>> and am always getting string in all types ( except date as i put the
>>> right format in my csv file *( 2014-07-24T13:58:25.000Z)*
>>>
>>> any idea ?
>>>
>>>
>>> On Wednesday, July 30, 2014 10:47:13 AM UTC+2, David Pilato wrote:

  I think you are doing something wrong.
  If you defined a mapping it should not be overwritt

Re: Elasticsearch always uses the default mapping

[David Pilato]

You applied a mapping to index _river and type newindex.

This is not what I said. You need to apply your mapping to newindex index and 
newindex type.

Basically something like:
PUT /newindex/

PUT /newindex/newindex/_mapping
{
 
      "newindex" : {
   "properties": {
   "marques": {"type": "string"},
   "ventes": {"type": "long"},
   "mois":{"type": "date"}
 }
 
 
 }
}



-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 30 juillet 2014 à 15:58:02, Amirah (beldjilal...@gmail.com) a écrit:

I don't really see the problem, i selected my newindex ( it exsists in my 
mapping with my types) 

PUT /newindex/

PUT /_river/newindex/_mapping
{
 
      "newindex" : {
   "properties": {
   "marques": {"type": "string"},
   "ventes": {"type": "long"},
   "mois":{"type": "date"}
 }
 
 
 }
}
PUT /_river/newindex/_meta
{
  "type": "csv",
  "csv_file": {
    "folder": "E:/elasticsearch-1.2.2/tmp",
    "poll": "5m",
     "first_line_is_header":"true",
    "field_separator": ";",
    "escape_character": " ",
    "quote_character": "'"
  }
,

"index" : {
       "index" : "newindex",
       "type" : "newindex",
       "bulk_size" : 100,
       "bulk_threshold" : 10
   }
 
   
}

In addition, when i create the newindex it showed {
   "_index": "_river",
   "_type": "newindex",
   "_id": "_meta",
   "_version": 1,
   "created": true
}
newindex exists in _river with correct types




On Wednesday, July 30, 2014 2:06:50 PM UTC+2, David Pilato wrote:
That's the problem.

A River creates documents in another index than _river.

If I look at the river documentation, you can set it using:

"index" : {
        "index" : "my_csv_data",
        "type" : "csv_type",
        "bulk_size" : 100,
        "bulk_threshold" : 10
    }

So basically, you need to define your aping for index my_csv_data and type 
csv_type

See https://github.com/AgileWorksOrg/elasticsearch-river-csv#full-request


-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 30 juillet 2014 à 13:54:53, Amirah (beldji...@gmail.com) a écrit:

there is a missing part ( copy paste error) /_river/
So, yes i use this 
PUT /_river/newindex/_mapping
{
     "newindex" : {
         
         "properties": {
   "MyStringValue": {"type": "string"},
   "MyLongValue": {"type": "long"},
   "MyDateValue":{"type": "date"}
 }
     }
 }
   
}

to create the mapping, my variables ( type names ) are : MyStringValue and its 
type is String, MyLongValue and its type is long, MyDateValue and its type is 
date.
It creates the mapping ( i already checked using GET _mapping ) it exists with 
my real types ! but once i use my csv file, it changes to string everywhere!



On Wednesday, July 30, 2014 12:34:25 PM UTC+2, David Pilato wrote:
This looks strange to me

PUT /newindex/_mapping
 {
      "newindex" : {
          
          "properties": {
    "MyStringValue": {"type": "string"},
    "MyLongValue": {"type": "long"},
    "MyDateValue":{"type": "date"}
  }
      }
  }
     
}

What is your type name?

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 30 juillet 2014 à 11:14:42, Amirah (beldji...@gmail.com) a écrit:

Thanks for the answer,
Am creating and defining my mapping ( and index) as following :


PUT /newindex/

PUT /newindex/_mapping
 {
      "newindex" : {
          
          "properties": {
    "MyStringValue": {"type": "string"},
    "MyLongValue": {"type": "long"},
    "MyDateValue":{"type": "date"}
  }
      }
  }
     
}
PUT /_river/newindex/_meta
{
   "type": "csv",
   "csv_file": {
     "folder": "E:/elasticsearch-1.2.2/tmp",
     "poll": "5m",
      "first_line_is_header":"true",
     "field_separator": ";",
     "escape_character": " ",
     "quote_character": "'"
   }
}

and am always getting string in all types ( except date as i put the right 
format in my csv file ( 2014-07-24T13:58:25.000Z)

any idea ?


On Wednesday, July 30, 2014 10:47:13 AM UTC+2, David Pilato wrote:
I think you are doing something wrong.
If you defined a mapping it should not be overwritten by the CSV river as far 
as I know.
Enter code here...





 
-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 30 juillet 2014 à 10:31:07, Amirah (beldji...@gmail.com) a écrit:

Hello,

Am trying to create an index using "CSV River Plugin for ElasticSearch", my csv 
file contains String, long and date values.

My problem is : 
 - ElasticSearch always consider long values as string ( with default mapping ) 
and when i try to create a specific mapping to be used, it changes long type to 
string type.
 - So, is there a way to tell ElasticSearch Don't map everything as string ?

Any suggestion please?

Thank you


--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearc...@googlegroups.com.
To view this discussion on

ORA-01882: timezone region not found

[George DRAGU]

Hello, 

Is it any possibility to specify a parameter value to java command line 
behind the JDBC River? 
I think at a -Duser.timezone=Europe/Istanbul, for exemple. 
When I try to create a JDBC River for an Oracle database (with jprante 
plugin) I catch this error. 

Thanks 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a75e8e67-01f8-4b20-a2e9-86caba59e5aa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch always uses the default mapping

[Amirah]

I don't really see the problem, i selected my newindex ( it exsists in my 
mapping with my types) 

PUT /newindex/

PUT /_river/newindex/_mapping
{
  
  "newindex" : {
   "properties": {
   "marques": {"type": "string"},
   "ventes": {"type": "long"},
   "mois":{"type": "date"}
 }
  
  
 }
} 
PUT /_river/newindex/_meta
{
  "type": "csv",
  "csv_file": {
"folder": "E:/elasticsearch-1.2.2/tmp",
"poll": "5m",
 "first_line_is_header":"true",
"field_separator": ";",
"escape_character": " ",
"quote_character": "'"
  }
,

"index" : {
   "index" : "newindex",
   "type" : "newindex",
   "bulk_size" : 100,
   "bulk_threshold" : 10
   }
  
   
}

In addition, when i create the newindex it showed {
   "_index": "_river",
   "_type": "newindex",
   "_id": "_meta",
   "_version": 1,
   "created": true
}
newindex exists in _river with correct types




On Wednesday, July 30, 2014 2:06:50 PM UTC+2, David Pilato wrote:
>
> That's the problem.
>
> A River creates documents in another index than _river.
>
> If I look at the river documentation, you can set it using:
>
> "index" : {
> "index" : "my_csv_data",
> "type" : "csv_type",
> "bulk_size" : 100,
> "bulk_threshold" : 10
> }
>
> So basically, you need to define your aping for index my_csv_data and type 
> csv_type
>
> See https://github.com/AgileWorksOrg/elasticsearch-river-csv#full-request
>
>
> -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
> @dadoonet  | @elasticsearchfr 
> 
>
>
> Le 30 juillet 2014 à 13:54:53, Amirah (beldji...@gmail.com ) 
> a écrit:
>
> there is a missing part ( copy paste error) /_river/
> So, yes i use this  
>  PUT /_river/newindex/_mapping
>  {
>   "newindex" : {
>   
>   "properties": {
> "MyStringValue": {"type": "string"},
> "MyLongValue": {"type": "long"},
> "MyDateValue":{"type": "date"}
>   }
>   }
>   }
> 
>  }
>  
> to create the mapping, my variables ( type names ) are : *MyStringValue* 
> and its type is *String*, *MyLongValue* and its type is *long*, 
> *MyDateValue* and its type is *date*. 
> It creates the mapping ( i already checked using GET _mapping ) it exists 
> with my real types ! but once i use my csv file, it changes to string 
> everywhere!
>
>
>
> On Wednesday, July 30, 2014 12:34:25 PM UTC+2, David Pilato wrote: 
>>
>>  This looks strange to me
>>
>> PUT /newindex/_mapping
>>  {
>>   "newindex" : {
>>   
>>   "properties": {
>> "MyStringValue": {"type": "string"},
>> "MyLongValue": {"type": "long"},
>> "MyDateValue":{"type": "date"}
>>   }
>>   }
>>   }
>>  
>> }
>>  
>> What is your type name?
>>  
>>  -- 
>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com* 
>>  @dadoonet  | @elasticsearchfr 
>> 
>>  
>>
>> Le 30 juillet 2014 à 11:14:42, Amirah (beldji...@gmail.com) a écrit:
>>
>>  Thanks for the answer, 
>> Am creating and defining my mapping ( and index) as following :
>>
>>   
>> PUT /newindex/
>>
>> PUT /newindex/_mapping
>>  {
>>   "newindex" : {
>>   
>>   "properties": {
>> "MyStringValue": {"type": "string"},
>> "MyLongValue": {"type": "long"},
>> "MyDateValue":{"type": "date"}
>>   }
>>   }
>>   }
>>  
>> }
>> PUT /_river/newindex/_meta
>> {
>>"type": "csv",
>>"csv_file": {
>>  "folder": "E:/elasticsearch-1.2.2/tmp",
>>  "poll": "5m",
>>   "first_line_is_header":"true",
>>  "field_separator": ";",
>>  "escape_character": " ",
>>  "quote_character": "'"
>>}
>> }
>>   
>> and am always getting string in all types ( except date as i put the 
>> right format in my csv file *( 2014-07-24T13:58:25.000Z)*
>>
>> any idea ?
>>
>>
>> On Wednesday, July 30, 2014 10:47:13 AM UTC+2, David Pilato wrote: 
>>>
>>>  I think you are doing something wrong.
>>>  If you defined a mapping it should not be overwritten by the CSV river 
>>> as far as I know.
>>>  Enter code here...
>>>  
>>>
>>>  
>>
>>
>>  
>>
>>>   -- 
>>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com* 
>>>  @dadoonet  | @elasticsearchfr 
>>> 
>>>  
>>>
>>> Le 30 juillet 2014 à 10:31:07, Amirah (beldji...@gmail.com) a écrit:
>>>
>>>  Hello, 
>>>
>>> Am trying to create an index using "CSV River Plugin for ElasticSearch" 
>>> , my csv file 
>>> contains *String*, *long* and *date* values.
>>>
>>> My problem is : 
>>>  - ElasticSearch always consider *long* values as *string* ( with 
>>> default mapping ) and when i try to create a specific mapping to be used, 
>>> it changes *long* type to *string* type.
>>>  - So, is there a way to tell ElasticSearch *Don't map everything as 
>>> string *?
>>>
>>> Any suggestion p

Re: Geo distance filter exceptions

[Madhavan Ramachandran]

Thanks for your response. 

I am using Nest dll (.Net) to index the data in ES (in windows as a 
service).  How to add the geopoint to my index columns? 

Regards
Madhavan.TR


On Tuesday, July 29, 2014 3:35:56 PM UTC-5, David Pilato wrote:
>
> No you can't out of the box. If you want to use built in geo filters you 
> need geo points.
>
> That said, you can use scripts to do it based on lat,lon fields but it 
> will be slower.
>
> My 2 cents
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
>
> Le 29 juil. 2014 à 22:30, Madhavan Ramachandran  > a écrit :
>
> Hi Team,
>
> I am trying to find a solution for the below
>
>1. Geo boundary based search.. My index have property for lat and lon 
>as double.. not as a geopoint.. Here is my mapping for my index..
>
> How do i use the lon and lat from the below mapping for geo distance 
> filter/geo distance range filter ?
>
> Name Type Format Store?
> data string
>
> getpath string
>
> id double
>
> Region string
>
> Submarket string
>
> addr1 string
>
> addr2 string
>
> city string
>
> citymarket string
>
> country string
>
> countryid long
>
> cultureid long
>
> data string
>
> details string
>
> fax string
>
> id string
>
> language string
>
> lat double
>
> lon double
>
> When I search for the documents.. i got the below exception..
>
> Query : 
> {
> "filter": {
> "geo_distance" : {
> "distance" : "300km",
> "location" : {
> "lat" : 45,
> "lon" : -122
> }
> }
>
> }
> }
>
> Exception: 
> "error": "SearchPhaseExecutionException[Failed to execute phase [query], 
> all shards failed; shardFailures 
> {[o3f66HetT3OSpVw895w0nA][offlocations][4]: 
> SearchParseException[[offlocations][4]: from[-1],size[-1]: Parse Failure 
> [Failed to parse source [{\n\"filter\": {\n \"geo_distance\" : {\n 
> \"distance\" : \"300km\",\n \"location\" : {\n \"lat\" : 45,\n \"lon\" : 
> -122\n }\n } \n }\n}]]]; nested:
>
> I tried with removing the location, which i dont have in my mapping.. 
>
> {"filter": { "geo_distance" : { "distance" : "300km",
> "lat" 
> : 45, "lon" : -122   }  }}
>
> I got the exception as lon is not a geo_point field..
>
> ElasticsearchIllegalArgumentException[the character '-' is not a valid 
> geohash character]; }]"
>
> If i remove the "-" infront of lon.. then the exception says : 
>
> QueryParsingException[[offlocations] field [lon] is not a geo_point field];
>
>
>  -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/85658110-e7ae-42c5-8ca9-13f10c7f3820%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b0d6e420-701b-4ace-8763-9a4d2c172cdc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: bulk indexing - optimal refresh_interval

[shikhar]

Thanks for the explanation! I'll switch over for the next time I need to
reindex.


On Tue, Jul 29, 2014 at 6:35 PM, Michael McCandless 
wrote:

> Disabling refresh (-1) is a good choice if you are fully maximizing your
> cluster's CPU/IO resources (using enough bulk client threads or async
> requests).  In that case it should give faster indexing throughput than 30s
> refresh.
>
> But if you are not saturating the cluster's resources, then a refresh
> interval of 30s may in fact get you faster indexing throughput because
> refreshes are done with a background thread in ES, so you effectively get
> one more thread working for you than if you disable refresh which causes
> the bulk indexing threads to do the flushing.
>
> Try both and see and then report back!
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Jul 29, 2014 at 3:11 AM, joergpra...@gmail.com <
> joergpra...@gmail.com> wrote:
>
>> There is no more a massive hit when opening an index for read once than
>> at every 30 seconds.
>>
>> The only explanation I can think of is that users perform searches while
>> indexing and somehow want up-to-date results while they search along.
>>
>> This is not the case when I do bulk indexing, search is disabled
>> completely. So I still recommend disabling refresh_interval in the case
>> when there is no search activity while bulk indexing. And before search is
>> opened again, the index is flushed, optimized, and extended to replica
>> levels as well.
>>
>> Jörg
>>
>>
>> On Tue, Jul 29, 2014 at 8:23 AM, Mark Walkom 
>> wrote:
>>
>>> I'd say because if you are inserting a lot of data, you will have a
>>> massive hit at the end when you need to index, as opposed to smaller ones
>>> along the way.
>>>
>>> Regards,
>>> Mark Walkom
>>>
>>> Infrastructure Engineer
>>> Campaign Monitor
>>> email: ma...@campaignmonitor.com
>>> web: www.campaignmonitor.com
>>>
>>>
>>> On 29 July 2014 16:20, shikhar  wrote:
>>>
 The 1.3.0 release notes state:


>- Increase the refresh_interval
>
> 
>  if
>you are doing heavy bulk indexing, or you are happy with your search
>results being refreshed less frequently than once every second. *It
>is better to use a refresh interval like “30s” rather than disabling it
>completely with “-1″.*
>
> I'd love to get an explanation on why 30s is better than -1, which is
 the setting we are using right now when reindexing.

 --
 You received this message because you are subscribed to the Google
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/CAHWG4DOaGphdmA%3DbQqV-0ic8HXxLM3ZmBzbW9YFDtZ_zWG8BHA%40mail.gmail.com
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/CAEM624Y6qPWwG5rtqh7V2WaqTTqcWJoRP6%2Bct5CYhGR2xUXPgQ%40mail.gmail.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFEKO7pkn97TY9CbNn3JNb%2B18qVyzaFiXw_rDU8fDg-YQ%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAD7smRfRmRbSYxixHxZBfQ-VKEOwPACc4xFC1tFmNhr7HaRxsQ%40mail.gmail.com
> 

Kibana Time Picker How to Set Default Custom

[Didjit]

Hi folks, 

I would like to have Kibana default to business hours for my location. 
Let's say 08:00 - 17:00. I know I can set up this in custom for a specific 
day and save. How can I set the default hours regardless of the day picked? 
So, every day a user accesses my dashboard, it would default to Today 08:00 
- 17:00

Thank you!

Chris

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2bf187a2-fb19-4b2d-ac7e-feb277c7487c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)

[Andrew Davidoff]

On Tuesday, July 29, 2014 3:27:13 PM UTC-4, Ivan Brusic wrote:
>
> Have you changed your gateway settings? 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html#recover-after
>
> It still remains a bit of black magic to me. Sometimes it works, sometimes 
> it does not.
>

Ivan,

I have read over that documentation several times and I don't understand 
how it would help me. By which I mean I honestly don't understand how it 
would help me - not that I am unwilling to try it. Those settings read like 
they control when recovery would begin, but my problem isn't that recovery 
is starting when I don't want it to, it's that when I start it (by setting 
shard allocation about to "all") shards that I'd think would just stick 
with the node they were previously on, get assigned to other nodes, then 
ultimately get rebalanced back to the original node.

At this point I am finding that for quick restarts, just doing them with no 
additional prep work allows me to recover in ~30m, vs ~24h. So for now I'm 
just going to do that. Whatever I am doing wrong here just isn't at all 
clear to me.

Thanks for your advice. If I have misunderstood the settings you pointed me 
at and you think you can help me understand, I'd be grateful for more 
information.

Andy



> Cheers,
>
> Ivan
>
>
> On Mon, Jul 28, 2014 at 1:52 PM, Andrew Davidoff  > wrote:
>
>> I guess I'll ask about this once more for now. This happened again today. 
>> I set allocation to new_primaries, restarted a node, set allocation back to 
>> all, and the cluster is assigning across all nodes the shards that were on 
>> the restarted node, and when it's done, which will probably take a day, 
>> it'll likely rebalance by moving them back to the original node. I have to 
>> assume I'm doing something wrong here. Am I?
>>
>> Thanks for any advice.
>> Andy
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b8cf076f-2494-4b0e-b6c3-5a21d1c9f9a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

log index creation API requests

[bitsofinfo . g]

Hi - any tips for how I should configure the logging.yml file to give me 
more verbose output, including source ip address if possible, to give more 
info when an index is created?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2b423a9b-1e60-4e53-87b0-4681be643683%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch always uses the default mapping

[David Pilato]

That's the problem.

A River creates documents in another index than _river.

If I look at the river documentation, you can set it using:

"index" : {
        "index" : "my_csv_data",
        "type" : "csv_type",
        "bulk_size" : 100,
        "bulk_threshold" : 10
    }

So basically, you need to define your aping for index my_csv_data and type 
csv_type

See https://github.com/AgileWorksOrg/elasticsearch-river-csv#full-request


-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 30 juillet 2014 à 13:54:53, Amirah (beldjilal...@gmail.com) a écrit:

there is a missing part ( copy paste error) /_river/
So, yes i use this 
PUT /_river/newindex/_mapping
{
     "newindex" : {
         
         "properties": {
   "MyStringValue": {"type": "string"},
   "MyLongValue": {"type": "long"},
   "MyDateValue":{"type": "date"}
 }
     }
 }
   
}

to create the mapping, my variables ( type names ) are : MyStringValue and its 
type is String, MyLongValue and its type is long, MyDateValue and its type is 
date.
It creates the mapping ( i already checked using GET _mapping ) it exists with 
my real types ! but once i use my csv file, it changes to string everywhere!



On Wednesday, July 30, 2014 12:34:25 PM UTC+2, David Pilato wrote:
This looks strange to me

PUT /newindex/_mapping
 {
      "newindex" : {
          
          "properties": {
    "MyStringValue": {"type": "string"},
    "MyLongValue": {"type": "long"},
    "MyDateValue":{"type": "date"}
  }
      }
  }
     
}

What is your type name?

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 30 juillet 2014 à 11:14:42, Amirah (beldji...@gmail.com) a écrit:

Thanks for the answer,
Am creating and defining my mapping ( and index) as following :


PUT /newindex/

PUT /newindex/_mapping
 {
      "newindex" : {
          
          "properties": {
    "MyStringValue": {"type": "string"},
    "MyLongValue": {"type": "long"},
    "MyDateValue":{"type": "date"}
  }
      }
  }
     
}
PUT /_river/newindex/_meta
{
   "type": "csv",
   "csv_file": {
     "folder": "E:/elasticsearch-1.2.2/tmp",
     "poll": "5m",
      "first_line_is_header":"true",
     "field_separator": ";",
     "escape_character": " ",
     "quote_character": "'"
   }
}

and am always getting string in all types ( except date as i put the right 
format in my csv file ( 2014-07-24T13:58:25.000Z)

any idea ?


On Wednesday, July 30, 2014 10:47:13 AM UTC+2, David Pilato wrote:
I think you are doing something wrong.
If you defined a mapping it should not be overwritten by the CSV river as far 
as I know.
Enter code here...





 
-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 30 juillet 2014 à 10:31:07, Amirah (beldji...@gmail.com) a écrit:

Hello,

Am trying to create an index using "CSV River Plugin for ElasticSearch", my csv 
file contains String, long and date values.

My problem is : 
 - ElasticSearch always consider long values as string ( with default mapping ) 
and when i try to create a specific mapping to be used, it changes long type to 
string type.
 - So, is there a way to tell ElasticSearch Don't map everything as string ?

Any suggestion please?

Thank you


--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearc...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/664de1fc-1e52-4846-9660-5f5808d7b6af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearc...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/69675304-f587-4ad2-b24c-550be4446039%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e8a17a8f-a98e-41fa-a774-36cdc0f4b684%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.53d8dfc5.51ead36b.f0d0%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch always uses the default mapping

[Amirah]

there is a missing part ( copy paste error) /_river/
So, yes i use this 
PUT /_river/newindex/_mapping
{
 "newindex" : {
 
 "properties": {
   "MyStringValue": {"type": "string"},
   "MyLongValue": {"type": "long"},
   "MyDateValue":{"type": "date"}
 }
 }
 }

}

to create the mapping, my variables ( type names ) are : *MyStringValue* 
and its type is *String*, *MyLongValue* and its type is *long*, 
*MyDateValue* and its type is *date*.
It creates the mapping ( i already checked using GET _mapping ) it exists 
with my real types ! but once i use my csv file, it changes to string 
everywhere!



On Wednesday, July 30, 2014 12:34:25 PM UTC+2, David Pilato wrote:
>
> This looks strange to me
>
> PUT /newindex/_mapping
>  {
>   "newindex" : {
>   
>   "properties": {
> "MyStringValue": {"type": "string"},
> "MyLongValue": {"type": "long"},
> "MyDateValue":{"type": "date"}
>   }
>   }
>   }
>  
> }
>
> What is your type name?
>
> -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
> @dadoonet  | @elasticsearchfr 
> 
>
>
> Le 30 juillet 2014 à 11:14:42, Amirah (beldji...@gmail.com ) 
> a écrit:
>
> Thanks for the answer, 
> Am creating and defining my mapping ( and index) as following :
>
>   
> PUT /newindex/
>
> PUT /newindex/_mapping
>  {
>   "newindex" : {
>   
>   "properties": {
> "MyStringValue": {"type": "string"},
> "MyLongValue": {"type": "long"},
> "MyDateValue":{"type": "date"}
>   }
>   }
>   }
>  
> }
> PUT /_river/newindex/_meta
> {
>"type": "csv",
>"csv_file": {
>  "folder": "E:/elasticsearch-1.2.2/tmp",
>  "poll": "5m",
>   "first_line_is_header":"true",
>  "field_separator": ";",
>  "escape_character": " ",
>  "quote_character": "'"
>}
> }
>   
> and am always getting string in all types ( except date as i put the right 
> format in my csv file *( 2014-07-24T13:58:25.000Z)*
>
> any idea ?
>
>
> On Wednesday, July 30, 2014 10:47:13 AM UTC+2, David Pilato wrote: 
>>
>>  I think you are doing something wrong.
>>  If you defined a mapping it should not be overwritten by the CSV river 
>> as far as I know.
>>  Enter code here...
>>  
>>
>>  
>
>
>  
>
>>   -- 
>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com* 
>>  @dadoonet  | @elasticsearchfr 
>> 
>>  
>>
>> Le 30 juillet 2014 à 10:31:07, Amirah (beldji...@gmail.com) a écrit:
>>
>>  Hello, 
>>
>> Am trying to create an index using "CSV River Plugin for ElasticSearch" 
>> , my csv file 
>> contains *String*, *long* and *date* values.
>>
>> My problem is : 
>>  - ElasticSearch always consider *long* values as *string* ( with 
>> default mapping ) and when i try to create a specific mapping to be used, 
>> it changes *long* type to *string* type.
>>  - So, is there a way to tell ElasticSearch *Don't map everything as 
>> string *?
>>
>> Any suggestion please?
>>
>> Thank you
>>
>>
>>  --
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/664de1fc-1e52-4846-9660-5f5808d7b6af%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>  
>>   --
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/69675304-f587-4ad2-b24c-550be4446039%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e8a17a8f-a98e-41fa-a774-36cdc0f4b684%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Cosine Similarity ElasticSearch

[peter SP]

I am also interested in this question.

I have found a fairly old code snippet [1] to calculate the cosine 
similarity in lucene, but I was wondering if elasticsearch provided an 
easier API to access this information.

[1] 
http://sujitpal.blogspot.ch/2011/10/computing-document-similarity-using.html

Hi, 
>
> I'm using elasticsearch to index documents and then, with an other 
> document, 
> I score similarity using the "more_like_this" query. 
> Just two questions: 
>
> Does the "more_like_this" query use cosine similarity to score documents 
> (I've read the documentation, but I'm still not sure)? 
> There is a way to get the scores between 0 and 1? 
>
> Thanks! 
>
>
>
> -- 
> View this message in context: 
> http://elasticsearch-users.115913.n3.nabble.com/Cosine-Similarity-ElasticSearch-tp4060620.html
>  
> Sent from the ElasticSearch Users mailing list archive at Nabble.com. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6a2e446d-ec13-4db6-8cff-651ab5108235%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: slow filter execution

[Clinton Gormley]

Don't use the `and` filter - use the `bool` filter instead.  They have
different execution modes and the `bool` filter works best with bitset
filters (but also knows how to handle non-bitset filters like geo etc).

Just remove the `and`, `or` and `not` filters from your DSL vocabulary.

Also, not sure why you are ANDing with a match_all filter - that doesn't
make much sense.

Depending on which version of ES you're using, you may be encountering a
bug in the filtered query which ended up always running the query first,
instead of the filter. This was fixed in v1.2.0
https://github.com/elasticsearch/elasticsearch/issues/6247 .  If you are on
an earlier version you can force filter-first execution manually by
specifying a "strategy" of "random_access_100".  See
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-filtered-query.html#_filter_strategy

In summary, (and taking your less granular datetime clause into account)
your query would be better written as:

GET /_search
{
  "query": {
"filtered": {
  "strategy": "random_access_100",   pre 1.2 only
  "filter": {
"bool": {
  "must": [
{
  "terms": {
"source_id": [ "s1", "s2", "s3" ]
  }
},
{
  "range": {
"published": {
  "gte": "now-1d/d"   coarse grained, cached
}
  }
},
{
  "range": {
"published": {
  "gte": "now-30m"  fine grained, not cached, could
use fielddata too
},
"_cache": false
  }
}
  ]
}
  }
}
  }
}





On 30 July 2014 10:55, David Pilato  wrote:

> May be a stupid question: why did you put that filter inside a query and
> not within the same filter you have at the end?
>
>
> For my test case it's the same every time. In the "real" query it will
>> change every time, but I planned to not cache this filter and have a less
>> granular date filter in the bool filter that would be cached. However while
>> debugging I noticed slowness with the date range filters even while testing
>> with the same value repeatedly.
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/af76ca41-9045-4a4f-b82c-b9c86d964ace%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPt3XKRQ6tyciPDVKVnCz0nzgq9B89y6irh3N1Ergf-oCW2Z%2Bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [java api] Trying to use groovy instead of mvel

[Laurent T.]

Ok well, anyway i think you may want to update the docs about this cause i 
think i won't be the only one facing this :)
Thanks again to both of you.

On Wednesday, July 30, 2014 12:30:09 PM UTC+2, David Pilato wrote:
>
> Ha! Right!  Thanks Jörg!
>
> I forgot that I run the same "issue" recently. I should add more memory to 
> my brain cluster :)
>
> -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
> @dadoonet  | @elasticsearchfr 
> 
>
>
> Le 30 juillet 2014 à 12:08:58, joerg...@gmail.com  (
> joerg...@gmail.com ) a écrit:
>
> The ES team decided to postpone groovy as default to Elasticsearch 1.4 
> version. 
>
> In 1.3, mvel is still the default, so authors have some time to rewrite 
> their scripts if they prefer to. So I think it is ok to not include groovy 
> jar by default, and make this optional to those who want to switch over to 
> groovy.
>
> Jörg
>  
>
> On Wed, Jul 30, 2014 at 11:53 AM, Laurent T.  > wrote:
>
>> Nice catch Jörg, that indeed did the trick. 
>>
>> @David Shouldn't groovy be bundled in the ES jar if it's the new default ?
>> Will it be provided by ES when i run on a live cluster ?
>>
>> Thanks!
>>  
>> On Wednesday, July 30, 2014 11:41:23 AM UTC+2, Jörg Prante wrote:
>>
>>>  You should try to add groovy jar to your classpath. It is not in the 
>>> dependencies in Maven's pom.xml. 
>>>
>>> Example:
>>>
>>>  
>>> org.codehaus.groovy
>>> groovy-all
>>> 2.3.5
>>> jar
>>> 
>>>  
>>> Jörg
>>>
>>>  
>>>
>>> On Wed, Jul 30, 2014 at 10:46 AM, Laurent T.  wrote:
>>>
 Hi David, 

 I tried, as you suggested, to activate dynamic scripting and to force 
 groovy as a default_lang but the results stay unchanged.
  
>>> --
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/0c285769-7872-41cd-9efe-6c97ed8d3df4%40googlegroups.com
>>  
>> .
>>  
>>
>>
>> For more options, visit https://groups.google.com/d/optout.
>>  
>  
> --
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG9mwEK_65v084icNeLvtk2S6eEwk4cPu2sZNXCPDgZ0Q%40mail.gmail.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b0339bd4-5009-418f-8b97-eca21eda0089%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch always uses the default mapping

[David Pilato]

This looks strange to me

PUT /newindex/_mapping
 {
      "newindex" : {
          
          "properties": {
    "MyStringValue": {"type": "string"},
    "MyLongValue": {"type": "long"},
    "MyDateValue":{"type": "date"}
  }
      }
  }
     
}

What is your type name?

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 30 juillet 2014 à 11:14:42, Amirah (beldjilal...@gmail.com) a écrit:

Thanks for the answer,
Am creating and defining my mapping ( and index) as following :


PUT /newindex/

PUT /newindex/_mapping
 {
      "newindex" : {
          
          "properties": {
    "MyStringValue": {"type": "string"},
    "MyLongValue": {"type": "long"},
    "MyDateValue":{"type": "date"}
  }
      }
  }
     
}
PUT /_river/newindex/_meta
{
   "type": "csv",
   "csv_file": {
     "folder": "E:/elasticsearch-1.2.2/tmp",
     "poll": "5m",
      "first_line_is_header":"true",
     "field_separator": ";",
     "escape_character": " ",
     "quote_character": "'"
   }
}

and am always getting string in all types ( except date as i put the right 
format in my csv file ( 2014-07-24T13:58:25.000Z)

any idea ?


On Wednesday, July 30, 2014 10:47:13 AM UTC+2, David Pilato wrote:
I think you are doing something wrong.
If you defined a mapping it should not be overwritten by the CSV river as far 
as I know.
Enter code here...





 
-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 30 juillet 2014 à 10:31:07, Amirah (beldji...@gmail.com) a écrit:

Hello,

Am trying to create an index using "CSV River Plugin for ElasticSearch", my csv 
file contains String, long and date values.

My problem is : 
 - ElasticSearch always consider long values as string ( with default mapping ) 
and when i try to create a specific mapping to be used, it changes long type to 
string type.
 - So, is there a way to tell ElasticSearch Don't map everything as string ?

Any suggestion please?

Thank you


--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearc...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/664de1fc-1e52-4846-9660-5f5808d7b6af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/69675304-f587-4ad2-b24c-550be4446039%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.53d8ca28.440badfc.f0d0%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: Integration testing a native script

[Thomas]

I have noticed that you mention native java script so you have implemented 
it as a plugin?
if so try the following in your settings:
   final Settings settings
= settingsBuilder()
 ...

.put("plugin.types", YourPlugin.class.getName())

Thomas


On Wednesday, 30 July 2014 12:31:06 UTC+3, Nick T wrote:
>
> Is there a way to have a native java script accessible in integration 
> tests? In my integration tests I am creating a test node in the /tmp 
> folder. 
>
> I've tried copying the script to /tmp/plugins/scripts but that was quite 
> hopeful and unfortunately does not work.
>
> Desperate for help.
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7baa9562-58be-4f16-9392-9cf07e4e989d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Integration testing a native script

[Thomas]

Hi,

I have tried the same approach and it worked for me, meaning to copy the 
script I want to perform an integration test and run my IT.

I do the following steps

1) Setup the required paths for elasticsearch

 final Settings settings
= settingsBuilder()
.put("http.enabled", "true")
.put("path.conf", confDir)
.put("path.data", dataDir)
.put("path.work", workDir)
.put("path.logs", logsDir)


2) copy your scripts to the appropriate location
3) fire up a local node
 node = 
nodeBuilder().local(true).settings(settings).clusterName(nodeName).node();
 node.start();

Maybe you first start the node and then add the script, this might not work 
because i think es does a per minute scan for new scripts and the IT test 
does not allow this to happen, hence you should first copy your script and 
then start the node

Hope it helps

Thomas

On Wednesday, 30 July 2014 12:31:06 UTC+3, Nick T wrote:
>
> Is there a way to have a native java script accessible in integration 
> tests? In my integration tests I am creating a test node in the /tmp 
> folder. 
>
> I've tried copying the script to /tmp/plugins/scripts but that was quite 
> hopeful and unfortunately does not work.
>
> Desperate for help.
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4b257e79-e424-4573-8f12-81e0a95b27b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [java api] Trying to use groovy instead of mvel

[David Pilato]

Ha! Right!  Thanks Jörg!

I forgot that I run the same "issue" recently. I should add more memory to my 
brain cluster :)

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 30 juillet 2014 à 12:08:58, joergpra...@gmail.com (joergpra...@gmail.com) a 
écrit:

The ES team decided to postpone groovy as default to Elasticsearch 1.4 version.

In 1.3, mvel is still the default, so authors have some time to rewrite their 
scripts if they prefer to. So I think it is ok to not include groovy jar by 
default, and make this optional to those who want to switch over to groovy.

Jörg


On Wed, Jul 30, 2014 at 11:53 AM, Laurent T.  wrote:
Nice catch Jörg, that indeed did the trick.

@David Shouldn't groovy be bundled in the ES jar if it's the new default ?
Will it be provided by ES when i run on a live cluster ?

Thanks!

On Wednesday, July 30, 2014 11:41:23 AM UTC+2, Jörg Prante wrote:
You should try to add groovy jar to your classpath. It is not in the 
dependencies in Maven's pom.xml.

Example:

            
                org.codehaus.groovy
                groovy-all
                2.3.5
                jar
            

Jörg



On Wed, Jul 30, 2014 at 10:46 AM, Laurent T.  wrote:
Hi David,

I tried, as you suggested, to activate dynamic scripting and to force groovy as 
a default_lang but the results stay unchanged.
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0c285769-7872-41cd-9efe-6c97ed8d3df4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG9mwEK_65v084icNeLvtk2S6eEwk4cPu2sZNXCPDgZ0Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.53d8c924.4516dde9.f0d0%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: Cloud-aws version for 1.3.1 of elasticsearch

[David Pilato]

I'm currently working on AWS plugin.

I think you can use cloud aws plugin 2.2.0 with es 1.3.1.

I'm going also to fix doc nightmare about the plugin versions so it will be 
more obvious in the future.

My plan is to release it before the end of this week.

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 30 juillet 2014 à 12:23:05, Thomas (thomas.bo...@gmail.com) a écrit:

Hi,

I wanted to ask whether the version of cloud-aws plugin is 2.1.1 for 
elasticsearch 1.3.1, by looking at the github page:
https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.3

How come the plugin version for 1.3.1 of elasticserach goes backwards? For 
elasticsearch 1.2.x the version of cloud-aws is 2.2.0.
Is this correct?

Thank you very much
Thomas
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ccb42a1f-a0f0-40ed-81d6-96b0e1b279c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.53d8c86f.737b8ddc.f0d0%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Cloud-aws version for 1.3.1 of elasticsearch

[Thomas]

Hi,

I wanted to ask whether the version of cloud-aws plugin is 2.1.1 for 
elasticsearch 1.3.1, by looking at the github page:
https://github.com/elasticsearch/elasticsearch-cloud-aws/tree/es-1.3

How come the plugin version for 1.3.1 of elasticserach goes backwards? For 
elasticsearch 1.2.x the version of cloud-aws is 2.2.0.
Is this correct?

Thank you very much
Thomas

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ccb42a1f-a0f0-40ed-81d6-96b0e1b279c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [java api] Trying to use groovy instead of mvel

[joergpra...@gmail.com]

The ES team decided to postpone groovy as default to Elasticsearch 1.4
version.

In 1.3, mvel is still the default, so authors have some time to rewrite
their scripts if they prefer to. So I think it is ok to not include groovy
jar by default, and make this optional to those who want to switch over to
groovy.

Jörg


On Wed, Jul 30, 2014 at 11:53 AM, Laurent T.  wrote:

> Nice catch Jörg, that indeed did the trick.
>
> @David Shouldn't groovy be bundled in the ES jar if it's the new default ?
> Will it be provided by ES when i run on a live cluster ?
>
> Thanks!
>
> On Wednesday, July 30, 2014 11:41:23 AM UTC+2, Jörg Prante wrote:
>
>> You should try to add groovy jar to your classpath. It is not in the
>> dependencies in Maven's pom.xml.
>>
>> Example:
>>
>> 
>> org.codehaus.groovy
>> groovy-all
>> 2.3.5
>> jar
>> 
>>
>> Jörg
>>
>>
>>
>> On Wed, Jul 30, 2014 at 10:46 AM, Laurent T.  wrote:
>>
>>> Hi David,
>>>
>>> I tried, as you suggested, to activate dynamic scripting and to force
>>> groovy as a default_lang but the results stay unchanged.
>>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/0c285769-7872-41cd-9efe-6c97ed8d3df4%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG9mwEK_65v084icNeLvtk2S6eEwk4cPu2sZNXCPDgZ0Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [java api] Trying to use groovy instead of mvel

[Laurent T.]

Nice catch Jörg, that indeed did the trick.

@David Shouldn't groovy be bundled in the ES jar if it's the new default ?
Will it be provided by ES when i run on a live cluster ?

Thanks!

On Wednesday, July 30, 2014 11:41:23 AM UTC+2, Jörg Prante wrote:
>
> You should try to add groovy jar to your classpath. It is not in the 
> dependencies in Maven's pom.xml.
>
> Example:
>
> 
> org.codehaus.groovy
> groovy-all
> 2.3.5
> jar
> 
>
> Jörg
>
>
>
> On Wed, Jul 30, 2014 at 10:46 AM, Laurent T.  > wrote:
>
>> Hi David,
>>
>> I tried, as you suggested, to activate dynamic scripting and to force 
>> groovy as a default_lang but the results stay unchanged.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0c285769-7872-41cd-9efe-6c97ed8d3df4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: sum-aggregation script doesn't allow negative values?

[Valentin]

Hi Colin,

I try increasing it up to 40 but nothing changes. I would post the stack 
trace but I don't know how to find them.

Thanks
Valentin

On Wednesday, July 30, 2014 10:24:09 AM UTC+2, Colin Goodheart-Smithe wrote:
>
> Also, your shard_size parameter should always be greater than the size 
> parameter.  So if you are asking for size of 10 then I would try setting 
> shard_size to 20 or 30.
>
> On Wednesday, 30 July 2014 09:22:16 UTC+1, Colin Goodheart-Smithe wrote:
>>
>> Would you be able to re-run your query and post the stack trace from the 
>> Elasticsearch server logs.  This might help to work out whats going on.
>>
>> Thanks
>>
>> Colin
>>
>> On Tuesday, 29 July 2014 12:29:00 UTC+1, Valentin wrote:
>>>
>>> Ok. I think I found the problem. As soon as I try to sort on the script 
>>> value it ceases to work
>>>
>>> works, but unsorted
>>> {
>>>   "size": 0,
>>>   "aggs": {
>>> "winners": {
>>>   "terms": {
>>> "field": "tit",
>>> "size": 10,
>>> "shard_size": 4
>>>   },
>>>   "aggs": {
>>> "articles_over_time": {
>>>   "date_histogram": {
>>> "field": "datetime",
>>> "interval": "1d"
>>>   }
>>> },
>>> "diff": {
>>>   "sum": {
>>> "script": "(doc['datetime'].value < 140641200) ? -1 : 1",
>>> "lang": "groovy"
>>>   }
>>> }
>>>   }
>>> }
>>>   }
>>> }
>>>
>>> does not work:
>>> {
>>>   "size": 0,
>>>   "aggs": {
>>> "winners": {
>>>   "terms": {
>>> "field": "tit",
>>> "size": 10,
>>> "order": {
>>>   "diff": "desc"
>>> },
>>> "shard_size": 4
>>>   },
>>>   "aggs": {
>>> "articles_over_time": {
>>>   "date_histogram": {
>>> "field": "datetime",
>>> "interval": "1d"
>>>   }
>>> },
>>> "diff": {
>>>   "sum": {
>>> "script": "(doc['datetime'].value < 140641200) ? -1 : 1",
>>> "lang": "groovy"
>>>   }
>>> }
>>>   }
>>> }
>>>   }
>>> }
>>>
>>>
>>>
>>>
>>> On Tuesday, July 29, 2014 12:40:15 PM UTC+2, Valentin wrote:

 Hi Colin,

 I could figure out the shard_size problem thanks to your help.

 For the 'datetime' error: I checked and it exists in all the indices. 
 It has the correct mappings and the therefor probably could not have wrong 
 values I guess. And using the elasticsearch-head plugin I dont get the 
 error but a wrong result which really seems strange.

 Thanks
 Valentin

 On Tuesday, July 29, 2014 11:54:08 AM UTC+2, Colin Goodheart-Smithe 
 wrote:
>
> Firstly, I think the reason you are only getting results from one 
> index when you are asking for a size of 1 in your terms aggregation is 
> because you are asking for the top 1 bucket from each shard on each 
> index. 
>  This will then be merged together and only the top bucket will be kept. 
>  If the top bucket is not the same on all indexes then you will not get 
> results from all indices.  Setting the shard_size parameter to something 
> like 10 can help with this (see 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_document_counts_are_approximate
>  
> for more information on this)
>
> Second, I wonder if the reason you are getting the error from your 
> script is that you don't have a 'datetime' value for all of your 
> documents 
> in some of your indices?
>
> Regards,
>
> Colin
>
> On Monday, 28 July 2014 16:04:55 UTC+1, Valentin wrote:
>>
>> Hi Colin,
>>
>> now it gets really strange. First my alias
>> curl 'http://localhost:9200/_alias?pretty'
>> { 
>>   "live-2014-07-27" : { 
>>
>> "aliases" : { 
>>
>>   "aggtest" : { } 
>>
>> } 
>>
>>   }, 
>>
>>   "live-2014-07-26" : { 
>>
>> "aliases" : { 
>>
>>   "aggtest" : { } 
>>
>> } 
>>
>>   } 
>>
>> }
>>
>>
>> I tried two different queries:
>> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true' 
>> -d '{
>>   "size": 0,
>>   "aggs": {
>> "winners": {
>>   "terms": {
>> "field": "tit",
>> "order": {
>>   "diff": "desc"
>> },
>> "size": 1
>>   },
>>   "aggs": {
>> "articles_over_time": {
>>   "date_histogram": {
>> "field": "datetime",
>> "interval": "1d"
>>   }
>> },
>> "diff": {
>>   "sum": {
>> "script": "(doc['datetime'].value < 140641200) ? -1 : 
>> 1",
>> "lang": "groovy"
>>   }
>> }
>

Re: Exception when using ES 1.3.1 Caused by: java.lang.IncompatibleClassChangeError: Implementing class

[joergpra...@gmail.com]

This is a dependency problem. Check your classpath if you have clean
dependencies to ES 1.3.1 code only.

Jörg


On Wed, Jul 30, 2014 at 10:41 AM, gregorymaertens via elasticsearch <
elasticsearch@googlegroups.com> wrote:

> Hello,
>
> I have a project using Play framework version 1.2.7 which used ES 1.1.1.
> I wanted to update it to the latest and greatest (1.3.1), but encountered
> the following exception when running the unit tests within the play
> framework:
>
> An unexpected error occured caused by exception
> IncompatibleClassChangeError: Implementing class
>
> (see full trace below)
>
> I'm using Java7 (1.7.0_51).
>
> ES 1.2.3 has no issue, it starts with 1.3.0 to exhibit this problem.
>
> I have looked at the issue and it seems it's caused by not using the right
> class version, but I wasn't able to pin point to the exact one.
>
> Has anyone encounter this or know which particular class may introduce
> this issue?
>
>
> Thanks,
>
> Greg
>
>
> An unexpected error occured caused by exception
> IncompatibleClassChangeError: Implementing class
>
>
> play.exceptions.UnexpectedException: Unexpected Error
>
> at play.Invoker$Invocation.onException(Invoker.java:244)
>
> at play.Invoker$Invocation.run(Invoker.java:286)
>
> at Invocation.HTTP Request(Play!)
>
> Caused by: java.lang.IncompatibleClassChangeError: Implementing class
>
> at java.lang.ClassLoader.defineClass1(Native Method)
>
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
> at java.lang.ClassLoader.defineClass1(Native Method)
>
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
> at
> play.classloading.enhancers.ContinuationEnhancer.enhanceThisClass(ContinuationEnhancer.java:74)
>
> at play.CorePlugin.enhance(CorePlugin.java:297)
>
> at play.plugins.PluginCollection.enhance(PluginCollection.java:551)
>
> at
> play.classloading.ApplicationClasses$ApplicationClass.enhance(ApplicationClasses.java:235)
>
> at
> play.classloading.ApplicationClassloader.loadApplicationClass(ApplicationClassloader.java:167)
>
> at
> play.classloading.ApplicationClassloader.getAllClasses(ApplicationClassloader.java:431)
>
> at play.Play.start(Play.java:523)
>
> at play.Play.detectChanges(Play.java:649)
>
> at play.Invoker$Invocation.init(Invoker.java:198)
>
> ... 1 more
>
> 10:21:21,858 ERROR ~ Error during the 500 response generation
>
> java.lang.IncompatibleClassChangeError: Implementing class
>
> at java.lang.ClassLoader.defineClass1(Native Method)
>
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
> at java.lang.ClassLoader.defineClass1(Native Method)
>
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.d

Re: [java api] Trying to use groovy instead of mvel

[joergpra...@gmail.com]

You should try to add groovy jar to your classpath. It is not in the
dependencies in Maven's pom.xml.

Example:


org.codehaus.groovy
groovy-all
2.3.5
jar


Jörg



On Wed, Jul 30, 2014 at 10:46 AM, Laurent T.  wrote:

> Hi David,
>
> I tried, as you suggested, to activate dynamic scripting and to force
> groovy as a default_lang but the results stay unchanged.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTkAHvnWZLc1bzjU4f54xtT4ptJYxz4h%3DD3KAqBVEWmA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Integration testing a native script

[David Pilato]

I might be wrong but I think that scripts should be located in config/scripts, 
right? 

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 30 juillet 2014 à 11:31:10, Nick T (nttod...@gmail.com) a écrit:

Is there a way to have a native java script accessible in integration tests? In 
my integration tests I am creating a test node in the /tmp folder. 

I've tried copying the script to /tmp/plugins/scripts but that was quite 
hopeful and unfortunately does not work.

Desperate for help.

Thanks
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b2ae43cc-22cd-4d8a-a79d-ddbf2c572750%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.53d8bce9.8138641.f0d0%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Integration testing a native script

[Nick T]

Is there a way to have a native java script accessible in integration 
tests? In my integration tests I am creating a test node in the /tmp 
folder. 

I've tried copying the script to /tmp/plugins/scripts but that was quite 
hopeful and unfortunately does not work.

Desperate for help.

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b2ae43cc-22cd-4d8a-a79d-ddbf2c572750%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch always uses the default mapping

[Amirah]

Thanks for the answer,
Am creating and defining my mapping ( and index) as following :


PUT /newindex/

PUT /newindex/_mapping
 {
  "newindex" : {
  
  "properties": {
"MyStringValue": {"type": "string"},
"MyLongValue": {"type": "long"},
"MyDateValue":{"type": "date"}
  }
  }
  }
 
}
PUT /_river/newindex/_meta
{
   "type": "csv",
   "csv_file": {
 "folder": "E:/elasticsearch-1.2.2/tmp",
 "poll": "5m",
  "first_line_is_header":"true",
 "field_separator": ";",
 "escape_character": " ",
 "quote_character": "'"
   }
}

and am always getting string in all types ( except date as i put the right 
format in my csv file* ( 2014-07-24T13:58:25.000Z)*

any idea ?


On Wednesday, July 30, 2014 10:47:13 AM UTC+2, David Pilato wrote:
>
> I think you are doing something wrong.
> If you defined a mapping it should not be overwritten by the CSV river as 
> far as I know.
> Enter code here...
>
>
>


 

> -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
> @dadoonet  | @elasticsearchfr 
> 
>
>
> Le 30 juillet 2014 à 10:31:07, Amirah (beldji...@gmail.com ) 
> a écrit:
>
> Hello, 
>
> Am trying to create an index using "CSV River Plugin for ElasticSearch" 
> , my csv file 
> contains *String*, *long* and *date* values.
>
> My problem is : 
>  - ElasticSearch always consider *long* values as *string* ( with default 
> mapping ) and when i try to create a specific mapping to be used, it 
> changes *long* type to *string* type.
>  - So, is there a way to tell ElasticSearch *Don't map everything as 
> string *?
>
> Any suggestion please?
>
> Thank you
>
>
>  --
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/664de1fc-1e52-4846-9660-5f5808d7b6af%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/69675304-f587-4ad2-b24c-550be4446039%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: slow filter execution

[David Pilato]

May be a stupid question: why did you put that filter inside a query and 
not within the same filter you have at the end?

For my test case it's the same every time. In the "real" query it will 
> change every time, but I planned to not cache this filter and have a less 
> granular date filter in the bool filter that would be cached. However while 
> debugging I noticed slowness with the date range filters even while testing 
> with the same value repeatedly.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/af76ca41-9045-4a4f-b82c-b9c86d964ace%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch always uses the default mapping

[David Pilato]

I think you are doing something wrong.
If you defined a mapping it should not be overwritten by the CSV river as far 
as I know.

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 30 juillet 2014 à 10:31:07, Amirah (beldjilal...@gmail.com) a écrit:

Hello,

Am trying to create an index using "CSV River Plugin for ElasticSearch", my csv 
file contains String, long and date values.

My problem is : 
 - ElasticSearch always consider long values as string ( with default mapping ) 
and when i try to create a specific mapping to be used, it changes long type to 
string type.
 - So, is there a way to tell ElasticSearch Don't map everything as string ?

Any suggestion please?

Thank you


--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/664de1fc-1e52-4846-9660-5f5808d7b6af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.53d8b102.b03e0c6.f0d0%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: [java api] Trying to use groovy instead of mvel

[Laurent T.]

Hi David,

I tried, as you suggested, to activate dynamic scripting and to force 
groovy as a default_lang but the results stay unchanged.
And yeah, no other node on the cluster.. Here's the test's output logs:

TestClient: Loading config files... 
> TestClient: Creating local node... 
> juil. 30, 2014 10:43:39 AM org.elasticsearch.node
> Infos: [Gardevoir] version[1.3.0], pid[6876], 
> build[1265b14/2014-07-23T13:46:36Z]
> juil. 30, 2014 10:43:39 AM org.elasticsearch.node
> Infos: [Gardevoir] initializing ...
> juil. 30, 2014 10:43:39 AM org.elasticsearch.plugins
> Infos: [Gardevoir] loaded [], sites []
> juil. 30, 2014 10:43:40 AM org.elasticsearch.node
> Infos: [Gardevoir] initialized
> juil. 30, 2014 10:43:40 AM org.elasticsearch.node
> Infos: [Gardevoir] starting ...
> juil. 30, 2014 10:43:40 AM org.elasticsearch.transport
> Infos: [Gardevoir] bound_address {local[1]}, publish_address {local[1]}
> juil. 30, 2014 10:43:40 AM org.elasticsearch.discovery
> Infos: [Gardevoir] elasticsearch/WFBIDGAJSgGjs2Pg2zCSiw
> juil. 30, 2014 10:43:40 AM org.elasticsearch.cluster.service
> Infos: [Gardevoir] new_master 
> [Gardevoir][WFBIDGAJSgGjs2Pg2zCSiw][Laurent-PC][local[1]]{local=true}, 
> reason: local-disco-initial_connect(master)
> juil. 30, 2014 10:43:40 AM org.elasticsearch.gateway
> Infos: [Gardevoir] recovered [0] indices into cluster_state
> juil. 30, 2014 10:43:41 AM org.elasticsearch.http
> Infos: [Gardevoir] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, 
> publish_address {inet[/192.168.1.22:9200]}
> juil. 30, 2014 10:43:41 AM org.elasticsearch.node
> Infos: [Gardevoir] started
> TestClient: Creating index test_testclient...
> juil. 30, 2014 10:43:41 AM org.elasticsearch.cluster.metadata
> Infos: [Gardevoir] [test_testclient] creating index, cause [api], shards 
> [1]/[0], mappings [testtype]
> TestClient: Seeding test data...
> TestClient: Seededjuil. 30, 2014 10:43:41 AM 
> org.elasticsearch.cluster.metadata
> Infos: [Gardevoir] [test_testclient] update_mapping [testtype] (dynamic)
> .. 3 documents
> TestClient: Deleting index test_testclient...
> TestClient: Shuting down local node... Done.


I extracted what i could from my code into this gist:
https://gist.github.com/Crystark/2deb118add9115bed6a6

I used those files in a brand new maven project and got to reproduce the 
error.

Hope you can help.
Thanks again
Regards,
Laurent

On Tuesday, July 29, 2014 6:10:02 PM UTC+2, David Pilato wrote:
>
> Anyway i thought i may need to explicitly declare i'm using groovy so i 
> added "groovy" as a second parameter to my scriptFunction
> And there i got the following error: script_lang not supported [groovy]
>
>
> Indeed, that's what you need to do. It should work I think. Never played 
> with it though.
>
> You may also need to activate explicitly dynamic scripting: 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_enabling_dynamic_scripting
> Although I think it's not required here as you are using Groovy.
>
> For sure, you don't need and you must not install groovy lang plugin. 
> That's why we did not release it for es-1.3 by the way.
>
> You don't have any other node connected running within your cluster I 
> guess.
>
> Any chance you could share your test class as a Gist? 
>
> -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
> @dadoonet  | @elasticsearchfr 
> 
>
>
> Le 29 juillet 2014 à 17:57:30, Laurent T. (lau.t...@gmail.com 
> ) a écrit:
>
> Hi, 
>
> I'm upgrading my plugin to 1.3.0 from 0.90.3 and i'm having some troubles 
> with my customScoreQuery that has become a functionScoreQuery.
>
> Here's what i did: I changed the ES version to 1.3.0 in my pom.xml, 
> adapted my code regarding classes and methods that did not exists anymore 
> so it compiles and launched the tests.
>
>  Just to be clear, my tests are executed in local mode. My client is 
> generated using
> client = NodeBuilder.nodeBuilder().local(true).node().client();
>  
> So the test about customizing the score fails with the following error: 
> dynamic 
> scripting for [mvel] disabled
> I expected that as I think i read it somewhere in the changelogs. Here's 
> what the code looks like:
>
>  
> QueryBuilders.functionScoreQuery(theQuery).add(ScoreFunctionBuilders.scriptFunction("_score
>  
>> * (doc['doc_boost'].empty ? 1 : doc['doc_boost'].value)"));
>
>
> If mvel is disabled, shouldn't groovy handle this automatically ? or is my 
> code not groovy-compliant ?
>
> Anyway i thought i may need to explicitly declare i'm using groovy so i 
> added "groovy" as a second parameter to my scriptFunction
> And there i got the following error: script_lang not supported [groovy]
>
> I tried adding the elasticsearch-lang-groovy artifact to my pom.xml but 
> that only resulted in hundred of errors that looked like conflicts as if ES 
> already carried groovy in it's package (which I just checked and seems to 
> be 

Exception when using ES 1.3.1 Caused by: java.lang.IncompatibleClassChangeError: Implementing class

[gregorymaertens via elasticsearch]

Hello,

I have a project using Play framework version 1.2.7 which used ES 1.1.1.
I wanted to update it to the latest and greatest (1.3.1), but encountered 
the following exception when running the unit tests within the play 
framework:

An unexpected error occured caused by exception 
IncompatibleClassChangeError: Implementing class

(see full trace below)

I'm using Java7 (1.7.0_51).

ES 1.2.3 has no issue, it starts with 1.3.0 to exhibit this problem.

I have looked at the issue and it seems it's caused by not using the right 
class version, but I wasn't able to pin point to the exact one.

Has anyone encounter this or know which particular class may introduce this 
issue?


Thanks,

Greg 


An unexpected error occured caused by exception 
IncompatibleClassChangeError: Implementing class


play.exceptions.UnexpectedException: Unexpected Error

at play.Invoker$Invocation.onException(Invoker.java:244)

at play.Invoker$Invocation.run(Invoker.java:286)

at Invocation.HTTP Request(Play!)

Caused by: java.lang.IncompatibleClassChangeError: Implementing class

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:800)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)

at java.net.URLClassLoader.access$100(URLClassLoader.java:71)

at java.net.URLClassLoader$1.run(URLClassLoader.java:361)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:800)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)

at java.net.URLClassLoader.access$100(URLClassLoader.java:71)

at java.net.URLClassLoader$1.run(URLClassLoader.java:361)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at 
play.classloading.enhancers.ContinuationEnhancer.enhanceThisClass(ContinuationEnhancer.java:74)

at play.CorePlugin.enhance(CorePlugin.java:297)

at play.plugins.PluginCollection.enhance(PluginCollection.java:551)

at 
play.classloading.ApplicationClasses$ApplicationClass.enhance(ApplicationClasses.java:235)

at 
play.classloading.ApplicationClassloader.loadApplicationClass(ApplicationClassloader.java:167)

at 
play.classloading.ApplicationClassloader.getAllClasses(ApplicationClassloader.java:431)

at play.Play.start(Play.java:523)

at play.Play.detectChanges(Play.java:649)

at play.Invoker$Invocation.init(Invoker.java:198)

... 1 more

10:21:21,858 ERROR ~ Error during the 500 response generation

java.lang.IncompatibleClassChangeError: Implementing class

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:800)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)

at java.net.URLClassLoader.access$100(URLClassLoader.java:71)

at java.net.URLClassLoader$1.run(URLClassLoader.java:361)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:800)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)

at java.net.URLClassLoader.access$100(URLClassLoader.java:71)

at java.net.URLClassLoader$1.run(URLClassLoader.java:361)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at 
play.classloading.enhancers.ContinuationEnhancer.enhanceThisClass(ContinuationEnhancer.java:74)

at play.CorePlugin.enhance(CorePlugin.java:297)

at play.plugins.PluginCollect

Elasticsearch always uses the default mapping

[Amirah]

Hello,

Am trying to create an index using "CSV River Plugin for ElasticSearch" 
, my csv file 
contains *String*, *long* and* date* values.

My problem is : 
 - ElasticSearch always consider* long* values as *string* ( with default 
mapping ) and when i try to create a specific mapping to be used, it changes* 
long* type to *string* type.
 - So, is there a way to tell ElasticSearch *Don't map everything as 
string *?

Any suggestion please?

Thank you


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/664de1fc-1e52-4846-9660-5f5808d7b6af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: slow filter execution

[Kireet Reddy]

For my test case it's the same every time. In the "real" query it will
change every time, but I planned to not cache this filter and have a less
granular date filter in the bool filter that would be cached. However while
debugging I noticed slowness with the date range filters even while testing
with the same value repeatedly.
On Jul 29, 2014 10:49 PM, "David Pilato"  wrote:

> Any chance your filter value changes for every call?
> Or are you using exactly the same value each time?
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
>
> Le 30 juil. 2014 à 05:03, Kireet Reddy  a écrit :
>
> One of my queries has been consistently taking 500ms-1s and I can't figure
> out why. Here is the query
>  (it looks a bit
> strange as I have removed things that didn't seem to affect execution
> time). When I remove the range filter, the query consistently takes < 10ms.
> The query itself only results 1 hit with or without the range filter, so I
> am not sure why simply including this filter adds so much time. My nodes
> are not experiencing any filter cache evictions. I also tried moving it to
> the bool section with no luck. Changing execution to "fielddata" does
> improve execution time to < 10ms though. Since I am sorting on the same
> field, I suppose this should be fine. But I would like to understand why
> the slowdown occurs. The published field is a date type and has eager field
> data loading enabled.
>
> Thanks
> Kireet
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/994f4700-7a52-4db4-a2a7-d252732517bd%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/N0z5eZRPO2A/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CE4B26B8-5837-46C5-9E89-2AFBADED9BB6%40pilato.fr
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CACkKG4iMwtd-i_NE2mWM6Ce3WeEGM_cpsJXzFsdOUc5n_PTU-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: sum-aggregation script doesn't allow negative values?

[Colin Goodheart-Smithe]

Also, your shard_size parameter should always be greater than the size 
parameter.  So if you are asking for size of 10 then I would try setting 
shard_size to 20 or 30.

On Wednesday, 30 July 2014 09:22:16 UTC+1, Colin Goodheart-Smithe wrote:
>
> Would you be able to re-run your query and post the stack trace from the 
> Elasticsearch server logs.  This might help to work out whats going on.
>
> Thanks
>
> Colin
>
> On Tuesday, 29 July 2014 12:29:00 UTC+1, Valentin wrote:
>>
>> Ok. I think I found the problem. As soon as I try to sort on the script 
>> value it ceases to work
>>
>> works, but unsorted
>> {
>>   "size": 0,
>>   "aggs": {
>> "winners": {
>>   "terms": {
>> "field": "tit",
>> "size": 10,
>> "shard_size": 4
>>   },
>>   "aggs": {
>> "articles_over_time": {
>>   "date_histogram": {
>> "field": "datetime",
>> "interval": "1d"
>>   }
>> },
>> "diff": {
>>   "sum": {
>> "script": "(doc['datetime'].value < 140641200) ? -1 : 1",
>> "lang": "groovy"
>>   }
>> }
>>   }
>> }
>>   }
>> }
>>
>> does not work:
>> {
>>   "size": 0,
>>   "aggs": {
>> "winners": {
>>   "terms": {
>> "field": "tit",
>> "size": 10,
>> "order": {
>>   "diff": "desc"
>> },
>> "shard_size": 4
>>   },
>>   "aggs": {
>> "articles_over_time": {
>>   "date_histogram": {
>> "field": "datetime",
>> "interval": "1d"
>>   }
>> },
>> "diff": {
>>   "sum": {
>> "script": "(doc['datetime'].value < 140641200) ? -1 : 1",
>> "lang": "groovy"
>>   }
>> }
>>   }
>> }
>>   }
>> }
>>
>>
>>
>>
>> On Tuesday, July 29, 2014 12:40:15 PM UTC+2, Valentin wrote:
>>>
>>> Hi Colin,
>>>
>>> I could figure out the shard_size problem thanks to your help.
>>>
>>> For the 'datetime' error: I checked and it exists in all the indices. It 
>>> has the correct mappings and the therefor probably could not have wrong 
>>> values I guess. And using the elasticsearch-head plugin I dont get the 
>>> error but a wrong result which really seems strange.
>>>
>>> Thanks
>>> Valentin
>>>
>>> On Tuesday, July 29, 2014 11:54:08 AM UTC+2, Colin Goodheart-Smithe 
>>> wrote:

 Firstly, I think the reason you are only getting results from one index 
 when you are asking for a size of 1 in your terms aggregation is because 
 you are asking for the top 1 bucket from each shard on each index.  This 
 will then be merged together and only the top bucket will be kept.  If the 
 top bucket is not the same on all indexes then you will not get results 
 from all indices.  Setting the shard_size parameter to something like 10 
 can help with this (see 
 http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_document_counts_are_approximate
  
 for more information on this)

 Second, I wonder if the reason you are getting the error from your 
 script is that you don't have a 'datetime' value for all of your documents 
 in some of your indices?

 Regards,

 Colin

 On Monday, 28 July 2014 16:04:55 UTC+1, Valentin wrote:
>
> Hi Colin,
>
> now it gets really strange. First my alias
> curl 'http://localhost:9200/_alias?pretty'
> { 
>   "live-2014-07-27" : { 
>
> "aliases" : { 
>
>   "aggtest" : { } 
>
> } 
>
>   }, 
>
>   "live-2014-07-26" : { 
>
> "aliases" : { 
>
>   "aggtest" : { } 
>
> } 
>
>   } 
>
> }
>
>
> I tried two different queries:
> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true' 
> -d '{
>   "size": 0,
>   "aggs": {
> "winners": {
>   "terms": {
> "field": "tit",
> "order": {
>   "diff": "desc"
> },
> "size": 1
>   },
>   "aggs": {
> "articles_over_time": {
>   "date_histogram": {
> "field": "datetime",
> "interval": "1d"
>   }
> },
> "diff": {
>   "sum": {
> "script": "(doc['datetime'].value < 140641200) ? -1 : 
> 1",
> "lang": "groovy"
>   }
> }
>   }
> }
>   }
> }'
>
> and
>
> curl -XPOST '
> http://localhost:9200/live-2014-07-26,live-2014-07-27/video/_search?pretty=true'
>  
> .
>
> both do give me a result (but a wrong one) when I do query using 
> elasticsearch-head but result in an error if I use the commandline
>
> {
>
>   "error" : "SearchPhaseExecutionException[F

Re: sum-aggregation script doesn't allow negative values?

[Colin Goodheart-Smithe]

Would you be able to re-run your query and post the stack trace from the 
Elasticsearch server logs.  This might help to work out whats going on.

Thanks

Colin

On Tuesday, 29 July 2014 12:29:00 UTC+1, Valentin wrote:
>
> Ok. I think I found the problem. As soon as I try to sort on the script 
> value it ceases to work
>
> works, but unsorted
> {
>   "size": 0,
>   "aggs": {
> "winners": {
>   "terms": {
> "field": "tit",
> "size": 10,
> "shard_size": 4
>   },
>   "aggs": {
> "articles_over_time": {
>   "date_histogram": {
> "field": "datetime",
> "interval": "1d"
>   }
> },
> "diff": {
>   "sum": {
> "script": "(doc['datetime'].value < 140641200) ? -1 : 1",
> "lang": "groovy"
>   }
> }
>   }
> }
>   }
> }
>
> does not work:
> {
>   "size": 0,
>   "aggs": {
> "winners": {
>   "terms": {
> "field": "tit",
> "size": 10,
> "order": {
>   "diff": "desc"
> },
> "shard_size": 4
>   },
>   "aggs": {
> "articles_over_time": {
>   "date_histogram": {
> "field": "datetime",
> "interval": "1d"
>   }
> },
> "diff": {
>   "sum": {
> "script": "(doc['datetime'].value < 140641200) ? -1 : 1",
> "lang": "groovy"
>   }
> }
>   }
> }
>   }
> }
>
>
>
>
> On Tuesday, July 29, 2014 12:40:15 PM UTC+2, Valentin wrote:
>>
>> Hi Colin,
>>
>> I could figure out the shard_size problem thanks to your help.
>>
>> For the 'datetime' error: I checked and it exists in all the indices. It 
>> has the correct mappings and the therefor probably could not have wrong 
>> values I guess. And using the elasticsearch-head plugin I dont get the 
>> error but a wrong result which really seems strange.
>>
>> Thanks
>> Valentin
>>
>> On Tuesday, July 29, 2014 11:54:08 AM UTC+2, Colin Goodheart-Smithe wrote:
>>>
>>> Firstly, I think the reason you are only getting results from one index 
>>> when you are asking for a size of 1 in your terms aggregation is because 
>>> you are asking for the top 1 bucket from each shard on each index.  This 
>>> will then be merged together and only the top bucket will be kept.  If the 
>>> top bucket is not the same on all indexes then you will not get results 
>>> from all indices.  Setting the shard_size parameter to something like 10 
>>> can help with this (see 
>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_document_counts_are_approximate
>>>  
>>> for more information on this)
>>>
>>> Second, I wonder if the reason you are getting the error from your 
>>> script is that you don't have a 'datetime' value for all of your documents 
>>> in some of your indices?
>>>
>>> Regards,
>>>
>>> Colin
>>>
>>> On Monday, 28 July 2014 16:04:55 UTC+1, Valentin wrote:

 Hi Colin,

 now it gets really strange. First my alias
 curl 'http://localhost:9200/_alias?pretty'
 { 
   "live-2014-07-27" : { 

 "aliases" : { 

   "aggtest" : { } 

 } 

   }, 

   "live-2014-07-26" : { 

 "aliases" : { 

   "aggtest" : { } 

 } 

   } 

 }


 I tried two different queries:
 curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true' 
 -d '{
   "size": 0,
   "aggs": {
 "winners": {
   "terms": {
 "field": "tit",
 "order": {
   "diff": "desc"
 },
 "size": 1
   },
   "aggs": {
 "articles_over_time": {
   "date_histogram": {
 "field": "datetime",
 "interval": "1d"
   }
 },
 "diff": {
   "sum": {
 "script": "(doc['datetime'].value < 140641200) ? -1 : 
 1",
 "lang": "groovy"
   }
 }
   }
 }
   }
 }'

 and

 curl -XPOST '
 http://localhost:9200/live-2014-07-26,live-2014-07-27/video/_search?pretty=true'
  
 .

 both do give me a result (but a wrong one) when I do query using 
 elasticsearch-head but result in an error if I use the commandline

 {

   "error" : "SearchPhaseExecutionException[Failed to execute phase 
 [query], all shards failed; shardFailures 
 {[_MxuihP3TfmZV4FYUQaRQQ][live-2014-07-26][1]: 
 QueryPhaseExecutionException[[live-2014-07-26][1]: 
 query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed 
 [Failed to execute main query]]; nested: 
 GroovyScriptExecutionException[MissingPropertyException[No such property: 
 datetime for class: Script126]]; 
 }{[FYhB58m

Re: changing number of shards for new indices

[Mark Walkom]

It doesn't change existing indexes only new ones.
You can either do the setting change via the API or in the config, if you
choose the latter you will need a restart.

See
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-create-index.html

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 30 July 2014 17:27, Kingdom Joy  wrote:

> Typo in my previous message, here's corrected post:
>
> Hello,
>
> Right now I have two nodes in my ES (part of ELK stack) cluster and 1
> shard for each index. I would like to change number of shards to two for
> future indices. Can I do this by changing config file and restarting ES?
> Will it change number of shards for indices created after restarting ES?
> Will it break existing indices?
>
> Thank you!
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/b1ba628e-c82f-4d68-a798-bc74255f34cc%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624b9fhhBO3dUz_reMy7jLmBoX23G-5zUpVagHk%2BBnd2NrA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: changing number of shards for new indices

[Kingdom Joy]

Typo in my previous message, here's corrected post:

Hello,

Right now I have two nodes in my ES (part of ELK stack) cluster and 1 shard 
for each index. I would like to change number of shards to two for future 
indices. Can I do this by changing config file and restarting ES? Will it 
change number of shards for indices created after restarting ES? Will it 
break existing indices?

Thank you!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b1ba628e-c82f-4d68-a798-bc74255f34cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

changing number of shards for new indices

[Kingdom Joy]

Hello,

Right now I have two nodes in my ES (part of ELK stack) cluster and 1 shard 
for each index. I would like to change number of shards to two for future 
indices. Can I do this by changing config file and restarting logstash? 
Will it change number of shards for indices created after restarting ES? 
Will it break existing indices?

Thank you!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5909f618-241d-4323-aa4b-d018dac7bf96%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Status red on Marvel overview raising shardFailures

[Idan]

Hi,
Wonder if anyone got any clue about this? maybe additional logs needed to 
nail this one?

thanks.


On Tuesday, July 29, 2014 10:14:42 AM UTC+3, Idan wrote:
>
> I have status red on marvel dashboard. If I check the  the 'Shared 
> allocation' tab on the overview I see this error:
>
> Oops! SearchPhaseExecutionException[Failed to execute phase 
> [query_fetch], all shards failed; shardFailures 
> {[u-gTT9MhQ7mG9nP2k28WWw][.marvel-2014.07.25][0]: 
> RemoteTransportException[[Atom 
> Bob][inet[/192.168.108.206:9300]][search/phase/query+fetch]]; nested: 
> SearchParseException[[.marvel-2014.07.25][0]: from[0],size[300]: Parse 
> Failure [Failed to parse source 
> [{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"to":"2014-07-28T10:58:25.753Z","from":"2014-07-22T07:12:52.477Z"}}]]];
>  
> nested: SearchParseException[[.marvel-2014.07.25][0]: from[0],size[300]: 
> Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }]
>
>
>
> Any idea?
> thanks.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/70f5a84e-04da-410d-964c-51675e610c9b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Update a field if _source is disabled

[David Pilato]

No you can't as behind the scene the full document is removed and inserted with 
new values (new version).

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 30 juil. 2014 à 08:52, "'Sandeep Ramesh Khanzode' via elasticsearch" 
 a écrit :

Hi, 

I read it here 
(http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-update.html)
 that the _source field needs to be enabled for Update API to work.

Does it mean that from Java or REST API, I cannot update any field defined in 
the type mapping unless the _source is enabled? 

Can I just use the stored:true on that field and update it? If so, can you 
please show an example?

Thanks,
Sandeep

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4abea481-e8d4-4295-a450-45caecb2eaf8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/13292F23-3013-4E5E-A0A5-981F6D5D45F4%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.