date:20141103

Well. I’m use to run demo where I can inject on my laptop (SSD drives) around 
8k to 10k doc per second.
I think the biggest problem you can have is to read your source documents not 
to write them to elasticsearch.

With a single index, I would probably reindex the 400 000 docs every day in a 
new a clean index and then switch the alias from old to new index.

But it depends on your read rate I guess.

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet  | @elasticsearchfr 
 | @scrutmydocs 




> Le 3 nov. 2014 à 23:43, Ori P  a écrit :
> 
> And if I may ask, do you have a suggestion on how to update the single index? 
> I need to replace on a daily basis a bulk of about 20,000 documents at once, 
> with as little performance and data availability implications as possible.
> 
> On Tuesday, November 4, 2014 12:21:51 AM UTC+2, David Pilato wrote:
> Hmmm. Sounds like I misread what you explained in 2.
> 
> I missed the fact you want to have one index per store. So let me change my 
> answer.
> If a single index, one shard, can hold your 400 000 docs which sounds 
> reasonable to me, then one single index will be faster than querying 20 
> indices.
> 
> My 2 cents
> 
> -- 
> David Pilato | Technical Advocate | Elasticsearch.com 
> 
> @dadoonet  | @elasticsearchfr 
>  | @scrutmydocs 
> 
> 
> 
> 
>> Le 3 nov. 2014 à 23:01, Ori P > a écrit :
>> 
>> Thanks for replying David.
>> 
>> I thought approach 2 might be problematic since the alias on multiple 
>> indices would cause a query to run on every index separately, which I 
>> thought might slow things down. Apparently I was wrong?
>> 
>> And thanks for the tip about the refresh interval :)
>> 
>> On Monday, November 3, 2014 11:54:38 PM UTC+2, David Pilato wrote:
>> I don't see any benefit of solution 1.
>> 
>> I would definitely do solution 2.
>> 
>> I don't really think you could see a difference search time wise. But in 
>> term of IO 2 is better.
>> Also, you should modify refresh interval while indexing to -1 and call 
>> refresh after the bulk load.
>> 
>> HTH
>> 
>> --
>> David ;-)
>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>> 
>> Le 3 nov. 2014 à 21:31, Ori P > a écrit :
>> 
>>> I would appreciate your suggestions in helping me design my elasticsearch 
>>> index.
>>> 
>>> I'm intending to index product feeds from about 20 on-line stores, each 
>>> store not having more than 20,000 products. each product has about 15 basic 
>>> fields.
>>> Most of the searches would be done on specific product categories, and not 
>>> specific stores.
>>> 
>>> Each store feed is updated every few days (each store separately), by 
>>> receiving an XML file containing all the products in the store (no deltas). 
>>> Each update, I need to remove from my index all the existing products from 
>>> that store and add the new ones.
>>> 
>>> I thought of two possibles approaches:
>>> 
>>> 1. Create a single index + an alias to that index. Once a new feed is 
>>> received, clone the existing index to a new index, remove from the new 
>>> index all the old products, add the new products and finally change the 
>>> alias to point to the new index.
>>> 
>>> 2. Create an index for each store, and an alias that points to all of the 
>>> indices. Once a new feed is received, just index it from scratch, remove 
>>> the old store index from the alias and add the new one.
>>> 
>>> I'm not sure which way will give me faster search results? or maybe there 
>>> is an even better approach I didn't think of...
>>> 
>>> Thanks in advance,
>>> 
>>> Ori
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to elasticsearc...@googlegroups.com <>.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/34f2766d-cada-4ba9-a4fa-961c34aa2f8b%40googlegroups.com
>>>  
>>> .
>>> For more options, visit https://groups.google.com/d/optout 
>>> .
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/6c85ec37-e93e-47d6-a29f-72207f9925d8%40googlegroups.com
>>  
>> .
>> For mor

Re: Bool Queries and MUST/SHOULD combinations

2014-11-03 Thread kazoompa

Thanks Ivan,

We would like to create complex queries explained in this 
page: 
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/combining-filters.html#bool-filter.
 
I have to admit I don't see why anybody would like to put MUSTs and SHOULDs 
at same level. After further analysis, it seems that if I like to do 
something like:

(For this example conside A, B and, C as terms filters and 'in' implying 
their possible values:)

A in [a1, a2,...] OR B in [b1, b2,...] AND C in [c1, c2. ...]  // order is 
important


I have to implement my bool filter as:

{
  "bool": {
"must": [
  {
"bool": {
  "should": [
{
  "terms": {
"A": [
  "a1",
  "a2"
]
  }
},
{
  "terms": {
"B": [
  "b1",
  "b2"
]
  }
}
  ]
},
"terms": {
  "C": [
"c1",
"c2"
  ]
}
  }
]
  }
}

   
It's sort of a Polish notation for queries ;)

Cheers.



On Monday, November 3, 2014 5:02:15 PM UTC-5, Ivan Brusic wrote:
>
> Must clauses are queries that must return a document. In the first query, 
> any document returned MUST have a location of Germany. The valueType should 
> clause is optional and actually pointless as a filter since it does not 
> contribute to scoring.
>
> Can you explain what your query should be doing in terms of boolean logic?
>
> -- 
> Ivan
>
> On Sat, Nov 1, 2014 at 4:39 PM, kazoompa > 
> wrote:
>
>> Hi,
>>
>> Below is my data and the two queries that I tested, first one failing and 
>> the latter working. I start to believe that if one wants to combine several 
>> SHOULD and MUST filters, the outer one must always be SHOULD. Is this a 
>> correct assumption? In our application, we have much more complex situation 
>> with several filters within each MUST and SHOULD. And lastly, where should 
>> place a MUST_NOT in this case?
>>
>> Many thanks.
>>
>>
>>
>> Here is my data:
>>
>> _index,_type,_id,_score,_source.id,_source.type,_source.valueType,_source.sentence,_source.location
>> "test","var","0","1","0","study","text","Lorem text is jumbled","spain"
>> "test","var","1","1","1","study","text","bla bla bla","spain"
>> "test","var","2","1","2","schema","decimal","ipsum","germany"
>> "test","var","3","1","3","study","integer","lorem","france"
>>
>>
>>
>>
>>
>> Here is my FAILING query:
>>
>> {
>>   "query": {
>> "filtered": {
>>   "query": {
>> "match_all": {}
>>   },
>>   "filter": {
>> "bool": {
>>   "must": {
>> "terms": {
>>   "location": [
>> "germany"
>>   ]
>> }
>>   },
>>   "should": {
>> "terms": {
>>   "valueType": [
>> "integer"
>>   ]
>> }
>>   }
>> }
>>   }
>> }
>>   }
>> }
>>
>> Here is my WORKING query returning IDs 2 and 3:
>>
>> {
>>   "query": {
>> "bool": {
>>   "should": [
>> {
>>   "terms": {
>> "location": [
>>   "germany"
>> ]
>>   }
>> },
>> {
>>   "bool": {
>> "must": [
>>   {
>> "terms": {
>>   "valueType": [
>> "integer"
>>   ]
>> }
>>   }
>> ]
>>   }
>> }
>>   ]
>> }
>>   }
>> }
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/d4597d15-8785-4e97-9c3f-8be9aacddf9b%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a33a6974-4a95-4632-9c56-bea3d19ce7f0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Designing an index that holds updating product data feeds

2014-11-03 Thread Ori P

And if I may ask, do you have a suggestion on how to update the single 
index? I need to replace on a daily basis a bulk of about 20,000 documents 
at once, with as little performance and data availability implications as 
possible.

On Tuesday, November 4, 2014 12:21:51 AM UTC+2, David Pilato wrote:
>
> Hmmm. Sounds like I misread what you explained in 2.
>
> I missed the fact you want to have one index per store. So let me change 
> my answer.
> If a single index, one shard, can hold your 400 000 docs which sounds 
> reasonable to me, then one single index will be faster than querying 20 
> indices.
>
> My 2 cents
>
> -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com 
> *
> @dadoonet  | @elasticsearchfr 
>  | @scrutmydocs 
> 
>
>
>  
> Le 3 nov. 2014 à 23:01, Ori P > a écrit :
>
> Thanks for replying David.
>
> I thought approach 2 might be problematic since the alias on multiple 
> indices would cause a query to run on every index separately, which I 
> thought might slow things down. Apparently I was wrong?
>
> And thanks for the tip about the refresh interval :)
>
> On Monday, November 3, 2014 11:54:38 PM UTC+2, David Pilato wrote:
>>
>> I don't see any benefit of solution 1.
>>
>> I would definitely do solution 2.
>>
>> I don't really think you could see a difference search time wise. But in 
>> term of IO 2 is better.
>> Also, you should modify refresh interval while indexing to -1 and call 
>> refresh after the bulk load.
>>
>> HTH
>>
>> --
>> David ;-)
>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>>
>> Le 3 nov. 2014 à 21:31, Ori P  a écrit :
>>
>> I would appreciate your suggestions in helping me design my elasticsearch 
>> index.
>>
>> I'm intending to index product feeds from about 20 on-line stores, each 
>> store not having more than 20,000 products. each product has about 15 basic 
>> fields.
>> Most of the searches would be done on specific product categories, and 
>> not specific stores.
>>
>> Each store feed is updated every few days (each store separately), by 
>> receiving an XML file containing all the products in the store (no deltas). 
>> Each update, I need to remove from my index all the existing products from 
>> that store and add the new ones.
>>
>> I thought of two possibles approaches:
>>
>> 1. Create a single index + an alias to that index. Once a new feed is 
>> received, clone the existing index to a new index, remove from the new 
>> index all the old products, add the new products and finally change the 
>> alias to point to the new index.
>>
>> 2. Create an index for each store, and an alias that points to all of the 
>> indices. Once a new feed is received, just index it from scratch, remove 
>> the old store index from the alias and add the new one.
>>
>> I'm not sure which way will give me faster search results? or maybe there 
>> is an even better approach I didn't think of...
>>
>> Thanks in advance,
>>
>> Ori
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/34f2766d-cada-4ba9-a4fa-961c34aa2f8b%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/6c85ec37-e93e-47d6-a29f-72207f9925d8%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6e4d869d-f09b-4f20-b2ca-4639c4a7bab4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Designing an index that holds updating product data feeds

Hmmm. Sounds like I misread what you explained in 2.

I missed the fact you want to have one index per store. So let me change my 
answer.
If a single index, one shard, can hold your 400 000 docs which sounds 
reasonable to me, then one single index will be faster than querying 20 indices.

My 2 cents

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet  | @elasticsearchfr 
 | @scrutmydocs 




> Le 3 nov. 2014 à 23:01, Ori P  a écrit :
> 
> Thanks for replying David.
> 
> I thought approach 2 might be problematic since the alias on multiple indices 
> would cause a query to run on every index separately, which I thought might 
> slow things down. Apparently I was wrong?
> 
> And thanks for the tip about the refresh interval :)
> 
> On Monday, November 3, 2014 11:54:38 PM UTC+2, David Pilato wrote:
> I don't see any benefit of solution 1.
> 
> I would definitely do solution 2.
> 
> I don't really think you could see a difference search time wise. But in term 
> of IO 2 is better.
> Also, you should modify refresh interval while indexing to -1 and call 
> refresh after the bulk load.
> 
> HTH
> 
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> 
> Le 3 nov. 2014 à 21:31, Ori P > a écrit :
> 
>> I would appreciate your suggestions in helping me design my elasticsearch 
>> index.
>> 
>> I'm intending to index product feeds from about 20 on-line stores, each 
>> store not having more than 20,000 products. each product has about 15 basic 
>> fields.
>> Most of the searches would be done on specific product categories, and not 
>> specific stores.
>> 
>> Each store feed is updated every few days (each store separately), by 
>> receiving an XML file containing all the products in the store (no deltas). 
>> Each update, I need to remove from my index all the existing products from 
>> that store and add the new ones.
>> 
>> I thought of two possibles approaches:
>> 
>> 1. Create a single index + an alias to that index. Once a new feed is 
>> received, clone the existing index to a new index, remove from the new index 
>> all the old products, add the new products and finally change the alias to 
>> point to the new index.
>> 
>> 2. Create an index for each store, and an alias that points to all of the 
>> indices. Once a new feed is received, just index it from scratch, remove the 
>> old store index from the alias and add the new one.
>> 
>> I'm not sure which way will give me faster search results? or maybe there is 
>> an even better approach I didn't think of...
>> 
>> Thanks in advance,
>> 
>> Ori
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/34f2766d-cada-4ba9-a4fa-961c34aa2f8b%40googlegroups.com
>>  
>> .
>> For more options, visit https://groups.google.com/d/optout 
>> .
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/6c85ec37-e93e-47d6-a29f-72207f9925d8%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4CB2DC5E-6512-4933-BA26-DDE45792D531%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Re: Bool Queries and MUST/SHOULD combinations

2014-11-03 Thread Ivan Brusic

Must clauses are queries that must return a document. In the first query,
any document returned MUST have a location of Germany. The valueType should
clause is optional and actually pointless as a filter since it does not
contribute to scoring.

Can you explain what your query should be doing in terms of boolean logic?

-- 
Ivan

On Sat, Nov 1, 2014 at 4:39 PM, kazoompa  wrote:

> Hi,
>
> Below is my data and the two queries that I tested, first one failing and
> the latter working. I start to believe that if one wants to combine several
> SHOULD and MUST filters, the outer one must always be SHOULD. Is this a
> correct assumption? In our application, we have much more complex situation
> with several filters within each MUST and SHOULD. And lastly, where should
> place a MUST_NOT in this case?
>
> Many thanks.
>
>
>
> Here is my data:
>
> _index,_type,_id,_score,_source.id,_source.type,_source.valueType,_source.sentence,_source.location
> "test","var","0","1","0","study","text","Lorem text is jumbled","spain"
> "test","var","1","1","1","study","text","bla bla bla","spain"
> "test","var","2","1","2","schema","decimal","ipsum","germany"
> "test","var","3","1","3","study","integer","lorem","france"
>
>
>
>
>
> Here is my FAILING query:
>
> {
>   "query": {
> "filtered": {
>   "query": {
> "match_all": {}
>   },
>   "filter": {
> "bool": {
>   "must": {
> "terms": {
>   "location": [
> "germany"
>   ]
> }
>   },
>   "should": {
> "terms": {
>   "valueType": [
> "integer"
>   ]
> }
>   }
> }
>   }
> }
>   }
> }
>
> Here is my WORKING query returning IDs 2 and 3:
>
> {
>   "query": {
> "bool": {
>   "should": [
> {
>   "terms": {
> "location": [
>   "germany"
> ]
>   }
> },
> {
>   "bool": {
> "must": [
>   {
> "terms": {
>   "valueType": [
> "integer"
>   ]
> }
>   }
> ]
>   }
> }
>   ]
> }
>   }
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/d4597d15-8785-4e97-9c3f-8be9aacddf9b%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCgojLxpzSrYQyW1%3DfeaF_TJdkx4dqgaxq0_sijvq6dvw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Designing an index that holds updating product data feeds

2014-11-03 Thread Ori P

Thanks for replying David.

I thought approach 2 might be problematic since the alias on multiple 
indices would cause a query to run on every index separately, which I 
thought might slow things down. Apparently I was wrong?

And thanks for the tip about the refresh interval :)

On Monday, November 3, 2014 11:54:38 PM UTC+2, David Pilato wrote:
>
> I don't see any benefit of solution 1.
>
> I would definitely do solution 2.
>
> I don't really think you could see a difference search time wise. But in 
> term of IO 2 is better.
> Also, you should modify refresh interval while indexing to -1 and call 
> refresh after the bulk load.
>
> HTH
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
> Le 3 nov. 2014 à 21:31, Ori P > a écrit :
>
> I would appreciate your suggestions in helping me design my elasticsearch 
> index.
>
> I'm intending to index product feeds from about 20 on-line stores, each 
> store not having more than 20,000 products. each product has about 15 basic 
> fields.
> Most of the searches would be done on specific product categories, and not 
> specific stores.
>
> Each store feed is updated every few days (each store separately), by 
> receiving an XML file containing all the products in the store (no deltas). 
> Each update, I need to remove from my index all the existing products from 
> that store and add the new ones.
>
> I thought of two possibles approaches:
>
> 1. Create a single index + an alias to that index. Once a new feed is 
> received, clone the existing index to a new index, remove from the new 
> index all the old products, add the new products and finally change the 
> alias to point to the new index.
>
> 2. Create an index for each store, and an alias that points to all of the 
> indices. Once a new feed is received, just index it from scratch, remove 
> the old store index from the alias and add the new one.
>
> I'm not sure which way will give me faster search results? or maybe there 
> is an even better approach I didn't think of...
>
> Thanks in advance,
>
> Ori
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/34f2766d-cada-4ba9-a4fa-961c34aa2f8b%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6c85ec37-e93e-47d6-a29f-72207f9925d8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Designing an index that holds updating product data feeds

I don't see any benefit of solution 1.

I would definitely do solution 2.

I don't really think you could see a difference search time wise. But in term 
of IO 2 is better.
Also, you should modify refresh interval while indexing to -1 and call refresh 
after the bulk load.

HTH

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

> Le 3 nov. 2014 à 21:31, Ori P  a écrit :
> 
> I would appreciate your suggestions in helping me design my elasticsearch 
> index.
> 
> I'm intending to index product feeds from about 20 on-line stores, each store 
> not having more than 20,000 products. each product has about 15 basic fields.
> Most of the searches would be done on specific product categories, and not 
> specific stores.
> 
> Each store feed is updated every few days (each store separately), by 
> receiving an XML file containing all the products in the store (no deltas). 
> Each update, I need to remove from my index all the existing products from 
> that store and add the new ones.
> 
> I thought of two possibles approaches:
> 
> 1. Create a single index + an alias to that index. Once a new feed is 
> received, clone the existing index to a new index, remove from the new index 
> all the old products, add the new products and finally change the alias to 
> point to the new index.
> 
> 2. Create an index for each store, and an alias that points to all of the 
> indices. Once a new feed is received, just index it from scratch, remove the 
> old store index from the alias and add the new one.
> 
> I'm not sure which way will give me faster search results? or maybe there is 
> an even better approach I didn't think of...
> 
> Thanks in advance,
> 
> Ori
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/34f2766d-cada-4ba9-a4fa-961c34aa2f8b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/AF883E98-1AD1-4309-8062-19CFF9EAA246%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch run as a service dies on first request

2014-11-03 Thread Alexandre Rafalovitch

These look like debug messages. Do you know where they go and what
they say. Might be where the clue is. Otherwise, I am not sure. The
usual debugging method is to try comparing failing instance with a
successful one (on file access sequence). You could try that with a
fresh local copy of the ES and see.

Regards,
   Alex.

On 3 November 2014 16:27, Jef Statham  wrote:
> [pid 17231] write(26, "[2014-11-03 16:18:31,273][DEBUG]"..., 179) = 179
> [pid 17231] write(26, "[2014-11-03 16:18:31,274][DEBUG]"..., 116) = 116
> [pid 17231] write(26, "[2014-11-03 16:18:31,275][DEBUG]"..., 207) = 207

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G-h%2BVbvaZJU53pP6Z8abh8OuLb1%3Dqh1Qz3ORXXFTiDSA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Designing an index that holds updating product data feeds

2014-11-03 Thread Ori P

I would appreciate your suggestions in helping me design my elasticsearch
index.

I'm intending to index product feeds from about 20 on-line stores, each
store not having more than 20,000 products. each product has about 15 basic
fields.
Most of the searches would be done on specific product categories, and not
specific stores.

Each store feed is updated every few days (each store separately), by
receiving an XML file containing all the products in the store (no deltas).
Each update, I need to remove from my index all the existing products from
that store and add the new ones.

I thought of two possibles approaches:

1. Create a single index + an alias to that index. Once a new feed is
received, clone the existing index to a new index, remove from the new
index all the old products, add the new products and finally change the
alias to point to the new index.

2. Create an index for each store, and an alias that points to all of the
indices. Once a new feed is received, just index it from scratch, remove
the old store index from the alias and add the new one.

I'm not sure which way will give me faster search results? or maybe there
is an even better approach I didn't think of...

Thanks in advance,

Ori

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/34f2766d-cada-4ba9-a4fa-961c34aa2f8b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch run as a service dies on first request

Updating my command, could there be an issue with the write lock on the
events-production index?

[root@localhost ~]# strace -f -e trace=file,read,write -p 17197

[pid 17231]
stat("/var/lib/elasticsearch/elasticsearch/nodes/0/indices/events-production/2/index/write.lock",
{st_mode=S_IFREG|0644, st_size=0, ...}) = 0

[pid 17231] write(26, "[2014-11-03 16:18:31,244][DEBUG]"..., 138) = 138

[pid 17231]
stat("/var/lib/elasticsearch/elasticsearch/nodes/0/indices/events-production/2/translog",
{st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

[pid 17231] write(26, "[2014-11-03 16:18:31,257][DEBUG]"..., 316) = 316

[pid 17231] open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 142

[pid 17231] read(142, "0\n", 8192)  = 2

[pid 17231] write(26, "[2014-11-03 16:18:31,273][DEBUG]"..., 179) = 179

[pid 17231] write(26, "[2014-11-03 16:18:31,274][DEBUG]"..., 116) = 116

[pid 17231] write(26, "[2014-11-03 16:18:31,275][DEBUG]"..., 207) = 207

) = ? 

[pid 17232] +++ killed by SIGKILL +++


*-Jef Statham*

Without vices there would be no virtues.
It’s a magical world, Hobbes, ol’ buddy…Let’s go exploring!


On Mon, Nov 3, 2014 at 4:16 PM, Jef Statham  wrote:

> These debugging commands are neat I'm learning as we go, here's some file
> access stats:
>
>
> [root@localhost ~]# service elasticsearch start
>
> Starting elasticsearch:[  OK  ]
>
> [root@localhost ~]# service elasticsearch status
>
> elasticsearch (pid  17129) is running...
>
> [root@localhost ~]# strace -f -e trace=file -p 17129
>
> Process 17129 attached with 35 threads - interrupt to quit
>
> [pid 17163] lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
>
> [pid 17163] lstat("/usr/share", {st_mode=S_IFDIR|0755, st_size=4096, ...})
> = 0
>
> [pid 17163] lstat("/usr/share/elasticsearch", {st_mode=S_IFDIR|0777,
> st_size=4096, ...}) = 0
>
> [pid 17163] lstat("/usr/share/elasticsearch/lib", {st_mode=S_IFDIR|0777,
> st_size=4096, ...}) = 0
>
> [pid 17163] lstat("/usr/share/elasticsearch/lib/lucene-core-4.9.1.jar",
> {st_mode=S_IFREG|0777, st_size=2507813, ...}) = 0
>
> [pid 17163]
> stat("/org/apache/lucene/codecs/lucene42/Lucene42DocValuesFormat.class",
> 0x7f1b301331c0) = -1 ENOENT (No such file or directory)
>
> [pid 17151] open("/usr/java/jdk1.8.0_25/jre/lib/amd64/server/libjvm.so",
> O_RDONLY) = 95
>
> [pid 17151] open("/usr/java/jdk1.8.0_25/jre/lib/amd64/server/libjvm.so",
> O_RDONLY) = 95
>
> ) = ? 
>
> [pid 17172] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17172 leader 17129
>
> [pid 17171] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17171 leader 17129
>
> [pid 17170] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17170 leader 17129
>
> [pid 17169] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17169 leader 17129
>
> [pid 17168] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17168 leader 17129
>
> [pid 17167] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17167 leader 17129
>
> [pid 17166] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17166 leader 17129
>
> [pid 17165] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17165 leader 17129
>
> [pid 17162] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17162 leader 17129
>
> [pid 17161] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17161 leader 17129
>
> [pid 17160] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17160 leader 17129
>
> [pid 17159] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17159 leader 17129
>
> [pid 17158] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17158 leader 17129
>
> [pid 17157] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17157 leader 17129
>
> [pid 17156] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17156 leader 17129
>
> [pid 17155] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17155 leader 17129
>
> [pid 17154] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17154 leader 17129
>
> [pid 17153] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17153 leader 17129
>
> [pid 17152] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17152 leader 17129
>
> [pid 17142] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17142 leader 17129
>
> [pid 17164] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17164 leader 17129
>
> [pid 17163] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17163 leader 17129
>
> [pid 17141] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17141 leader 17129
>
> [pid 17140] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17140 leader 17129
>
> [pid 17139] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17139 leader 17129
>
> [pid 17138] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17138 leader 17129
>
> [pid 17137] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17137 leader 17129
>
> [pid 17136] +++ killed by SIGKILL +++
>
> PANIC: handle_group_exit: 17136 leader 17129

Dinamically assing template

2014-11-03 Thread Alejandro Alves

Hello,
I have read that it is possible to assing dynamic names to the indexes like 
this:

elasticsearch {
cluster => "logstash"
index => "logstash-%{clientid}-%{+.MM.dd}"
}

What I am wondering is if it is possible to assing the template dynamically 
as well:

elasticsearch {
cluster => "logstash"
template => "/etc/logstash/conf.d/%{clientid}-template.json"
}

Also where does the variable %{clientid} come from?

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/819fe763-74b7-4a37-817c-9152547e9e42%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch run as a service dies on first request

These debugging commands are neat I'm learning as we go, here's some file
access stats:


[root@localhost ~]# service elasticsearch start

Starting elasticsearch:[  OK  ]

[root@localhost ~]# service elasticsearch status

elasticsearch (pid  17129) is running...

[root@localhost ~]# strace -f -e trace=file -p 17129

Process 17129 attached with 35 threads - interrupt to quit

[pid 17163] lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

[pid 17163] lstat("/usr/share", {st_mode=S_IFDIR|0755, st_size=4096, ...})
= 0

[pid 17163] lstat("/usr/share/elasticsearch", {st_mode=S_IFDIR|0777,
st_size=4096, ...}) = 0

[pid 17163] lstat("/usr/share/elasticsearch/lib", {st_mode=S_IFDIR|0777,
st_size=4096, ...}) = 0

[pid 17163] lstat("/usr/share/elasticsearch/lib/lucene-core-4.9.1.jar",
{st_mode=S_IFREG|0777, st_size=2507813, ...}) = 0

[pid 17163]
stat("/org/apache/lucene/codecs/lucene42/Lucene42DocValuesFormat.class",
0x7f1b301331c0) = -1 ENOENT (No such file or directory)

[pid 17151] open("/usr/java/jdk1.8.0_25/jre/lib/amd64/server/libjvm.so",
O_RDONLY) = 95

[pid 17151] open("/usr/java/jdk1.8.0_25/jre/lib/amd64/server/libjvm.so",
O_RDONLY) = 95

) = ? 

[pid 17172] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17172 leader 17129

[pid 17171] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17171 leader 17129

[pid 17170] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17170 leader 17129

[pid 17169] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17169 leader 17129

[pid 17168] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17168 leader 17129

[pid 17167] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17167 leader 17129

[pid 17166] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17166 leader 17129

[pid 17165] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17165 leader 17129

[pid 17162] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17162 leader 17129

[pid 17161] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17161 leader 17129

[pid 17160] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17160 leader 17129

[pid 17159] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17159 leader 17129

[pid 17158] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17158 leader 17129

[pid 17157] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17157 leader 17129

[pid 17156] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17156 leader 17129

[pid 17155] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17155 leader 17129

[pid 17154] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17154 leader 17129

[pid 17153] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17153 leader 17129

[pid 17152] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17152 leader 17129

[pid 17142] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17142 leader 17129

[pid 17164] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17164 leader 17129

[pid 17163] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17163 leader 17129

[pid 17141] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17141 leader 17129

[pid 17140] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17140 leader 17129

[pid 17139] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17139 leader 17129

[pid 17138] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17138 leader 17129

[pid 17137] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17137 leader 17129

[pid 17136] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17136 leader 17129

[pid 17135] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17135 leader 17129

[pid 17134] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17134 leader 17129

[pid 17133] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17133 leader 17129

[pid 17132] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17132 leader 17129

[pid 17131] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17131 leader 17129

[pid 17151] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17151 leader 17129

+++ killed by SIGKILL +++


*-Jef Statham*

Without vices there would be no virtues.
It’s a magical world, Hobbes, ol’ buddy…Let’s go exploring!


On Mon, Nov 3, 2014 at 3:43 PM, Alexandre Rafalovitch 
wrote:

> I was actually thinking more about using trace to monitor file system
> access (not just open, access as well).
>
> Regards,
>Alex.
>
> On 3 November 2014 14:08, Jef Statham  wrote:
> > Thanks for the strace suggestion, this is what my trace returns. I'm
> looking
> > into what futex is now.
>
>
>
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.g

Re: Embedded Elasticsearch single node OutOfMemoryError

2014-11-03 Thread Mark Walkom

Don't use java 6, you should be using 7 at a minimum, see
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html#jvm-version

On 4 November 2014 02:00, Doru Sular  wrote:

> I am using java6 and elasticsearch 1.1.2 version.
> I am using only 1 shard, and 0 replica.
> Any ideas?
>
> On Monday, November 3, 2014 10:16:53 AM UTC+1, Doru Sular wrote:
>>
>> Hello Elasticsearch gurus,
>> I have Elasticsearch running as a single embedded node inside my web
>> application.
>> I noticed that as soon the index folder size on the disk exceed the size
>> of of the heap size (4GB in my case) the application become unresponsive
>> and soon I got an OutOfMemory error.
>> The elasticsearch.yml has default settings, nothing is change there.
>> Do you have any suggestions how can I configure elasticsearch to get rid
>> of this problem, except the obvious solution to increase the heap size
>> value?
>> Thank you very much
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/2b6c5df0-a3fc-4bf0-8b47-40d89961b6ba%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZ%3DQ0w0rTftz9Fv8667R5rWhxtoPS1ukYfUPXgknjii9Bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch run as a service dies on first request

2014-11-03 Thread Alexandre Rafalovitch

I was actually thinking more about using trace to monitor file system
access (not just open, access as well).

Regards,
   Alex.

On 3 November 2014 14:08, Jef Statham  wrote:
> Thanks for the strace suggestion, this is what my trace returns. I'm looking
> into what futex is now.



Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEFAe-Gu8nQf%3DgfxvWvkSwX1vrN50-zkkauA8iPnWb4n8zfPcA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Cannot access elasticsearch from a Mac host

2014-11-03 Thread Mark Walkom

What have you tried?
Can you curl localhost:9200 from your laptop?

On 4 November 2014 01:13, Xollie Mdlalose  wrote:

> I am new to elasticsearch and have just installed elasticsearch on mac via
> brew but I cannot access the server from another computer on the same LAN
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/7cf4a334-7b0f-402d-bde1-3cfbfb7f4edb%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZ%3DUkEvhSKF2dhNDZDFd-8sG6koQzjdYy1VRFSeAojwb-w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch run as a service dies on first request

Thanks for the strace suggestion, this is what my trace returns. I'm
looking into what futex is now.


[root@localhost elasticsearch]# sudo service elasticsearch start

Starting elasticsearch:[  OK  ]

[root@localhost elasticsearch]# sudo service elasticsearch status

elasticsearch (pid  16887) is running...

[root@localhost elasticsearch]# strace -p 16887

\Process 16887 attached - interrupt to quit

futex(0x7f97ed4279d0, FUTEX_WAIT, 16889, NUL 

+++ killed by SIGKILL +++

[root@localhost elasticsearch]# sudo service elasticsearch start

Starting elasticsearch:[  OK  ]

[root@localhost elasticsearch]# sudo service elasticsearch status

elasticsearch (pid  16958) is running...

[root@localhost elasticsearch]# strace -p 16958

Process 16958 attached - interrupt to quit

futex(0x7f0a7e8089d0, FUTEX_WAIT, 16960, NULL 

+++ killed by SIGKILL +++


*-Jef Statham*

Without vices there would be no virtues.
It’s a magical world, Hobbes, ol’ buddy…Let’s go exploring!


On Mon, Nov 3, 2014 at 2:01 PM, Alexandre Rafalovitch 
wrote:

> Lack of permissions on filesystem access? Unexpected directory.
>
> Something like dtrace/truss/strace might be useful here.
> On 03/11/2014 1:19 pm, "Jef Statham"  wrote:
>
>> I'm running elasticsearch on Centos using the  *serivce elasticsearch
>> start. *The service happily stays running until it receives its first
>> request causing it to die.
>> *service elasticsearch status* returns
>>
>> elasticsearch dead but pid file exists
>>
>> This same problem doesn't exist when I run elasticsearch with
>> *./bin/elasticsearch*
>>
>> With logging set to trace I see no stacktrace in the logs or any
>> indication the service went down.
>>
>>
>> Where else can I look to find why elasticsearch will not accept request
>> running as a service?
>>
>>
>> More info:
>> Elasticsearch
>> version:{
>> number: "1.3.4",
>> build_hash: "a70f3ccb52200f8f2c87e9c370c6597448eb3e45",
>> build_timestamp: "2014-09-30T09:07:17Z",
>> build_snapshot: false,
>> lucene_version: "4.9"
>>
>> CentOS 6.5
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/2tIe5i2b-TA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G463EttK_d0uS4w-9fxAxter0gDx-M3ppHWhEuSB4L6g%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CABj%3DZj8sWJPb2ATE8o%2Bpf36GNSF%3Dws7%3DkBo4pR%2BJmfjiXZ8rjw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch run as a service dies on first request

2014-11-03 Thread Alexandre Rafalovitch

Lack of permissions on filesystem access? Unexpected directory.

Something like dtrace/truss/strace might be useful here.
On 03/11/2014 1:19 pm, "Jef Statham"  wrote:

> I'm running elasticsearch on Centos using the  *serivce elasticsearch
> start. *The service happily stays running until it receives its first
> request causing it to die.
> *service elasticsearch status* returns
>
> elasticsearch dead but pid file exists
>
> This same problem doesn't exist when I run elasticsearch with
> *./bin/elasticsearch*
>
> With logging set to trace I see no stacktrace in the logs or any
> indication the service went down.
>
>
> Where else can I look to find why elasticsearch will not accept request
> running as a service?
>
>
> More info:
> Elasticsearch
> version:{
> number: "1.3.4",
> build_hash: "a70f3ccb52200f8f2c87e9c370c6597448eb3e45",
> build_timestamp: "2014-09-30T09:07:17Z",
> build_snapshot: false,
> lucene_version: "4.9"
>
> CentOS 6.5
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G463EttK_d0uS4w-9fxAxter0gDx-M3ppHWhEuSB4L6g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Using function_score error

2014-11-03 Thread Manuel Sciuto

I have an error

My mapping 


   - "mappings": {
  - "comida": {
 - "dynamic": "true",
 - "numeric_detection": true,
 - "properties": {
- "id": {
   - "type": "integer"
},
- "reviews": {
   - "type": "integer"
},
- "name": {
   - "analyzer": "myAnalyzerDestinos",
   - "type": "string"
}
 }
  },
  - "actividades": {
 - "dynamic": "true",
 - "numeric_detection": true,
 - "properties": {
- "id": {
   - "type": "integer"
},
- "reviews": {
   - "type": "integer"
},
- "name": {
   - "analyzer": "myAnalyzerDestinos",
   - "type": "string"
}
 }
  },
  - "alojamiento": {
 - "dynamic": "true",
 - "numeric_detection": true,
 - "properties": {
- "id": {
   - "type": "integer"
},
- "reviews": {
   - "type": "integer"
},
- "name": {
   - "analyzer": "myAnalyzerDestinos",
   - "type": "string"
}
 }
  },
  - "transporte_&_servicios": {
 - "dynamic": "true",
 - "numeric_detection": true,
 - "properties": {
- "id": {
   - "type": "integer"
},
- "reviews": {
   - "type": "integer"
},
- "name": {
   - "analyzer": "myAnalyzerDestinos",
   - "type": "string"
}
 }
  }
   },



My Query 

GET /business/_search
{
 "query": {
   "function_score": {
 "query": {"match": {"name": "sheraton"}},
 "script_score": {
   "script": "_score  doc['reviews'].value",
   "lang": "groovy"
 }
   }
 }
}

Response 

{
   "error": "SearchPhaseExecutionException[Failed to execute phase [query], 
all shards failed; shardFailures {[pGQYzpifRMumKUcblgTp2Q][business][0]: 
QueryPhaseExecutionException[[business][0]: query[function score (name:she 
name:sher name:shera name:sherat name:sherato 
name:sheraton,function=script[_score  doc['reviews'].value], params 
[null])],from[0],size[10]: Query Failed [Failed to execute main query]]; 
nested: GroovyScriptExecutionException[MissingMethodException[No signature 
of method: 
org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript$UpdateableFloat.call()
 
is applicable for argument types: (java.lang.Long) values: [11]\nPossible 
solutions: wait(long), wait(), abs(), any(), wait(long, int), 
and(java.lang.Number)]]; }{[pGQYzpifRMumKUcblgTp2Q][business][1]: 
QueryPhaseExecutionException[[business][1]: query[function score (name:she 
name:sher name:shera name:sherat name:sherato 
name:sheraton,function=script[_score  doc['reviews'].value], params 
[null])],from[0],size[10]: Query Failed [Failed to execute main query]]; 
nested: GroovyScriptExecutionException[MissingMethodException[No signature 
of method: 
org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript$UpdateableFloat.call()
 
is applicable for argument types: (java.lang.Long) values: [16]\nPossible 
solutions: wait(long), wait(), abs(), any(), wait(long, int), 
and(java.lang.Number)]]; }]",
   "status": 500
}

Why?



El sábado, 1 de noviembre de 2014 13:02:13 UTC-3, Ryan Ernst escribió:
>
> The root cause of the error is here:
> "ScriptException[dynamic scripting for [mvel] disabled]; "
>
> I would guess you are running on ES 1.2 or 1.3? Dynamic scripting was 
> disabled by default in 1.2, and for non sandboxed languages in 1.3.  In 
> 1.4, the default script language was changed to Groovy, which is sandboxed, 
> and thus can be safely compiled dynamically.
>
> See this blog for more details:
> http://www.elasticsearch.org/blog/scripting-security/
>
> If running in 1.3, you can simply change the language of the script:
> GET /searchtube/_search
> {
>  "query": {
>"function_score": {
>  "query": {"match": {"_all": "severed"}},
>  "script_score": {
>"script": "_score * log(doc['likes'].value + doc['views'].value + 
> 1)",
>"lang": "groovy"
>  }
>}
>  }
> }
>
> Although you could also use the "expr" lang (expressions) for this simple 
> script, which will be much faster!
>
> On Wednesday, October 29, 2014 11:44:07 AM UTC-7, Manuel Sciuto wrote:
>>
>> Hello everyone 
>>
>>
>> Do not understand why it does not work 
>>
>> # Create some docs
>> PUT /searchtube/video/1 
>> {
>>   "title": "Sick Sad World: Cold Breeze on the Interstate",
>>   "description": "Is your toll collector wearing pants, a skirt, or 
>> nothing but a smile? Cold Breeze on the Interstate, next on Sick, Sad 
>> World.",
>>   "views": 500,
>>   "likes":2,
>>   "created_at": "2014-04-22T08:00:00"
>> }
>>
>> PUT /searchtube/video/2
>> {
>> "title": "Sick Sad World: The Severed Pia

Elasticsearch run as a service dies on first request

I'm running elasticsearch on Centos using the  *serivce elasticsearch 
start. *The service happily stays running until it receives its first 
request causing it to die.
*service elasticsearch status* returns

elasticsearch dead but pid file exists

This same problem doesn't exist when I run elasticsearch with 
*./bin/elasticsearch*

With logging set to trace I see no stacktrace in the logs or any indication 
the service went down.


Where else can I look to find why elasticsearch will not accept request 
running as a service?


More info: 
Elasticsearch
version:{
number: "1.3.4",
build_hash: "a70f3ccb52200f8f2c87e9c370c6597448eb3e45",
build_timestamp: "2014-09-30T09:07:17Z",
build_snapshot: false,
lucene_version: "4.9"

CentOS 6.5

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Avoid loading plugin in tests

You can disable loading plugins by using a custom class loader that only
points to the 'lib' folder of the ES home, or includes the Elasticsearch
jars from dependency path, respectively.

Settings clientSettings = ImmutableSettings.settingsBuilder()
  .classLoader(yourcustomclassloader),
 ...);
NodeBuilder.nodeBuilder().local(true).settings(clientSettings).build();
...

Jörg

On Mon, Nov 3, 2014 at 6:50 PM, Laurent T.  wrote:

> Hi,
>
> I have an ES plugin that basically intercepts POST requests on a specific
> path and builds ES queries depending on the post data before executing
> these and returning a result.
> I have some unit tests that create a local node to tests the queries. My
> problem is that it loads my plugin which I do not want.
>
> Is there any way I can avoid that ? Perhaps with some specific settings ?
>
> Here's how I'm currently getting the test es client:
>
> NodeBuilder.nodeBuilder().local(true).node().client()
>
>
> The reason I do not want it to load the plugin  is that it creates two
> instances of the plugin: one in the tests and one in the local node.
>
> Thanks
> Regards,
>
> Laurent
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/57bbfb19-341c-4d39-9c90-218596406b75%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG2WyMtYtHeUh8xBCuDE7%3DoR5Xq24TWXQAY53KKQ2H_Bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Avoid loading plugin in tests

2014-11-03 Thread Nikolas Everett

This has been in flux a bit lately but the last time I checked you had to
intentionally load plugins by adding something like this to your test:
/**
 * Enable plugin loading.
 */
@Override
protected Settings nodeSettings(int nodeOrdinal) {
return
ImmutableSettings.builder().put(super.nodeSettings(nodeOrdinal))
.put("plugins." +
PluginsService.LOAD_PLUGIN_FROM_CLASSPATH, true).build();
}

I imagine you could always set that to false.

I believe it'd be important to add this annotation to your test though:
@ElasticsearchIntegrationTest.ClusterScope(scope =
ElasticsearchIntegrationTest.Scope.SUITE, transportClientRatio = 0.0)
so that the cluster you build without your plugin doesn't get reused for
other tests that need your plugin.

Nik

On Mon, Nov 3, 2014 at 12:50 PM, Laurent T.  wrote:

> Hi,
>
> I have an ES plugin that basically intercepts POST requests on a specific
> path and builds ES queries depending on the post data before executing
> these and returning a result.
> I have some unit tests that create a local node to tests the queries. My
> problem is that it loads my plugin which I do not want.
>
> Is there any way I can avoid that ? Perhaps with some specific settings ?
>
> Here's how I'm currently getting the test es client:
>
> NodeBuilder.nodeBuilder().local(true).node().client()
>
>
> The reason I do not want it to load the plugin  is that it creates two
> instances of the plugin: one in the tests and one in the local node.
>
> Thanks
> Regards,
>
> Laurent
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/57bbfb19-341c-4d39-9c90-218596406b75%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2BeV9LOHyxLGcs2D5xP9axHh0tPE-X%2B0b%3DDTb%3D6z8_1A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Avoid loading plugin in tests

2014-11-03 Thread Laurent T.

Hi,

I have an ES plugin that basically intercepts POST requests on a specific 
path and builds ES queries depending on the post data before executing 
these and returning a result.
I have some unit tests that create a local node to tests the queries. My 
problem is that it loads my plugin which I do not want.

Is there any way I can avoid that ? Perhaps with some specific settings ?

Here's how I'm currently getting the test es client:

NodeBuilder.nodeBuilder().local(true).node().client()


The reason I do not want it to load the plugin  is that it creates two 
instances of the plugin: one in the tests and one in the local node.

Thanks
Regards,

Laurent

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/57bbfb19-341c-4d39-9c90-218596406b75%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issue with facets searching multiple indices when field is missing

2014-11-03 Thread Erbin Lim

Awesome thanks. Seems to work.

On Monday, November 3, 2014 2:36:14 PM UTC+1, Johan Rask wrote:
>
> Hi,
>
> The only solution I could find was to set these fields in other indices as 
> well, like this.
>
> if "collectd" not in [tags] {
>  mutate {
>add_field => {
>   "tx" => "0"
>   "rx" => "0"
>   "shortterm" => "0"
>   "longterm" => "0"
>   "midterm" => "0"
>   "value" => "0"
>}
>   }
>   }
>
> Den måndagen den 3:e november 2014 kl. 14:32:04 UTC+1 skrev Erbin Lim:
>>
>> Does anyone have a solution to this? I'm facing the same problem as well. 
>> Not every index will have specific fields, but I would like to plot the 
>> histograph over a period of time showing indices if the field does indeed 
>> appear.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/825fcced-8d83-4b2a-9043-5e00b9c3fc38%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: generating correct manifest.mf file when creating jar file for native elasticsearch plugin

I'm using IntelliJ Ultimate, in a linux environment, jdk 1.70_71, I have
maven and ant but haven't used them much in the past :/.

Anyway thanks for trying to help :)

Matt

On Mon, Nov 3, 2014 at 3:27 PM, joergpra...@gmail.com  wrote:

> It is hard to guess what commands you executed and what your environment
> is.
>
> You have to set up a useable IDE environment. Currently:
>
> - I recommend IntelliJ IDEA Community Edition
> - Java JDK 7u65 or Java JDK 8u25
> - Maven 3.2.3
> - Elasticsearch 1.3 or 1.4.Beta1
>
> With IDE support and Maven, detecting dependencies like JVM and
> Elasticsearch jars is a snap.
>
> For example, you could fire up a simple plugin
>
> https://github.com/jprante/elasticsearch-simple-action-plugin
>
> then change the code to add your native script, and save it as your
> project.
>
> Jörg
>
>
>
> On Mon, Nov 3, 2014 at 3:56 PM, Matthew Hall 
> wrote:
>
>> Thanks Jorg, unfortunately when I tried to install the plugin using the
>> plugin manager I get an error:
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org.elasticsearch.plugins.PluginManager
>>at gnu.java.lang.MainThread.run(libgcj.so.7rh)
>> Caused by: java.lang.ClassNotFoundException:
>> org.elasticsearch.plugins.PluginManager not found in
>> gnu.gcj.runtime.SystemClassLoader{urls=[],
>> parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
>>at java.net.URLClassLoader.findClass(libgcj.so.7rh)
>>at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
>>at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
>>at gnu.java.lang.MainThread.run(libgcj.so.7rh)
>>
>> I have also tried installing the plugin by following the instructions
>>  from
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html
>>
>> "Registering them can either be done by settings, for example:
>> script.native.my.type set to sample.MyNativeScriptFactory will register
>> a script named my. Another option is in a plugin, access ScriptModule and
>> call registerScript on it."
>>
>> if I register the script using the settings file (elasticsearch.yml)  I
>> get the error (as before)
>>
>>  Initialization Failed ...
>> 1) No implementation for
>> java.util.Set was bound.2)
>> NoClassSettingsException[Failed to load class setting [type] with value
>> [com.elasticsearch.locationscript.CustomScriptFactory]]
>>
>> ClassNotFoundException[com.elasticsearch.locationscript.CustomScriptFactory]
>>
>> I also tried the steps here as the problems seem identical
>>
>> *https://groups.google.com/a/elasticsearch.com/forum/#!msg/users/BcaI0z-LAtU/bfFIjsHP4Q8J
>> *
>>
>> but with no luck :(
>>
>> Thanks
>>
>> Matt
>>
>> On Mon, Nov 3, 2014 at 9:18 AM, joergpra...@gmail.com <
>> joergpra...@gmail.com> wrote:
>>
>>> Yes, I understood native scripts. They should go into a plugin. Quoting
>>> the docs:
>>>
>>> "Plugins are a way to enhance the basic elasticsearch functionality in a
>>> custom manner. They range from adding custom mapping types, custom
>>> analyzers (in a more built in fashion), native scripts, custom discovery
>>> and more."
>>>
>>> Jörg
>>>
>>> On Mon, Nov 3, 2014 at 10:12 AM, Matthew Hall 
>>> wrote:
>>>
 Hi thanks for the reply! I actually made a mistake in my post :/. I
 meant native scripts rather than plugins.  I tried adding the class-path to
 Manifest.mf before but this did not work.  Thanks for your suggestions
 anyway and sorry about my typo!

 Matt

 On Sat, Nov 1, 2014 at 4:54 PM, joergpra...@gmail.com <
 joergpra...@gmail.com> wrote:

> In Java, you have to add "Class-Path" to MANIFEST.MF or the class
> loader will get confused about dependencies. It depends on the class path
> when you start Elastisearch.
>
> Also, in ES, simply adding extra jars is not the preferred method for
> extensions.
>
> It is supposed that you put a plugin (a zip archive with jars) into
> the plugins folder and use es-plugin.properties instead of fiddling with
> jars in the lib folder. These jars are treated specially at start up time,
> and do not need "Class-Path" in MANIFEST.MF.
>
> See also:
>
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html
>
> Jörg
>
> On Fri, Oct 31, 2014 at 5:08 PM, Matthew Hall 
> wrote:
>
>> Hi sorry but I am new to java and elasticsearch.  I have written a
>> simple native script in intellij which compiles into classes fine.  I 
>> then
>> use the command line jar command to create the jar from the class files. 
>>  A
>> MANIFEST.MF file is generated but is always empty (apart from version and
>> created by).  When I put the jar into the elastic search lib directory 
>> and
>> add the following lines to the elasticsearch.yml
>>
>> script.native:
>>   MyNativeScript.t

Re: generating correct manifest.mf file when creating jar file for native elasticsearch plugin

It is hard to guess what commands you executed and what your environment is.

You have to set up a useable IDE environment. Currently:

- I recommend IntelliJ IDEA Community Edition
- Java JDK 7u65 or Java JDK 8u25
- Maven 3.2.3
- Elasticsearch 1.3 or 1.4.Beta1

With IDE support and Maven, detecting dependencies like JVM and
Elasticsearch jars is a snap.

For example, you could fire up a simple plugin

https://github.com/jprante/elasticsearch-simple-action-plugin

then change the code to add your native script, and save it as your project.

Jörg



On Mon, Nov 3, 2014 at 3:56 PM, Matthew Hall  wrote:

> Thanks Jorg, unfortunately when I tried to install the plugin using the
> plugin manager I get an error:
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org.elasticsearch.plugins.PluginManager
>at gnu.java.lang.MainThread.run(libgcj.so.7rh)
> Caused by: java.lang.ClassNotFoundException:
> org.elasticsearch.plugins.PluginManager not found in
> gnu.gcj.runtime.SystemClassLoader{urls=[],
> parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
>at java.net.URLClassLoader.findClass(libgcj.so.7rh)
>at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
>at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
>at gnu.java.lang.MainThread.run(libgcj.so.7rh)
>
> I have also tried installing the plugin by following the instructions
>  from
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html
>
> "Registering them can either be done by settings, for example:
> script.native.my.type set to sample.MyNativeScriptFactory will register a
> script named my. Another option is in a plugin, access ScriptModule and
> call registerScript on it."
>
> if I register the script using the settings file (elasticsearch.yml)  I
> get the error (as before)
>
>  Initialization Failed ...
> 1) No implementation for
> java.util.Set was bound.2)
> NoClassSettingsException[Failed to load class setting [type] with value
> [com.elasticsearch.locationscript.CustomScriptFactory]]
>
> ClassNotFoundException[com.elasticsearch.locationscript.CustomScriptFactory]
>
> I also tried the steps here as the problems seem identical
>
> *https://groups.google.com/a/elasticsearch.com/forum/#!msg/users/BcaI0z-LAtU/bfFIjsHP4Q8J
> *
>
> but with no luck :(
>
> Thanks
>
> Matt
>
> On Mon, Nov 3, 2014 at 9:18 AM, joergpra...@gmail.com <
> joergpra...@gmail.com> wrote:
>
>> Yes, I understood native scripts. They should go into a plugin. Quoting
>> the docs:
>>
>> "Plugins are a way to enhance the basic elasticsearch functionality in a
>> custom manner. They range from adding custom mapping types, custom
>> analyzers (in a more built in fashion), native scripts, custom discovery
>> and more."
>>
>> Jörg
>>
>> On Mon, Nov 3, 2014 at 10:12 AM, Matthew Hall 
>> wrote:
>>
>>> Hi thanks for the reply! I actually made a mistake in my post :/. I
>>> meant native scripts rather than plugins.  I tried adding the class-path to
>>> Manifest.mf before but this did not work.  Thanks for your suggestions
>>> anyway and sorry about my typo!
>>>
>>> Matt
>>>
>>> On Sat, Nov 1, 2014 at 4:54 PM, joergpra...@gmail.com <
>>> joergpra...@gmail.com> wrote:
>>>
 In Java, you have to add "Class-Path" to MANIFEST.MF or the class
 loader will get confused about dependencies. It depends on the class path
 when you start Elastisearch.

 Also, in ES, simply adding extra jars is not the preferred method for
 extensions.

 It is supposed that you put a plugin (a zip archive with jars) into the
 plugins folder and use es-plugin.properties instead of fiddling with jars
 in the lib folder. These jars are treated specially at start up time, and
 do not need "Class-Path" in MANIFEST.MF.

 See also:


 http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html

 Jörg

 On Fri, Oct 31, 2014 at 5:08 PM, Matthew Hall 
 wrote:

> Hi sorry but I am new to java and elasticsearch.  I have written a
> simple native script in intellij which compiles into classes fine.  I then
> use the command line jar command to create the jar from the class files.  
> A
> MANIFEST.MF file is generated but is always empty (apart from version and
> created by).  When I put the jar into the elastic search lib directory and
> add the following lines to the elasticsearch.yml
>
> script.native:
>   MyNativeScript.type:
> com.elasticsearch.locationscript.CustomScriptFactory
>
> Unfortunately I get the following errors
>
>  Initialization Failed ...
> 1) No implementation for
> java.util.Set was bound.2)
> NoClassSettingsException[Failed to load class setting [type] with value
> [com.elasticsearch.locationscript.CustomScriptFactory]]
>
> ClassNotFoundException[com.e

Re: Embedded Elasticsearch single node OutOfMemoryError

2014-11-03 Thread Doru Sular

I am using java6 and elasticsearch 1.1.2 version.
I am using only 1 shard, and 0 replica.
Any ideas?

On Monday, November 3, 2014 10:16:53 AM UTC+1, Doru Sular wrote:
>
> Hello Elasticsearch gurus,
> I have Elasticsearch running as a single embedded node inside my web 
> application.
> I noticed that as soon the index folder size on the disk exceed the size 
> of of the heap size (4GB in my case) the application become unresponsive 
> and soon I got an OutOfMemory error.
> The elasticsearch.yml has default settings, nothing is change there.
> Do you have any suggestions how can I configure elasticsearch to get rid 
> of this problem, except the obvious solution to increase the heap size 
> value?
> Thank you very much
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2b6c5df0-a3fc-4bf0-8b47-40d89961b6ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Using function_score error

2014-11-03 Thread Manuel Sciuto

Thank you very much !!! 

How do you recommend that I do my script_score?

El sábado, 1 de noviembre de 2014 13:02:13 UTC-3, Ryan Ernst escribió:
>
> The root cause of the error is here:
> "ScriptException[dynamic scripting for [mvel] disabled]; "
>
> I would guess you are running on ES 1.2 or 1.3? Dynamic scripting was 
> disabled by default in 1.2, and for non sandboxed languages in 1.3.  In 
> 1.4, the default script language was changed to Groovy, which is sandboxed, 
> and thus can be safely compiled dynamically.
>
> See this blog for more details:
> http://www.elasticsearch.org/blog/scripting-security/
>
> If running in 1.3, you can simply change the language of the script:
> GET /searchtube/_search
> {
>  "query": {
>"function_score": {
>  "query": {"match": {"_all": "severed"}},
>  "script_score": {
>"script": "_score * log(doc['likes'].value + doc['views'].value + 
> 1)",
>"lang": "groovy"
>  }
>}
>  }
> }
>
> Although you could also use the "expr" lang (expressions) for this simple 
> script, which will be much faster!
>
> On Wednesday, October 29, 2014 11:44:07 AM UTC-7, Manuel Sciuto wrote:
>>
>> Hello everyone 
>>
>>
>> Do not understand why it does not work 
>>
>> # Create some docs
>> PUT /searchtube/video/1 
>> {
>>   "title": "Sick Sad World: Cold Breeze on the Interstate",
>>   "description": "Is your toll collector wearing pants, a skirt, or 
>> nothing but a smile? Cold Breeze on the Interstate, next on Sick, Sad 
>> World.",
>>   "views": 500,
>>   "likes":2,
>>   "created_at": "2014-04-22T08:00:00"
>> }
>>
>> PUT /searchtube/video/2
>> {
>> "title": "Sick Sad World: The Severed Pianist",
>>   "description": "When he turned up his nose at accordion lessons, they 
>> cut off his inheritance molto allegro. The Severed Pianist, next on Sick, 
>> Sad World.",
>>   "views": 6000,
>>   "likes": 100,
>>   "created_at": "2014-04-22T12:00:00"
>> }
>>
>> #SEARCH FUCNTION_SCORE
>> GET /searchtube/_search
>> {
>>   "query": {
>> "function_score": {
>>   "query": {"match": {"_all": "severed"}},
>>   "script_score": {
>> "script": "_score * log(doc['likes'].value + doc['views'].value + 
>> 1)"
>>   }
>> }
>>   }
>> }
>>
>>
>> Error Response
>>
>> {
>>"error": "SearchPhaseExecutionException[Failed to execute phase 
>> [query], all shards failed; shardFailures 
>> {[vrJl1dg1RV2wqGZ2Hqv3zQ][searchtube][0]: 
>> SearchParseException[[searchtube][0]: from[-1],size[-1]: Parse Failure 
>> [Failed to parse source [{\n  \"query\": {\n\"function_score\": {\n 
>>  \"query\": {\"match\": {\"_all\": \"severed\"}},\n  \"script_score\": 
>> {\n\"script\": \"_score * log(doc['likes'].value + 
>> doc['views'].value + 1)\"\n  }\n}\n  }\n}\n]]]; nested: 
>> QueryParsingException[[searchtube] script_score the script could not be 
>> loaded]; nested: ScriptException[dynamic scripting for [mvel] disabled]; 
>> }{[vrJl1dg1RV2wqGZ2Hqv3zQ][searchtube][1]: 
>> SearchParseException[[searchtube][1]: from[-1],size[-1]: Parse Failure 
>> [Failed to parse source [{\n  \"query\": {\n\"function_score\": {\n 
>>  \"query\": {\"match\": {\"_all\": \"severed\"}},\n  \"script_score\": 
>> {\n\"script\": \"_score * log(doc['likes'].value + 
>> doc['views'].value + 1)\"\n  }\n}\n  }\n}\n]]]; nested: 
>> QueryParsingException[[searchtube] script_score the script could not be 
>> loaded]; nested: ScriptException[dynamic scripting for [mvel] disabled]; 
>> }{[vrJl1dg1RV2wqGZ2Hqv3zQ][searchtube][2]: 
>> SearchParseException[[searchtube][2]: from[-1],size[-1]: Parse Failure 
>> [Failed to parse source [{\n  \"query\": {\n\"function_score\": {\n 
>>  \"query\": {\"match\": {\"_all\": \"severed\"}},\n  \"script_score\": 
>> {\n\"script\": \"_score * log(doc['likes'].value + 
>> doc['views'].value + 1)\"\n  }\n}\n  }\n}\n]]]; nested: 
>> QueryParsingException[[searchtube] script_score the script could not be 
>> loaded]; nested: ScriptException[dynamic scripting for [mvel] disabled]; 
>> }{[vrJl1dg1RV2wqGZ2Hqv3zQ][searchtube][3]: 
>> SearchParseException[[searchtube][3]: from[-1],size[-1]: Parse Failure 
>> [Failed to parse source [{\n  \"query\": {\n\"function_score\": {\n 
>>  \"query\": {\"match\": {\"_all\": \"severed\"}},\n  \"script_score\": 
>> {\n\"script\": \"_score * log(doc['likes'].value + 
>> doc['views'].value + 1)\"\n  }\n}\n  }\n}\n]]]; nested: 
>> QueryParsingException[[searchtube] script_score the script could not be 
>> loaded]; nested: ScriptException[dynamic scripting for [mvel] disabled]; 
>> }{[vrJl1dg1RV2wqGZ2Hqv3zQ][searchtube][4]: 
>> SearchParseException[[searchtube][4]: from[-1],size[-1]: Parse Failure 
>> [Failed to parse source [{\n  \"query\": {\n\"function_score\": {\n 
>>  \"query\": {\"match\": {\"_all\": \"severed\"}},\n  \"script_score\": 
>> {\n\"script\": \"_score * log(doc['likes'].value + 
>> doc['views'].value

Re: generating correct manifest.mf file when creating jar file for native elasticsearch plugin

Thanks Jorg, unfortunately when I tried to install the plugin using the
plugin manager I get an error:

Exception in thread "main" java.lang.NoClassDefFoundError:
org.elasticsearch.plugins.PluginManager
   at gnu.java.lang.MainThread.run(libgcj.so.7rh)
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.plugins.PluginManager not found in
gnu.gcj.runtime.SystemClassLoader{urls=[],
parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
   at java.net.URLClassLoader.findClass(libgcj.so.7rh)
   at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
   at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
   at gnu.java.lang.MainThread.run(libgcj.so.7rh)

I have also tried installing the plugin by following the instructions
 from
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html

"Registering them can either be done by settings, for example:
script.native.my.type set to sample.MyNativeScriptFactory will register a
script named my. Another option is in a plugin, access ScriptModule and
call registerScript on it."

if I register the script using the settings file (elasticsearch.yml)  I get
the error (as before)

 Initialization Failed ...
1) No implementation for
java.util.Set was bound.2)
NoClassSettingsException[Failed to load class setting [type] with value
[com.elasticsearch.locationscript.CustomScriptFactory]]

ClassNotFoundException[com.elasticsearch.locationscript.CustomScriptFactory]

I also tried the steps here as the problems seem identical

*https://groups.google.com/a/elasticsearch.com/forum/#!msg/users/BcaI0z-LAtU/bfFIjsHP4Q8J
*

but with no luck :(

Thanks

Matt

On Mon, Nov 3, 2014 at 9:18 AM, joergpra...@gmail.com  wrote:

> Yes, I understood native scripts. They should go into a plugin. Quoting
> the docs:
>
> "Plugins are a way to enhance the basic elasticsearch functionality in a
> custom manner. They range from adding custom mapping types, custom
> analyzers (in a more built in fashion), native scripts, custom discovery
> and more."
>
> Jörg
>
> On Mon, Nov 3, 2014 at 10:12 AM, Matthew Hall 
> wrote:
>
>> Hi thanks for the reply! I actually made a mistake in my post :/. I meant
>> native scripts rather than plugins.  I tried adding the class-path to
>> Manifest.mf before but this did not work.  Thanks for your suggestions
>> anyway and sorry about my typo!
>>
>> Matt
>>
>> On Sat, Nov 1, 2014 at 4:54 PM, joergpra...@gmail.com <
>> joergpra...@gmail.com> wrote:
>>
>>> In Java, you have to add "Class-Path" to MANIFEST.MF or the class loader
>>> will get confused about dependencies. It depends on the class path when you
>>> start Elastisearch.
>>>
>>> Also, in ES, simply adding extra jars is not the preferred method for
>>> extensions.
>>>
>>> It is supposed that you put a plugin (a zip archive with jars) into the
>>> plugins folder and use es-plugin.properties instead of fiddling with jars
>>> in the lib folder. These jars are treated specially at start up time, and
>>> do not need "Class-Path" in MANIFEST.MF.
>>>
>>> See also:
>>>
>>>
>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html
>>>
>>> Jörg
>>>
>>> On Fri, Oct 31, 2014 at 5:08 PM, Matthew Hall 
>>> wrote:
>>>
 Hi sorry but I am new to java and elasticsearch.  I have written a
 simple native script in intellij which compiles into classes fine.  I then
 use the command line jar command to create the jar from the class files.  A
 MANIFEST.MF file is generated but is always empty (apart from version and
 created by).  When I put the jar into the elastic search lib directory and
 add the following lines to the elasticsearch.yml

 script.native:
   MyNativeScript.type:
 com.elasticsearch.locationscript.CustomScriptFactory

 Unfortunately I get the following errors

  Initialization Failed ...
 1) No implementation for
 java.util.Set was bound.2)
 NoClassSettingsException[Failed to load class setting [type] with value
 [com.elasticsearch.locationscript.CustomScriptFactory]]

 ClassNotFoundException[com.elasticsearch.locationscript.CustomScriptFactory]


 I have tried to compile a simple helloworld jar and I found this did
 not work either UNTIL I modified the auto-generated (empty) Manifest file.

 Any help or a point to a comprehensive tutorial covering how to create
 the jar correctly would be much appreciated!!

 ttyl

 Matt

 --
 You received this message because you are subscribed to the Google
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/32f808bb-941f-4639-9a7d-dd57252d959e%40googlegroups.com
>>>

Non-constant scoring

2014-11-03 Thread Sofiane Cherchalli

I'm doing a multi_match query on multi_fields and I'm expecting a constant 
scoring because the search string is in all the documents. I'm running ES 
1.3.4 with ICU plugin. Here are the steps to reproduce it:

# index settings & mappings
PUT /blogs
{
  "settings": {
"analysis": {
  "filter": {
"length_filter": {
  "type": "length",
  "min": 2
}
  },
  "tokenizer": {
"nGram_tokenizer": {
  "type": "nGram",
  "min_gram": 2,
  "max_gram": 30,
  "token_chars": [ "letter", "digit", "symbol" ]
}
  },
  "analyzer": {
"nGram_analyzer": {
  "type": "custom",
  "filter": ["icu_normalizer", "icu_folding", "length_filter"],
  "tokenizer": "nGram_tokenizer"
},
"search_analyzer": {
  "type": "custom",
  "filter": ["icu_normalizer", "icu_folding", "length_filter"],
  "tokenizer": "icu_tokenizer"
},
"def_analyzer": {
  "alias": ["icu_analyzer"],
  "type": "custom",
  "filter": ["icu_normalizer", "icu_folding"],
  "tokenizer": "icu_tokenizer"
}
  }
}
  },
  "mappings": {
"_default_": {
  "_source": {
"enabled": true,
"compress": true,
"compress_threshold": "200b"
  },
  "_all": {
"enabled": true
  },  
  "dynamic_date_formats": ["-MM-dd", "date_optional_time"]
},
"post": {
  "dynamic": "strict",
  "properties": {
"title": {
  "type": "string",
  "analyzer": "def_analyzer",
  "fields": {
"ngram": {
  "type": "string",
  "analyzer": "nGram_analyzer",
  "term_vector": "with_positions_offsets"
}
  }
},
"content": {
  "type": "string",
  "analyzer": "def_analyzer",
  "fields": {
"ngram": {
  "type": "string",
  "analyzer": "nGram_analyzer",
  "term_vector": "with_positions_offsets"
}
  }
},
"visible": {
  "type": "boolean",
  "index": "not_analyzed"
},
"post_date": {
  "type": "date",
  "index": "not_analyzed"
}
  }
}
  }
}


# index some documents
PUT blogs/post/1
{
  "title": "title 1",
  "content": "",
  "visible": true,
  "post_date": "2014-11-03T12:50:00"
}

PUT blogs/post/2
{
  "title": "title 2",
  "content": "",
  "visible": true,
  "post_date": "2014-11-03T12:51:00"
}

PUT blogs/post/3
{
  "title": "title 3",
  "content": "",
  "visible": true,
  "post_date": "2014-11-03T12:52:00"
}

PUT blogs/post/4
{
  "title": "title 4",
  "content": "",
  "visible": true,
  "post_date": "2014-11-03T12:53:00"
}

PUT blogs/post/5
{
  "title": "title 5",
  "content": "",
  "visible": true,
  "post_date": "2014-11-03T12:54:00"
}


# run the multi_match query and sort the results by score (desc) and 
post_date (asc)
GET /blogs/post/_search
{
  "query": {
"filtered": {
  "query": {
"multi_match": {
  "query": "tit",
  "fields": ["title.ngram^2", "content.ngram"],
  "type": "most_fields",
  "operator": "and",
  "analyzer": "search_analyzer",
  "minimum_should_match" : 1
}
  },
  "filter": {
"term": { "visible": "true" }
  }
}
  },
  "sort" : [
{ "_score": { "order": "desc" }},
{ "post_date": { "order": "asc" }}
  ]
}


{
   "took": 4,
   "timed_out": false,
   "_shards": {
  "total": 5,
  "successful": 5,
  "failed": 0
   },
   "hits": {
  "total": 5,
  "max_score": null,
  "hits": [
 {
"_index": "blogs",
"_type": "post",
"_id": "1",
"_score": 0.025078464,
"_source": {
   "title": "title 1",
   "content": "",
   "visible": true,
   "post_date": "2014-11-03T12:50:00"
},
"sort": [
   0.025078464,
   141501900
]
 },
 {
"_index": "blogs",
"_type": "post",
"_id": "2",
"_score": 0.025078464,
"_source": {
   "title": "title 2",
   "content": "",
   "visible": true,
   "post_date": "2014-11-03T12:51:00"
},
"sort": [
   0.025078464,
   141501906
]
 },
 {
"_index": "blogs",
"_type": "post",
"_id": "3",
"_score": 0.025078464,
"_source": {
   "title": "title 3",
   "content": "",
   "visible": true,
   "post_date": "2014-11-03T12:52:00"
},
"sort": [
   0.025078464,
   1415019120

Re: generating correct manifest.mf file when creating jar file for native elasticsearch plugin

Thanks Jorge, unfortunately when I tried to install the plugin using the
plugin manager I get an error:

Exception in thread "main" java.lang.NoClassDefFoundError:
org.elasticsearch.plugins.PluginManager
   at gnu.java.lang.MainThread.run(libgcj.so.7rh)
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.plugins.PluginManager not found in
gnu.gcj.runtime.SystemClassLoader{urls=[],
parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
   at java.net.URLClassLoader.findClass(libgcj.so.7rh)
   at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
   at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
   at gnu.java.lang.MainThread.run(libgcj.so.7rh)

I have also tried installing the plugin by following the instructions
 from
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html

"Registering them can either be done by settings, for example:
script.native.my.type set to sample.MyNativeScriptFactory will register a
script named my. Another option is in a plugin, access ScriptModule and
call registerScript on it."

if I register the script using the settings file (elasticsearch.yml)  I get
the error (as before)

 Initialization Failed ...
1) No implementation for
java.util.Set was bound.2)
NoClassSettingsException[Failed to load class setting [type] with value
[com.elasticsearch.locationscript.CustomScriptFactory]]

ClassNotFoundException[com.elasticsearch.locationscript.CustomScriptFactory]

I also tried the steps here as the problems seem identical

*https://groups.google.com/a/elasticsearch.com/forum/#!msg/users/BcaI0z-LAtU/bfFIjsHP4Q8J
*

but with no luck :(

Thanks

Matt



On Mon, Nov 3, 2014 at 9:18 AM, joergpra...@gmail.com  wrote:

> Yes, I understood native scripts. They should go into a plugin. Quoting
> the docs:
>
> "Plugins are a way to enhance the basic elasticsearch functionality in a
> custom manner. They range from adding custom mapping types, custom
> analyzers (in a more built in fashion), native scripts, custom discovery
> and more."
>
> Jörg
>
> On Mon, Nov 3, 2014 at 10:12 AM, Matthew Hall 
> wrote:
>
>> Hi thanks for the reply! I actually made a mistake in my post :/. I meant
>> native scripts rather than plugins.  I tried adding the class-path to
>> Manifest.mf before but this did not work.  Thanks for your suggestions
>> anyway and sorry about my typo!
>>
>> Matt
>>
>> On Sat, Nov 1, 2014 at 4:54 PM, joergpra...@gmail.com <
>> joergpra...@gmail.com> wrote:
>>
>>> In Java, you have to add "Class-Path" to MANIFEST.MF or the class loader
>>> will get confused about dependencies. It depends on the class path when you
>>> start Elastisearch.
>>>
>>> Also, in ES, simply adding extra jars is not the preferred method for
>>> extensions.
>>>
>>> It is supposed that you put a plugin (a zip archive with jars) into the
>>> plugins folder and use es-plugin.properties instead of fiddling with jars
>>> in the lib folder. These jars are treated specially at start up time, and
>>> do not need "Class-Path" in MANIFEST.MF.
>>>
>>> See also:
>>>
>>>
>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html
>>>
>>> Jörg
>>>
>>> On Fri, Oct 31, 2014 at 5:08 PM, Matthew Hall 
>>> wrote:
>>>
 Hi sorry but I am new to java and elasticsearch.  I have written a
 simple native script in intellij which compiles into classes fine.  I then
 use the command line jar command to create the jar from the class files.  A
 MANIFEST.MF file is generated but is always empty (apart from version and
 created by).  When I put the jar into the elastic search lib directory and
 add the following lines to the elasticsearch.yml

 script.native:
   MyNativeScript.type:
 com.elasticsearch.locationscript.CustomScriptFactory

 Unfortunately I get the following errors

  Initialization Failed ...
 1) No implementation for
 java.util.Set was bound.2)
 NoClassSettingsException[Failed to load class setting [type] with value
 [com.elasticsearch.locationscript.CustomScriptFactory]]

 ClassNotFoundException[com.elasticsearch.locationscript.CustomScriptFactory]


 I have tried to compile a simple helloworld jar and I found this did
 not work either UNTIL I modified the auto-generated (empty) Manifest file.

 Any help or a point to a comprehensive tutorial covering how to create
 the jar correctly would be much appreciated!!

 ttyl

 Matt

 --
 You received this message because you are subscribed to the Google
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/32f808bb-941f-4639-9a7d-dd57252d959e%40googlegroups.com

Cannot access elasticsearch from a Mac host

2014-11-03 Thread Xollie Mdlalose

I am new to elasticsearch and have just installed elasticsearch on mac via 
brew but I cannot access the server from another computer on the same LAN 

 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7cf4a334-7b0f-402d-bde1-3cfbfb7f4edb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Find the 100 closest neighbors to a point (lng, lat)

2014-11-03 Thread Michael Lumbroso

Anyone has an idea on which tool is the best to perform this task? This 
would be of a major help :-)

Thanks for your answers!

Le vendredi 24 octobre 2014 11:11:24 UTC+2, Michael Lumbroso a écrit :
>
> Hello,
>
> sorry if this question has already been asked, but I didn't find much 
> material during my search.
> Basically, what I need to do, is find the exact 100 closest objects around 
> a spatial point (longitude, latitude), among a 1 million geolocalized 
> object all around the world.
>
> Is there an efficient way to do that? (performance is the most important 
> parameter here)
> Are there plugins/libraries to help me do so? 
> Are there better options than Elasticsearch for this very problem?
>
> Thanks for your help, and keep up the good work on this wonderful tool
>
> Michael
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6e2d21bc-3a38-4174-ba1a-3e4dfa3e6e17%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: cant check icu_tokenizer

2014-11-03 Thread Alexander Dorogikh

Thanks Jun
I was inattentive

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/18dd1b99-b02e-4811-8113-0859b1b8f18a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: cant check icu_tokenizer

2014-11-03 Thread Jun Ohtani

Hi Alexander,

You should use the plugin version 2.3.0 instead of 2.4.0, if yo use
elasticsearch 1.3.x.

https://github.com/elasticsearch/elasticsearch-analysis-icu
2014/11/03 22:25 "Alexander Dorogikh" :

> Salutations
>
> here is examples
>
> http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/icu-tokenizer.html
>
> $ curl -XGET http://127.0.0.1:9200
> {
>   "status" : 200,
>   "name" : "Victor Creed",
>   "version" : {
> "number" : "1.3.4",
> "build_hash" : "a70f3ccb52200f8f2c87e9c370c6597448eb3e45",
> "build_timestamp" : "2014-09-30T09:07:17Z",
> "build_snapshot" : false,
> "lucene_version" : "4.9"
>   },
>   "tagline" : "You Know, for Search"
> }
>
>
>
> $bin/plugin -install elasticsearch/elasticsearch-analysis-icu/2.4.0
>
>
>
> [alex@localhost ~]$ /usr/share/elasticsearch/bin/plugin -l
> Installed plugins:
> - langdetect
> - analysis-morphology
> - analysis-icu
>
> and test
>
> $ curl -XGET '
> http://127.0.0.1:9200/_analyze?tokenizer=icu_tokenizer&pretty=true' -d
> 'สวัสดี ผมมาจากกรุงเทพฯ'
> {
>   "error" : "ElasticsearchIllegalArgumentException[failed to find global
> tokenizer under [icu_tokenizer]]",
>   "status" : 400
> }
>
> Please help, What I am doing wrong ?
>
> thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/8d3d4059-477a-4bfc-9f8e-cddcf3b285d7%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPW8A5y8RdFCDJr2JbFt%3DRQGeZPoUH17S9X5ez8xDn5%2BLJ%3DWBA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Disabling default fields ("_index", "_type", "_id", "_score") in result list

2014-11-03 Thread Lasse Schou

Hi,

I want to know if it's possible to disable the "_index", "_type", "_id" and 
"_score" fields in the output list when performing a search query.

Example: 

"hits": [
 {
"_index": "eventlist_2014_10",
"_type": "eventlist",
"_id": "lcJu1j5Jvyh9ywJHsPplXA_300343y7a0ktK4iXjeccFse_EDPw_2",
"_score": 1,
"fields": {
.


Those fields add a dramatic overhead to my result list, so I'd really like 
to disable them. 

Thanks in advance,
Lasse

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1d870c9d-0714-4a78-b888-f2bb810db966%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issue with facets searching multiple indices when field is missing

2014-11-03 Thread Johan Rask

Hi,

The only solution I could find was to set these fields in other indices as 
well, like this.

if "collectd" not in [tags] {
 mutate {
   add_field => {
  "tx" => "0"
  "rx" => "0"
  "shortterm" => "0"
  "longterm" => "0"
  "midterm" => "0"
  "value" => "0"
   }
  }
  }

Den måndagen den 3:e november 2014 kl. 14:32:04 UTC+1 skrev Erbin Lim:
>
> Does anyone have a solution to this? I'm facing the same problem as well. 
> Not every index will have specific fields, but I would like to plot the 
> histograph over a period of time showing indices if the field does indeed 
> appear.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/139d1d02-a1c1-4fb0-942a-cabce7678e4b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issue with facets searching multiple indices when field is missing

2014-11-03 Thread Erbin Lim

Does anyone have a solution to this? I'm facing the same problem as well. 
Not every index will have specific fields, but I would like to plot the 
histograph over a period of time showing indices if the field does indeed 
appear.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4d61c743-fd9a-40b6-8bfd-c4b4eee7cfb8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

cant check icu_tokenizer

2014-11-03 Thread Alexander Dorogikh

Salutations

here is examples 
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/icu-tokenizer.html

$ curl -XGET http://127.0.0.1:9200
{
  "status" : 200,
  "name" : "Victor Creed",
  "version" : {
"number" : "1.3.4",
"build_hash" : "a70f3ccb52200f8f2c87e9c370c6597448eb3e45",
"build_timestamp" : "2014-09-30T09:07:17Z",
"build_snapshot" : false,
"lucene_version" : "4.9"
  },
  "tagline" : "You Know, for Search"
}



$bin/plugin -install elasticsearch/elasticsearch-analysis-icu/2.4.0



[alex@localhost ~]$ /usr/share/elasticsearch/bin/plugin -l
Installed plugins:
- langdetect
- analysis-morphology
- analysis-icu

and test

$ curl -XGET 
'http://127.0.0.1:9200/_analyze?tokenizer=icu_tokenizer&pretty=true' -d 
'สวัสดี ผมมาจากกรุงเทพฯ'
{
  "error" : "ElasticsearchIllegalArgumentException[failed to find global 
tokenizer under [icu_tokenizer]]",
  "status" : 400
}

Please help, What I am doing wrong ?

thanks

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8d3d4059-477a-4bfc-9f8e-cddcf3b285d7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch is not rebalancing

2014-11-03 Thread Bernd Fehling

I don't understand what you are trying to tell me but to sum it up, a 
rebalancing of primary nodes with ES is not possible.
-1 for ES.

So I have to stick with SOLR which can rebalance leaders "
/admin/collections?action=REBALANCELEADERS"

Regards
Bernd

Am Montag, 3. November 2014 13:43:11 UTC+1 schrieb Jörg Prante:
>
> I think the write load issue has not a relation to primary/replica shard 
> setting. All writes should go to all shards of an index.
>
> If you have a shard count that is not dividable by the count of nodes, 
> write load could be skewed, so one or more nodes have more work than 
> others. 
>
> This can be fixed at index creation time by setting a shard number so that 
> each node carries the same number of shards for an index.
>
> Best,
>
> Jörg
>
> On Mon, Nov 3, 2014 at 1:32 PM, Bernd Fehling  > wrote:
>
>> Exactly what I see, that one server which has now 2 primaries has higher 
>> load because of two primaries writing while bulk loading.
>> And this is the cause why I want to have a rebalance and the primary back 
>> to its origin to have a well distributed load.
>> So no chance to realize this with ES?
>>
>> But if the index is generated I get a well distributed system.
>> Or is this just luck and generating the index several times I get always 
>> a different result?
>>
>> Regards,
>> Bernd
>>
>> Am Montag, 3. November 2014 13:09:53 UTC+1 schrieb Jörg Prante:
>>>
>>> There is nothing wrong, this is expected behavior.
>>>
>>> It does not matter where the primaries are. Primaries and replica have 
>>> exactly the same information about cluster state and do exactly the same 
>>> amount of work. The only exception is that primary shards do the write 
>>> operation first before sending them to replicas.
>>>
>>> So there is no need to "rebalance".
>>>
>>> Best,
>>>
>>> Jörg
>>>
>>> On Mon, Nov 3, 2014 at 1:05 PM, Bernd Fehling  
>>> wrote:
>>>
 I have an index with 4 shards and 1 replica so that each server has 1 
 primary and 1 replica.
 If the system is set up everything is well distributed.
 If 1 server goes down the primary is moved to one of the replicas but 
 if the server comes up 
 again the system is not rebalanced. The primary says where it is and 
 does not move
 back to its origin server.
 Any idea whats wrong, is there a setting for this?

 Regards
 Bernd

  -- 
 You received this message because you are subscribed to the Google 
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/e2bb0dad-8540-4f34-87a1-253f984c2d22%
 40googlegroups.com 
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/36b7cba2-7d2c-49c0-8323-944961289f10%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4a5ba3bd-7a65-4bb4-a054-e573d587b955%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch is not rebalancing

I think the write load issue has not a relation to primary/replica shard
setting. All writes should go to all shards of an index.

If you have a shard count that is not dividable by the count of nodes,
write load could be skewed, so one or more nodes have more work than
others.

This can be fixed at index creation time by setting a shard number so that
each node carries the same number of shards for an index.

Best,

Jörg

On Mon, Nov 3, 2014 at 1:32 PM, Bernd Fehling 
wrote:

> Exactly what I see, that one server which has now 2 primaries has higher
> load because of two primaries writing while bulk loading.
> And this is the cause why I want to have a rebalance and the primary back
> to its origin to have a well distributed load.
> So no chance to realize this with ES?
>
> But if the index is generated I get a well distributed system.
> Or is this just luck and generating the index several times I get always a
> different result?
>
> Regards,
> Bernd
>
> Am Montag, 3. November 2014 13:09:53 UTC+1 schrieb Jörg Prante:
>>
>> There is nothing wrong, this is expected behavior.
>>
>> It does not matter where the primaries are. Primaries and replica have
>> exactly the same information about cluster state and do exactly the same
>> amount of work. The only exception is that primary shards do the write
>> operation first before sending them to replicas.
>>
>> So there is no need to "rebalance".
>>
>> Best,
>>
>> Jörg
>>
>> On Mon, Nov 3, 2014 at 1:05 PM, Bernd Fehling 
>> wrote:
>>
>>> I have an index with 4 shards and 1 replica so that each server has 1
>>> primary and 1 replica.
>>> If the system is set up everything is well distributed.
>>> If 1 server goes down the primary is moved to one of the replicas but if
>>> the server comes up
>>> again the system is not rebalanced. The primary says where it is and
>>> does not move
>>> back to its origin server.
>>> Any idea whats wrong, is there a setting for this?
>>>
>>> Regards
>>> Bernd
>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/e2bb0dad-8540-4f34-87a1-253f984c2d22%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/36b7cba2-7d2c-49c0-8323-944961289f10%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHTHJVLmyFzzWP5R_1YchMA41wS9J%3D60ipdVgsRoXYf3w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to extract statistics for a period?

2014-11-03 Thread Alexandre

Thank you all for information.

Have a nice day.

Alex.

On 03/11/14 09:51, David Pilato wrote:

Nodes stats give you a picture of the current status. It does not
collect and store data over time.
You need to either build that buy yourself or use another tool like Marvel.

--
*David Pilato* | /Technical Advocate/ | *Elasticsearch.com
*
@dadoonet | @elasticsearchfr
|@scrutmydocs

Le 3 nov. 2014 à 09:46, Alexandre mailto:in...@opendoc.net>> a écrit :

Thank you David, but it is not possible to extract this information
between two dates?

Alex.

On 03/11/14 09:33, David Pilato wrote:

I don’t see in nodes stats that you can provide a query:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html

I think that the body is purely ignored here, right?

That said, you could look at Marvel.
http://www.elasticsearch.com/products/marvel/
Marvel collects data every x seconds and store that in elasticsearch.
Note that it’s free to use in DEV but requires a license for production
usage.

HTH

--
*David Pilato* | /Technical Advocate/ | *Elasticsearch.com
>*
@dadoonet | @elasticsearchfr
|@scrutmydocs

Le 3 nov. 2014 à 09:25, Alexandre mailto:in...@opendoc.net>
> a écrit :

Hello all, I try to check statistics of ElasticSearch for a period. I
followed this document :

http://stackoverflow.com/questions/23015308/elasticsearch-date-field-statistics

My test
---
curl -XGET
'http://X:9203/_nodes/node_elastic_01/stats?pretty=true'
-d '
{
"query":{
"match_all": {}
},
"facets":{
"histo1":{
"date_histogram":{
"field":"published_date",
"interval":"day"
}
}
}
}
' | less
---

I would to see search statistics for last 10 minutes. Do you have any
idea?

Thank you all.

Alex.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email toelasticsearch+unsubscr...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/54573C0B.1080907%40opendoc.net.
For more options, visithttps://groups.google.com/d/optout.

.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6445D292-702D-4DA1-A4F7-437C9F75E0AB%40pilato.fr
.
For more options, visithttps://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email toelasticsearch+unsubscr...@googlegroups.com
.
To view this discussion on the web
visithttps://groups.google.com/d/msgid/elasticsearch/545740F0.6060206%40opendoc.net.
For more options, visithttps://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscr...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1E04C889-FBC7-4682-A794-0B37B53B1B97%40pilato.fr
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/545777B4.6090801%40opendoc.net.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch is not rebalancing

2014-11-03 Thread Bernd Fehling

Exactly what I see, that one server which has now 2 primaries has higher 
load because of two primaries writing while bulk loading.
And this is the cause why I want to have a rebalance and the primary back 
to its origin to have a well distributed load.
So no chance to realize this with ES?

But if the index is generated I get a well distributed system.
Or is this just luck and generating the index several times I get always a 
different result?

Regards,
Bernd

Am Montag, 3. November 2014 13:09:53 UTC+1 schrieb Jörg Prante:
>
> There is nothing wrong, this is expected behavior.
>
> It does not matter where the primaries are. Primaries and replica have 
> exactly the same information about cluster state and do exactly the same 
> amount of work. The only exception is that primary shards do the write 
> operation first before sending them to replicas.
>
> So there is no need to "rebalance".
>
> Best,
>
> Jörg
>
> On Mon, Nov 3, 2014 at 1:05 PM, Bernd Fehling  > wrote:
>
>> I have an index with 4 shards and 1 replica so that each server has 1 
>> primary and 1 replica.
>> If the system is set up everything is well distributed.
>> If 1 server goes down the primary is moved to one of the replicas but if 
>> the server comes up 
>> again the system is not rebalanced. The primary says where it is and does 
>> not move
>> back to its origin server.
>> Any idea whats wrong, is there a setting for this?
>>
>> Regards
>> Bernd
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/e2bb0dad-8540-4f34-87a1-253f984c2d22%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/36b7cba2-7d2c-49c0-8323-944961289f10%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch is not rebalancing

There is nothing wrong, this is expected behavior.

It does not matter where the primaries are. Primaries and replica have
exactly the same information about cluster state and do exactly the same
amount of work. The only exception is that primary shards do the write
operation first before sending them to replicas.

So there is no need to "rebalance".

Best,

Jörg

On Mon, Nov 3, 2014 at 1:05 PM, Bernd Fehling 
wrote:

> I have an index with 4 shards and 1 replica so that each server has 1
> primary and 1 replica.
> If the system is set up everything is well distributed.
> If 1 server goes down the primary is moved to one of the replicas but if
> the server comes up
> again the system is not rebalanced. The primary says where it is and does
> not move
> back to its origin server.
> Any idea whats wrong, is there a setting for this?
>
> Regards
> Bernd
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/e2bb0dad-8540-4f34-87a1-253f984c2d22%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFsUoEbTCqCktpj1S8TyY-KvN9JE-e2Edru5LH81FByzg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

elasticsearch is not rebalancing

2014-11-03 Thread Bernd Fehling

I have an index with 4 shards and 1 replica so that each server has 1 
primary and 1 replica.
If the system is set up everything is well distributed.
If 1 server goes down the primary is moved to one of the replicas but if 
the server comes up 
again the system is not rebalanced. The primary says where it is and does 
not move
back to its origin server.
Any idea whats wrong, is there a setting for this?

Regards
Bernd

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e2bb0dad-8540-4f34-87a1-253f984c2d22%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Better to use "must" and "terms", or "should", in a bool query?

2014-11-03 Thread kazoompa

You will only see the difference when you have more complex queries that 
combine MUST and SHOULD together. In your example you could have just 
change MUST to SHOULD (i.e. used TERMS). By the definition, SHOULD is like 
an OR and without a MUST clause it implies 'at least' one should match.

R

On Friday, January 10, 2014 11:17:57 AM UTC-5, Nick Hoffman wrote:
>
> Hi guys. These 2 queries produce the same results via different 
> approaches. Is there any reason why one should be used over another?
>
> Thanks,
> Nick
>
> {
>   "size": 10,
>   "fields": ["id", "name"],
>   "sort": [ { "name.orig": "asc" } ],
>   "query": {
> "bool": {
>   "must": [
> { 
>   "terms": {
> "catalog_ids": ["4736349c302bd14e", 
> "511a70be63051f6a64000217"],
> "minimum_should_match": 1
>   }
> }
>   ]
> }
>   }
> }
>
> {
>   "size": 10,
>   "fields": ["id", "name"],
>   "sort": [ { "name.orig": "asc" } ],
>   "query": {
> "bool": {
>   "minimum_should_match": 1,
>   "should": [
> { "term": { "catalog_ids": "4736349c302bd14e" } },
> { "term": { "catalog_ids": "511a70be63051f6a64000217" } }
>   ]
> }
>   }
> }
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c8973ae2-2b7c-44d1-99f3-5c587acbdc51%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch Hadoop WRITE operation not using Reducer

2014-11-03 Thread Sarath

Hi Costin,

  Thanks for the response. You are right. Map/Reduce integration relies on 
the Input/OutputFormat. Even after removing EsOutputFormat my custom 
reducer is not invoked. Should be some issue with hadoop configuration.

Thanks,
Sarath

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7f0b88d0-fe76-41c2-8c51-398308e913bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch Hadoop WRITE operation not using Reducer

2014-11-03 Thread Costin Leau


Hi,

es-hadoop does not use either mapper or reducers; the Map/Reduce integration relies on the Input/OutputFormat which can 
be invoked either from a Mapper

or from a Reducer.

Your Reducer might not be invoked for a variety of reasons; typically the map and reduce phases have different output 
types and the job fails silently
after invoking context.write method. In fact, you can just remove the EsOutputFormat and see whether it makes any 
difference (it shouldn't).



On 11/1/14 7:20 AM, Sarath wrote:

Hi All,

   Will Elasticsearch Hadoop WRITE operation doesn't use our custom reducer? I 
tried with following code and observed
that our customer reducer is not invoked.
|
 job.setOutputFormatClass(EsOutputFormat.class);
job.setMapOutputKeyClass(NullWritable.class);
job.setMapOutputValueClass(BytesWritable.class);
job.setMapperClass(MyMapper.class);
job.setReducerClass(MyReducer.class);

 configuration.set("es.nodes", "master:9200");
configuration.set("es.resource.write","{indexName}/{indexType}");
configuration.set("es.input.json", "yes");
//configuration.set("es.write.operation", "upsert");
|



Thanks,
Sarath

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to
elasticsearch+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6607380d-48f1-4ba3-9b06-06de0ab0841c%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.


--
Costin

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/54575ED6.2010302%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Shards UNASSIGNED and OOM in logs

2014-11-03 Thread Matteo Cremolini

Hi,

We are using Elasticsearch 1.3.2 on a 2 nodes cluster in a production 
environment.
Currently only a few indexes contain about 1000 documents.
Mapper-attachments plugin is used in model.
Some shards in one cluster node (SERVER 1) are in the following states:

{
state: INITIALIZING
primary: false
node: KkuMLz0_TKONN77uOoWE7A
relocating_node: null
shard: 3
index: programma_mare-trashcan

}


{
state: UNASSIGNED
primary: false
node: null
relocating_node: null
shard: 0
index: programma_mare_index_initial

}

There are a lot of warning messages like the following:

SERVER 1:

org.elasticsearch.transport.RemoteTransportException: 
[inl-cdcl-ind2][inet[/10.73.193.51:9300]][index/shard/recovery/startRecovery]
Caused by: org.elasticsearch.index.engine.RecoveryEngineException: 
[context][1] Phase[1] Execution failed
at 
org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1078)
at 
org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636)
at 
org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135)
at 
org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72)
at 
org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440)
at 
org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: 
org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: 
[context][1] Failed to transfer [0] files with total size of [0b]
at 
org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280)
at 
org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1074)
... 9 more
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
at sun.nio.ch.IOUtil.read(IOUtil.java:195)
at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:700)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:685)
at 
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:176)
at 
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
at 
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at 
org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:96)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:346)
at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:457)
at 
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:907)
at 
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:753)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:453)
at 
org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:98)
at 
org.elasticsearch.index.store.Store.readLastCommittedSegmentsInfo(Store.java:124)
at org.elasticsearch.index.store.Store.access$300(Store.java:74)
at 
org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:442)
at 
org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:433)
at org.elasticsearch.index.store.Store.getMetadata(Store.java:144)
at 
org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145)
... 10 more



SERVER 2:


[2014-11-02 00:02:12,250][WARN ][cluster.action.shard ] [inl-cdcl-ind2] 
[context][1] 
received shard failed for [context][1], node[KkuMLz0_TKONN77uOoWE7A], [R], 
s[INITIALIZING], indexUUID [4GkRH6dNR--kmx0FtXwcRA], reason 
[Failed to start shard, message [RecoveryFailedException[[context][1]:
 Recovery failed from 
[inl-cdcl-ind2][2eBj7ijRS82V8md2a78U-A][inl-cdcl-ind2][inet[/10.73.193.51:9300]]
 

 into 
[inl-cdcl-ind1][KkuMLz0_TKONN77uOoWE7A][inl-cdcl-ind1][inet[/10.73.193.50:9300]]];
 
nested: 
 
RemoteTransportException[[inl-cdcl-ind2][inet[/10.73.193.51:9300]][index/shard/recovery/startRecovery]];
 

 nested: RecoveryEngineException[[context][1] Phase[1] Execution failed]; 
nested: RecoverFilesRecoveryException[[context][1]
 Failed to transfer [0] files with total size of [0b]]; nested: 
OutOfMemoryError[Direct buffer memory]; ]

Re: Percolator with lookup terms filter not working?

2014-11-03 Thread Alexander Jiteg

Ok. Thanks for the clarification. 

On Monday, November 3, 2014 10:20:38 AM UTC+1, Martijn v Groningen wrote:
>
> In the case of percolator queries are indexed before actual data is 
> evaluated, which triggers the automatic addition of fields in mappings. 
> Most of the times this is ok, but there are a number of queries that 
> require the field mapping to exist before the percolator query is indexed.
>
> Because of this from 1.4.0 and onwards it is required that a field mapping 
> exists for fields used in percolator queries: 
> https://github.com/elasticsearch/elasticsearch/pull/6928
>
> I hope this explains.
>
> On 30 October 2014 13:23, Alexander Jiteg  > wrote:
>
>> Seems that If I index a document with the given type first, it works.  
>> That is probably why my second run works but not the first.
>>
>> On Thursday, October 30, 2014 9:59:49 AM UTC+1, Alexander Jiteg wrote:
>>>
>>> Hi!
>>>
>>> I'm trying to use a lookup terms filter for percolation but for some 
>>> reason I'm not getting any matches when percolating documents that should 
>>> match the registered percolator.
>>>
>>> Example: 
>>> https://gist.github.com/alexndr79/760314b8b5f49157a839#file-
>>> percolation_with_terms_lookup-txt
>>>
>>> I have noted that if I try to index the same percolator a second time 
>>> after the first percolation (that gives not matches) it seems that 
>>> following percolations will give the expected result. 
>>>
>>> I'm running ES 1.3.4. 
>>>
>>> Suggestions?
>>>
>>> /Alex
>>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/b185c84f-328f-4f4b-85c6-50a746919cb6%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Met vriendelijke groet,
>
> Martijn van Groningen
>  

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9d195f44-21b0-4376-a171-e2dc7fbce7e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

elasticsearch-hadoop

2014-11-03 Thread Pavan Kumar

hi,
  i am planning to start using elasticsearch for tweet analytics on company 
product. is it feasible "if i index and store data directly into the hdfs 
via elasticsearch, instead of storing raw data and fetching that raw data 
back to elasticsearch"?? and can we communicate and do some computation 
with that indexed data in hdfs without elasticsearch??  if i store raw data 
in hdfs and indexed data in elastic search then its like increasing my 
storage space.. i am just started with elasticsearch so my question will be 
silly.can u pls give clarity on this ??? 
  Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4d715fdb-8a39-4c16-bdf5-6597cc581418%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Percolator with lookup terms filter not working?

2014-11-03 Thread Martijn v Groningen

In the case of percolator queries are indexed before actual data is
evaluated, which triggers the automatic addition of fields in mappings.
Most of the times this is ok, but there are a number of queries that
require the field mapping to exist before the percolator query is indexed.

Because of this from 1.4.0 and onwards it is required that a field mapping
exists for fields used in percolator queries:
https://github.com/elasticsearch/elasticsearch/pull/6928

I hope this explains.

On 30 October 2014 13:23, Alexander Jiteg 
wrote:

> Seems that If I index a document with the given type first, it works.
> That is probably why my second run works but not the first.
>
> On Thursday, October 30, 2014 9:59:49 AM UTC+1, Alexander Jiteg wrote:
>>
>> Hi!
>>
>> I'm trying to use a lookup terms filter for percolation but for some
>> reason I'm not getting any matches when percolating documents that should
>> match the registered percolator.
>>
>> Example:
>> https://gist.github.com/alexndr79/760314b8b5f49157a839#file-
>> percolation_with_terms_lookup-txt
>>
>> I have noted that if I try to index the same percolator a second time
>> after the first percolation (that gives not matches) it seems that
>> following percolations will give the expected result.
>>
>> I'm running ES 1.3.4.
>>
>> Suggestions?
>>
>> /Alex
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/b185c84f-328f-4f4b-85c6-50a746919cb6%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Met vriendelijke groet,

Martijn van Groningen

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CA%2BA76Txpxivfh1bn4K6dqcHvz5qxGd_0xPOPgvDbxn-p%3Dgx%3DYA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: generating correct manifest.mf file when creating jar file for native elasticsearch plugin

Yes, I understood native scripts. They should go into a plugin. Quoting the
docs:

"Plugins are a way to enhance the basic elasticsearch functionality in a
custom manner. They range from adding custom mapping types, custom
analyzers (in a more built in fashion), native scripts, custom discovery
and more."

Jörg

On Mon, Nov 3, 2014 at 10:12 AM, Matthew Hall  wrote:

> Hi thanks for the reply! I actually made a mistake in my post :/. I meant
> native scripts rather than plugins.  I tried adding the class-path to
> Manifest.mf before but this did not work.  Thanks for your suggestions
> anyway and sorry about my typo!
>
> Matt
>
> On Sat, Nov 1, 2014 at 4:54 PM, joergpra...@gmail.com <
> joergpra...@gmail.com> wrote:
>
>> In Java, you have to add "Class-Path" to MANIFEST.MF or the class loader
>> will get confused about dependencies. It depends on the class path when you
>> start Elastisearch.
>>
>> Also, in ES, simply adding extra jars is not the preferred method for
>> extensions.
>>
>> It is supposed that you put a plugin (a zip archive with jars) into the
>> plugins folder and use es-plugin.properties instead of fiddling with jars
>> in the lib folder. These jars are treated specially at start up time, and
>> do not need "Class-Path" in MANIFEST.MF.
>>
>> See also:
>>
>>
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html
>>
>> Jörg
>>
>> On Fri, Oct 31, 2014 at 5:08 PM, Matthew Hall 
>> wrote:
>>
>>> Hi sorry but I am new to java and elasticsearch.  I have written a
>>> simple native script in intellij which compiles into classes fine.  I then
>>> use the command line jar command to create the jar from the class files.  A
>>> MANIFEST.MF file is generated but is always empty (apart from version and
>>> created by).  When I put the jar into the elastic search lib directory and
>>> add the following lines to the elasticsearch.yml
>>>
>>> script.native:
>>>   MyNativeScript.type:
>>> com.elasticsearch.locationscript.CustomScriptFactory
>>>
>>> Unfortunately I get the following errors
>>>
>>>  Initialization Failed ...
>>> 1) No implementation for
>>> java.util.Set was bound.2)
>>> NoClassSettingsException[Failed to load class setting [type] with value
>>> [com.elasticsearch.locationscript.CustomScriptFactory]]
>>>
>>> ClassNotFoundException[com.elasticsearch.locationscript.CustomScriptFactory]
>>>
>>>
>>> I have tried to compile a simple helloworld jar and I found this did not
>>> work either UNTIL I modified the auto-generated (empty) Manifest file.
>>>
>>> Any help or a point to a comprehensive tutorial covering how to create
>>> the jar correctly would be much appreciated!!
>>>
>>> ttyl
>>>
>>> Matt
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/32f808bb-941f-4639-9a7d-dd57252d959e%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/_nQm8Y4EE9o/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGP6xYB5gHpN%2BauVRby9DKwAijqR_zATesK%2Bimo5JcH%2Bw%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAOAWKWFkzCUqrdoo4iO8YqZuYscV_ex7qNzELr9H-1JruXX7Qg%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.go

Embedded Elasticsearch single node OutOfMemoryError

2014-11-03 Thread Doru Sular

Hello Elasticsearch gurus,
I have Elasticsearch running as a single embedded node inside my web 
application.
I noticed that as soon the index folder size on the disk exceed the size of 
of the heap size (4GB in my case) the application become unresponsive and 
soon I got an OutOfMemory error.
The elasticsearch.yml has default settings, nothing is change there.
Do you have any suggestions how can I configure elasticsearch to get rid of 
this problem, except the obvious solution to increase the heap size value?
Thank you very much

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/dbe12dff-7f4f-4401-91e5-d6f4b5c655d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Children aggregation (1.4.0.Beta1) Round-Robin result

2014-11-03 Thread Martijn v Groningen

I missed this email... The `children` agg relies on field data and that
grows as your data grows. Technically field data is tied to the Lucene
segments on each Lucene index (shard). As segments are created and removed
so do the field data entries. For field data if there is no more heap
available then the circuit breaker kicks and fails requests that try to
load field data that don't fit into heap anymore.

How large is your index? Did you look into stats api how much heap memory
field data is actually taking? (for example the node stats api tells this
for each node in your cluster).
Maybe something else is taking it, but that is difficult to tell without
looking into the stats api outputs or looking at a heap dump.

Martijn

On 22 October 2014 11:00, Vlad Vlaskin  wrote:

> Hi Martijn,
>
> Would you help with another question considering this topic
>
> I red that ES stores parent-child relations in a heap, could it be that
> this bug prevents some objects from being GC-ed, e.g. there is a memory
> leak?
> And what happens if there is no more heap but there are more parent-child
> relations incoming?
>
> The reason Im asking is that our cluster (8 rxlarge, etc etc) went down
> after 2 days updating paren-child relations.
> Index volume is tiny, but the number of child documents updated is huge.
>
> Thank you.
>
> Vlad
>
>
> On Tuesday, October 21, 2014 4:38:55 PM UTC+2, Martijn v Groningen wrote:
>>
>> Hi Vlad,
>>
>> I opened: https://github.com/elasticsearch/elasticsearch/pull/8180
>>
>> Many thanks for reporting this issue!
>> Besides this bug the parent/child model works well, so I recommend to
>> keep it. I don't know exactly when the next 1.4 release is released, but I
>> expect within a week or 2.
>>
>> Martijn
>>
>>
>> On 21 October 2014 16:17, Vlad Vlaskin  wrote:
>>
>>> Hi Martijn,
>>>
>>> great news, thank you!
>>>
>>> Would you recommend to keep parent-child data model and wait for a
>>> release?  (Do you have a feeling of the date?).
>>>
>>> Thank you
>>>
>>> Vlad
>>>
>>>
>>>
>>> On Tuesday, October 21, 2014 4:01:47 PM UTC+2, Martijn v Groningen wrote:

 Hi Vlad,

 I reproduced it. The children agg doesn't take documents marked as
 deleted into account properly.

 When documents are deleted they are initially marked as deleted before
 they're removed from the index. This also applies to updates, because that
 translate into an index + delete.

 The issue you're experiencing can also happen when not using the bulk
 api. It may just be a bit less likely to manifest.

 The fix for this bug is small. I'll open a PR soon.

 Martijn

 On 21 October 2014 15:51, Vlad Vlaskin  wrote:

> Hi Martijn,
>
> Couple hours age I tried to submit a bug on ES Github issues and
> during creating steps of reproduce realized one more thing.
>
> *It happens only if you update the same child document within one bulk
> request.*
>
> Because I didn't manage to reproduce the "arithmetic progression"
> effect with curling my localhost, but it is still reproducible from java
> code doing bulk-update (script + upsert doc).
> I understand that bulk-updating the same document is a pretty ugly
> thing
> and I was surprised when it worked normally (without exceptions about
> version conflicts) from java client.
>
> If it might be helpful: these are the steps and queries to curl your
> localhost with parent-child.
> Unfortunately I don't know how to create a curl with bulk updates.
>
>
>  #Create index "test" with parent-cild mappings
>
>  curl -XPUT localhost:9200/test -d '{"mappings":{"root":{"propert
> ies":{"country":{"type":"string"}}},"metric":{"_parent"
> :{"type":"root"},"properties":{"count":{"type":"long"}'
>
> #Index parent document:
> curl -XPUT localhost:9200/test/root/1 -d '{"country":"de"}'
>
> #Index child document:
> curl -XPUT 'http://localhost:9200/test/metric/1?parent=1' -d
> '{"count":1}'
>  #Update child document:
> curl -XPOST 'http://localhost:9200/test/metric/1/_update?parent=1' -d
> '{"script":"ctx._source.count+=ct", "params":{"ct":1}}'
> #Query with benchmark query, it should return 2
> curl -XGET localhost:9200/test/_search -d '{"size":0,"query":{"match_
> all":{}},"aggs":{"requests":{"sum":{"field":"count"'
> #Query with child aggregation query, exepected 2
>  curl -XGET localhost:9200/test/metric/_search -d
> '{"size":0,"query":{"match_all":{}},"aggs":{"child":{"childr
> en":{"type":"metric"},"aggs":{"requests":{"sum":{"field":"
> count"}}'
>
>
>
> Thank you
>
> On Tuesday, October 21, 2014 3:33:35 PM UTC+2, Martijn v Groningen
> wrote:
>>
>> Hi Vlad,
>>
>> What you're describing shouldn't happen. The child docs should get
>> detached. I think this is a bug.
>> Let me verify and get ba

Re: generating correct manifest.mf file when creating jar file for native elasticsearch plugin

Hi thanks for the reply! I actually made a mistake in my post :/. I meant
native scripts rather than plugins.  I tried adding the class-path to
Manifest.mf before but this did not work.  Thanks for your suggestions
anyway and sorry about my typo!

Matt

On Sat, Nov 1, 2014 at 4:54 PM, joergpra...@gmail.com  wrote:

> In Java, you have to add "Class-Path" to MANIFEST.MF or the class loader
> will get confused about dependencies. It depends on the class path when you
> start Elastisearch.
>
> Also, in ES, simply adding extra jars is not the preferred method for
> extensions.
>
> It is supposed that you put a plugin (a zip archive with jars) into the
> plugins folder and use es-plugin.properties instead of fiddling with jars
> in the lib folder. These jars are treated specially at start up time, and
> do not need "Class-Path" in MANIFEST.MF.
>
> See also:
>
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html
>
> Jörg
>
> On Fri, Oct 31, 2014 at 5:08 PM, Matthew Hall 
> wrote:
>
>> Hi sorry but I am new to java and elasticsearch.  I have written a simple
>> native script in intellij which compiles into classes fine.  I then use the
>> command line jar command to create the jar from the class files.  A
>> MANIFEST.MF file is generated but is always empty (apart from version and
>> created by).  When I put the jar into the elastic search lib directory and
>> add the following lines to the elasticsearch.yml
>>
>> script.native:
>>   MyNativeScript.type:
>> com.elasticsearch.locationscript.CustomScriptFactory
>>
>> Unfortunately I get the following errors
>>
>>  Initialization Failed ...
>> 1) No implementation for
>> java.util.Set was bound.2)
>> NoClassSettingsException[Failed to load class setting [type] with value
>> [com.elasticsearch.locationscript.CustomScriptFactory]]
>>
>> ClassNotFoundException[com.elasticsearch.locationscript.CustomScriptFactory]
>>
>>
>> I have tried to compile a simple helloworld jar and I found this did not
>> work either UNTIL I modified the auto-generated (empty) Manifest file.
>>
>> Any help or a point to a comprehensive tutorial covering how to create
>> the jar correctly would be much appreciated!!
>>
>> ttyl
>>
>> Matt
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/32f808bb-941f-4639-9a7d-dd57252d959e%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/_nQm8Y4EE9o/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGP6xYB5gHpN%2BauVRby9DKwAijqR_zATesK%2Bimo5JcH%2Bw%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAOAWKWFkzCUqrdoo4iO8YqZuYscV_ex7qNzELr9H-1JruXX7Qg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Does doc_values supports array of docs?

2014-11-03 Thread Adrien Grand

Hi,

I believe you are trying to enable doc_values on an existing index?
Unfortunately, it cannot be added on the fly and needs to be configured at
index creation time (which means you will need to reindex if you already
have some data indexed).

On Mon, Nov 3, 2014 at 9:44 AM, Garbin Huang  wrote:

> I have a doc like this
>
> {
> title:'',
> distr_sentiment:[
> {k:"key1", v:"value1"},
> {k:"key2", v:"value2"}
> ]
> }
>
> I want to set distr_sentiment.k and distr_sentiment.v doc_values=true, But
> I got an error with this message "MergeMappingException[Merge failed with
> failures {[mapper [distr_sentiment.v] has different doc_values values,
> mapper [distr_sentiment.k] has different doc_values values]}]", How can I
> use doc_values in array of docs? Oh what can I do to reduce the memory
> useage on sorting and aggs on this fields?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/f48b8220-eaef-4c52-847f-2c292b259f88%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7zYE2CixxnQ7aaC5cPkAHSd5ympFooQL%2B98%3DTuP6Lwyw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to extract statistics for a period?

Nodes stats give you a picture of the current status. It does not collect and 
store data over time.
You need to either build that buy yourself or use another tool like Marvel. 


-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet  | @elasticsearchfr 
 | @scrutmydocs 




> Le 3 nov. 2014 à 09:46, Alexandre  a écrit :
> 
> Thank you David, but it is not possible to extract this information between 
> two dates?
> 
> Alex.
> 
> On 03/11/14 09:33, David Pilato wrote:
>> I don’t see in nodes stats that you can provide a query:
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html
>> 
>> I think that the body is purely ignored here, right?
>> 
>> That said, you could look at Marvel.
>> http://www.elasticsearch.com/products/marvel/
>> Marvel collects data every x seconds and store that in elasticsearch.
>> Note that it’s free to use in DEV but requires a license for production
>> usage.
>> 
>> HTH
>> 
>> --
>> *David Pilato* | /Technical Advocate/ | *Elasticsearch.com
>> >*
>> @dadoonet > | 
>> @elasticsearchfr
>> > 
>> |@scrutmydocs
>> >
>> 
>> 
>> 
>>> Le 3 nov. 2014 à 09:25, Alexandre >> 
>>> >> a écrit :
>>> 
>>> Hello all, I try to check statistics of ElasticSearch for a period. I
>>> followed this document :
>>> 
>>> http://stackoverflow.com/questions/23015308/elasticsearch-date-field-statistics
>>>  
>>> 
>>> 
>>> My test
>>> ---
>>> curl -XGET
>>> 'http://X:9203/_nodes/node_elastic_01/stats?pretty=true' 
>>>   -d '
>>> {
>>>   "query":{
>>>  "match_all": {}
>>>   },
>>>  "facets":{
>>>   "histo1":{
>>>  "date_histogram":{
>>>   "field":"published_date",
>>>   "interval":"day"
>>>   }
>>>   }
>>>   }
>>> }
>>> ' | less
>>> ---
>>> 
>>> I would to see search statistics for last 10 minutes. Do you have any
>>> idea?
>>> 
>>> Thank you all.
>>> 
>>> Alex.
>>> 
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com 
>>> .
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/54573C0B.1080907%40opendoc.net
>>>  
>>> .
>>> For more options, visit https://groups.google.com/d/optout 
>>> .
>> 
>> --
>> You received this message because you are subscribed to the Google
>> Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to elasticsearch+unsubscr...@googlegroups.com 
>> 
>> > >.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/6445D292-702D-4DA1-A4F7-437C9F75E0AB%40pilato.fr
>>  
>> 
>> >  
>> >.
>> For more options, visit https://groups.google.com/d/optout 
>> .
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/545740F0.6060206%40opendoc.net
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.co

Re: How to extract statistics for a period?

2014-11-03 Thread Alexandre

Thank you David, but it is not possible to extract this information
between two dates?

Alex.

On 03/11/14 09:33, David Pilato wrote:

I don’t see in nodes stats that you can provide a query:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html

I think that the body is purely ignored here, right?

HTH

--
*David Pilato* | /Technical Advocate/ | *Elasticsearch.com
*
@dadoonet | @elasticsearchfr
|@scrutmydocs

Le 3 nov. 2014 à 09:25, Alexandre mailto:in...@opendoc.net>> a écrit :

Hello all, I try to check statistics of ElasticSearch for a period. I
followed this document :

http://stackoverflow.com/questions/23015308/elasticsearch-date-field-statistics

My test
---
curl -XGET
'http://X:9203/_nodes/node_elastic_01/stats?pretty=true' -d '
{
"query":{
"match_all": {}
},
"facets":{
"histo1":{
"date_histogram":{
"field":"published_date",
"interval":"day"
}
}
}
}
' | less
---

I would to see search statistics for last 10 minutes. Do you have any
idea?

Thank you all.

Alex.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/54573C0B.1080907%40opendoc.net.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscr...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6445D292-702D-4DA1-A4F7-437C9F75E0AB%40pilato.fr
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/545740F0.6060206%40opendoc.net.
For more options, visit https://groups.google.com/d/optout.

Does doc_values supports array of docs?

2014-11-03 Thread Garbin Huang

I have a doc like this

{
title:'',
distr_sentiment:[
{k:"key1", v:"value1"},
{k:"key2", v:"value2"}
]
}

I want to set distr_sentiment.k and distr_sentiment.v doc_values=true, But 
I got an error with this message "MergeMappingException[Merge failed with 
failures {[mapper [distr_sentiment.v] has different doc_values values, 
mapper [distr_sentiment.k] has different doc_values values]}]", How can I 
use doc_values in array of docs? Oh what can I do to reduce the memory 
useage on sorting and aggs on this fields?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f48b8220-eaef-4c52-847f-2c292b259f88%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to extract statistics for a period?