Re: Enable/disable Shield in Node Client.

[Tom]

Thank you, Jay. It actually worked for me, I should have checked the 
solution from the SO thread despite authors negative claims. :)

On Tuesday, 28 April 2015 15:29:14 UTC+2, Jay Modi wrote:
>
> Hi Tom,
>
> For the nodes that you don't want to use Shield with, you should be able 
> to add the following to your node creation line:
>
> .settings(ImmutableSettings.builder().put("shield.enabled", false))
>
> -Jay
>
> On Tuesday, April 28, 2015 at 7:27:51 AM UTC-4, Tom wrote:
>>
>> Hello all,
>>
>> I am coding a Java application that will connect to various Elasticseach 
>> instances, some of them are secured by Shield, some are unsecured. I need 
>> have a choice to connect via NodeClient (only not secured instances) or 
>> TransportClient (both secured and not secured instances). This requires me 
>> to have shield as Maven dependency in my project, but in a situation when I 
>> connect via NodeClient to a not secured instance, Shield comes in a way and 
>> complains about the lack of license plugin. I would like to 
>> disable/bypass/not use Shield plugin in such a scenario - is there a 
>> property I can set?
>>
>> Example code:
>>
>> final Node node = 
>> nodeBuilder().clusterName(clusterName).client(true).node();
>> client = node.client();
>>
>> I can connect via this code to a not secured instance, but in a moment I 
>> add Shield as a Maven dependency, it starts to complain about the license.
>>
>> I found a similar thread on StackOverflow, but there is no response 
>> there: 
>> http://stackoverflow.com/questions/29744120/disabling-shield-for-intergration-tests
>>
>> Thank you in advance for advice.
>>
>> Tom
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ce3ff310-f9c1-4e27-b28b-5ce6da07d54a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Enable/disable Shield in Node Client.

[Tom]

Hello all,

I am coding a Java application that will connect to various Elasticseach 
instances, some of them are secured by Shield, some are unsecured. I need 
have a choice to connect via NodeClient (only not secured instances) or 
TransportClient (both secured and not secured instances). This requires me 
to have shield as Maven dependency in my project, but in a situation when I 
connect via NodeClient to a not secured instance, Shield comes in a way and 
complains about the lack of license plugin. I would like to 
disable/bypass/not use Shield plugin in such a scenario - is there a 
property I can set?

Example code:

final Node node = 
nodeBuilder().clusterName(clusterName).client(true).node();
client = node.client();

I can connect via this code to a not secured instance, but in a moment I 
add Shield as a Maven dependency, it starts to complain about the license.

I found a similar thread on StackOverflow, but there is no response there: 
http://stackoverflow.com/questions/29744120/disabling-shield-for-intergration-tests

Thank you in advance for advice.

Tom


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3941330e-9e85-49a0-b95f-cf7fe5c51a4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Different sort field responses from different (but identical) nodes?

[Tom Miller]

I have a 2 node cluster, and both nodes are identical (version, plugins, 
memory - everything from the node API is identical other than the IPs and 
Mac Addresses). The mappings for my indexes are also identical. My cluster 
is also in Green state.

However, when I perform a search, the sort response field (which is a date 
field) is different depending on the node the request hits.

For example, here is a result from node A (condensed for clarity):

{
"took": 189,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 6,
"max_score": null,
"hits": [
{
"_index": "egroup",
"_type": "invoice",
"_id": "6_12902",
"_score": null,
"_source": {...},
"sort": [
"P\u0002\u\u"
]
},
{
"_index": "egroup",
"_type": "invoice",
"_id": "6_10297",
"_score": null,
"_source": {...},
"sort": [
"P\u0002\u\u"
]
},
{
"_index": "egroup",
"_type": "cashBook",
"_id": "6_255",
"_score": null,
"_source": {...},
"sort": [
"20140103t00z"
]
}
]
}
}

Check out the sort field - weird!

Here is a result from Node B (which is a correct response):

{
"took": 528,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 6,
"max_score": null,
"hits": [
{
"_index": "egroup",
"_type": "invoice",
"_id": "6_12902",
"_score": null,
"_source": {...},
"sort": [
142171200
]
},
{
"_index": "egroup",
"_type": "invoice",
"_id": "6_10297",
"_score": null,
"_source": {...},
"sort": [
141143040
]
},
{
"_index": "egroup",
"_type": "cashBook",
"_id": "6_255",
"_score": null,
"_source": {...},
"sort": [
-9223372036854776000
]
}
]
}
}

Anyone have any idea what is going on?

Thanks,


Tom.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/df533bfb-bf67-4766-b3ed-f6e6cc2f89b7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Best way to get exact matches in query string searches, while also using a stemmer

[Tom Weingarten]

Hi,

In my current setup, I use a stemmer to analyze a field called summary. The 
user is able to search over the summary field using full lucene syntax, 
which we pass directly to elasticsearch through a query_string query, 
inside a filter. We always sort results based on their date field, so we 
use a constant score.

Unfortunately, our use of the stemmer has problems when users search for 
phrases in quotations. For instance, a search for "Crowd Valley" returns 
hits for the phrase "crowded valley". Ideally, I'd like to be able to 
ignore the stemmer for the portion of a query in a quoted phrase, but use 
the stemmer for all parts of the query outside of quotes.

Is such a thing in possible for ElasticSearch? If not, does anyone have any 
best practices for getting close to this behavior as possible?

Our current best idea is to use a multi_field with two analyzers - one with 
a stemmer, one without a stemmer. This would allow us to search either one 
or the other, depending on if the user has a quotation mark present in 
their query. However this would not be ideal for mixed queries such as: 
"Crowd Valley" or running clubs  [because the running clubs part would not 
be stemmed]

Many thanks,
Tom Weingarten
CTO & Co-founder, Wiser

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/88e6b645-73e5-4bca-9b99-082fcdc2c655%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Concurrency problem when automatically creating an index

[Tom]

Sorry, didn't mean to say "the same indexing operation" but multiple 
indexing operations (distinct data) on the same non-existent index.

El viernes, 9 de enero de 2015, 16:13:52 (UTC-3), Tom escribió:
>
> Well yes. We also have a cluster for the app where each node talks to the 
> elastic cluster independently. 
>
> Remember that we are not creating the index manually. Each app node issues 
> an index operation on an index that may yet not exist and we expect ES to 
> take care of the index creation on demand. Many processes may issue the 
> same indexing operation on the ES cluster "simultaneously" and only one of 
> them must succeed in triggering the index creation.
>
> Tom;
>
> El viernes, 9 de enero de 2015, 15:53:01 (UTC-3), Jörg Prante escribió:
>>
>> It seems there are more than one process trying to create the index, it 
>> that possible?
>>
>> Jörg
>>
>> On Fri, Jan 9, 2015 at 7:16 PM, Tomas Andres Rossi  
>> wrote:
>>
>>> We enlarged our cluster to 5 nodes and now the QUORUM error message 
>>> seems to have disappeared. 
>>> "failed to process cluster event (acquire index lock) within 1s" kind 
>>> of messages are still happening though.
>>>
>>> :(
>>>
>>> Tom;
>>>
>>> On Fri, Jan 9, 2015 at 3:11 PM, Tomas Andres Rossi  
>>> wrote:
>>>
>>>> We enlarged our cluster to 5 nodes and now the QUORUM error message 
>>>> seems to have disappeared. 
>>>> "failed to process cluster event (acquire index lock) within 1s" kind 
>>>> of messages are still happening though.
>>>>
>>>> :(
>>>>
>>>> Tom;
>>>>
>>>> On Fri, Jan 9, 2015 at 1:25 PM, joerg...@gmail.com  
>>>> wrote:
>>>>
>>>>> Exactly, with 3 nodes, the error will be gone.
>>>>>
>>>>> Please, always use an odd number of data nodes, in particular with 
>>>>> replica > 0, in order not to confuse ES quorum formula, and also to avoid 
>>>>> split brains with minimun_master_nodes
>>>>>
>>>>> Jörg
>>>>>
>>>>> On Fri, Jan 9, 2015 at 3:17 PM, Tom  wrote:
>>>>>
>>>>>> Also, we have another cluster (for different purposes) that has 3 
>>>>>> nodes but we didn't experience such errors with it (for this ES we 
>>>>>> create 
>>>>>> indices on a daily basis).
>>>>>>
>>>>>> El jueves, 8 de enero de 2015, 16:23:12 (UTC-3), Tom escribió:
>>>>>>
>>>>>>> 4
>>>>>>>
>>>>>>> El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió:
>>>>>>>>
>>>>>>>> How many nodes do you have in the cluster?
>>>>>>>>
>>>>>>>> Jörg
>>>>>>>>
>>>>>>>> On Thu, Jan 8, 2015 at 6:57 PM, Tom  wrote:
>>>>>>>>
>>>>>>>>> Hi, we'd been using ES for a while now. Specifically version 
>>>>>>>>> 0.90.3. A couple of months ago we decided to migrate to the latest 
>>>>>>>>> version 
>>>>>>>>> which was finally frozen to be 1.4.1. No data migration was necessary 
>>>>>>>>> because we have a redundant MongoDB, but yesterday we enabled data 
>>>>>>>>> writing 
>>>>>>>>> to the new ES cluster. All was running smoothly when we noticed that 
>>>>>>>>> at 
>>>>>>>>> o'clock times there were bursts of four or five log messages of the 
>>>>>>>>> following kinds:
>>>>>>>>>
>>>>>>>>> Error indexing None into index ind-analytics-2015.01.08. Total 
>>>>>>>>> elapsed time: 1065 ms. org.elasticsearch.cluster.metadata.
>>>>>>>>> ProcessClusterEventTimeoutException: failed to process cluster 
>>>>>>>>> event (acquire index lock) within 1s
>>>>>>>>> at org.elasticsearch.cluster.metadata.
>>>>>>>>> MetaDataCreateIndexService$1.run(MetaDataCreateIndexService.java:148) 
>>>>>>>>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
>>>>>>>>> at 
>>>>>>&

Re: Concurrency problem when automatically creating an index

[Tom]

Well yes. We also have a cluster for the app where each node talks to the 
elastic cluster independently. 

Remember that we are not creating the index manually. Each app node issues 
an index operation on an index that may yet not exist and we expect ES to 
take care of the index creation on demand. Many processes may issue the 
same indexing operation on the ES cluster "simultaneously" and only one of 
them must succeed in triggering the index creation.

Tom;

El viernes, 9 de enero de 2015, 15:53:01 (UTC-3), Jörg Prante escribió:
>
> It seems there are more than one process trying to create the index, it 
> that possible?
>
> Jörg
>
> On Fri, Jan 9, 2015 at 7:16 PM, Tomas Andres Rossi  > wrote:
>
>> We enlarged our cluster to 5 nodes and now the QUORUM error message seems 
>> to have disappeared. 
>> "failed to process cluster event (acquire index lock) within 1s" kind of 
>> messages are still happening though.
>>
>> :(
>>
>> Tom;
>>
>> On Fri, Jan 9, 2015 at 3:11 PM, Tomas Andres Rossi > > wrote:
>>
>>> We enlarged our cluster to 5 nodes and now the QUORUM error message 
>>> seems to have disappeared. 
>>> "failed to process cluster event (acquire index lock) within 1s" kind 
>>> of messages are still happening though.
>>>
>>> :(
>>>
>>> Tom;
>>>
>>> On Fri, Jan 9, 2015 at 1:25 PM, joerg...@gmail.com  <
>>> joerg...@gmail.com > wrote:
>>>
>>>> Exactly, with 3 nodes, the error will be gone.
>>>>
>>>> Please, always use an odd number of data nodes, in particular with 
>>>> replica > 0, in order not to confuse ES quorum formula, and also to avoid 
>>>> split brains with minimun_master_nodes
>>>>
>>>> Jörg
>>>>
>>>> On Fri, Jan 9, 2015 at 3:17 PM, Tom >>> > wrote:
>>>>
>>>>> Also, we have another cluster (for different purposes) that has 3 
>>>>> nodes but we didn't experience such errors with it (for this ES we create 
>>>>> indices on a daily basis).
>>>>>
>>>>> El jueves, 8 de enero de 2015, 16:23:12 (UTC-3), Tom escribió:
>>>>>
>>>>>> 4
>>>>>>
>>>>>> El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió:
>>>>>>>
>>>>>>> How many nodes do you have in the cluster?
>>>>>>>
>>>>>>> Jörg
>>>>>>>
>>>>>>> On Thu, Jan 8, 2015 at 6:57 PM, Tom  wrote:
>>>>>>>
>>>>>>>> Hi, we'd been using ES for a while now. Specifically version 
>>>>>>>> 0.90.3. A couple of months ago we decided to migrate to the latest 
>>>>>>>> version 
>>>>>>>> which was finally frozen to be 1.4.1. No data migration was necessary 
>>>>>>>> because we have a redundant MongoDB, but yesterday we enabled data 
>>>>>>>> writing 
>>>>>>>> to the new ES cluster. All was running smoothly when we noticed that 
>>>>>>>> at 
>>>>>>>> o'clock times there were bursts of four or five log messages of the 
>>>>>>>> following kinds:
>>>>>>>>
>>>>>>>> Error indexing None into index ind-analytics-2015.01.08. Total 
>>>>>>>> elapsed time: 1065 ms. org.elasticsearch.cluster.metadata.
>>>>>>>> ProcessClusterEventTimeoutException: failed to process cluster 
>>>>>>>> event (acquire index lock) within 1s
>>>>>>>> at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.
>>>>>>>> run(MetaDataCreateIndexService.java:148) ~[org.elasticsearch.
>>>>>>>> elasticsearch-1.4.1.jar:na]
>>>>>>>> at 
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>  
>>>>>>>> ~[na:1.7.0_17]
>>>>>>>> at 
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>  
>>>>>>>> ~[na:1.7.0_17]
>>>>>>>> at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0_17]
>>>>>>>>
>>>>>>>> [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy -

Re: Concurrency problem when automatically creating an index

[Tom]

Also, we have another cluster (for different purposes) that has 3 nodes but 
we didn't experience such errors with it (for this ES we create indices on 
a daily basis).

El jueves, 8 de enero de 2015, 16:23:12 (UTC-3), Tom escribió:
>
> 4
>
> El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió:
>>
>> How many nodes do you have in the cluster?
>>
>> Jörg
>>
>> On Thu, Jan 8, 2015 at 6:57 PM, Tom  wrote:
>>
>>> Hi, we'd been using ES for a while now. Specifically version 0.90.3. A 
>>> couple of months ago we decided to migrate to the latest version which was 
>>> finally frozen to be 1.4.1. No data migration was necessary because we have 
>>> a redundant MongoDB, but yesterday we enabled data writing to the new ES 
>>> cluster. All was running smoothly when we noticed that at o'clock times 
>>> there were bursts of four or five log messages of the following kinds:
>>>
>>> Error indexing None into index ind-analytics-2015.01.08. Total elapsed 
>>> time: 1065 ms. 
>>> org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: 
>>> failed to process cluster event (acquire index lock) within 1s
>>> at 
>>> org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.run(MetaDataCreateIndexService.java:148)
>>>  
>>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
>>> at 
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>  
>>> ~[na:1.7.0_17]
>>> at 
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>  
>>> ~[na:1.7.0_17]
>>> at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0_17]
>>>
>>> [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - 
>>> Scattering to failed in 1043ms 
>>> org.elasticsearch.action.UnavailableShardsException: [ind-2015.01.08.00][0] 
>>> Not enough active copies to meet write consistency of [QUORUM] (have 1, 
>>> needed 2). Timeout: [1s], request: index {[ind-2015.01.08.00][search][...]}
>>> at 
>>> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.retryBecauseUnavailable(TransportShardReplicationOperationAction.java:784)
>>>  
>>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
>>> at 
>>> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseFailureIfHaveNotEnoughActiveShardCopies(TransportShardReplicationOperationAction.java:776)
>>>  
>>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
>>> at 
>>> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:507)
>>>  
>>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
>>> at 
>>> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)
>>>  
>>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
>>> at 
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>  
>>> ~[na:1.7.0_17]
>>> at 
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>  
>>> ~[na:1.7.0_17]
>>> at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0_17]
>>>
>>> This occurs at o'clock times because we write over hour-based indices. 
>>> For example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to 
>>> ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and 
>>> so on.
>>>
>>> With 0.90.3 version of ES, automatic index creation was working 
>>> flawlessly (with no complaints) but the new version doesn't seem to handle 
>>> that feature very well. It looks like, when all those concurrent writes 
>>> competes to be the first to create the index, all but one fails. Of course 
>>> we could just create such indices manually to avoid this situation 
>>> altogether, but this would only be a workaround for a feature that 
>>> previously worked.
>>>
>>> Also, we use ES through the native Java client and the configuration for 
>>> all our indices is 
>>>
>>> settings = {
>>>   number_of_shards = 5,
>>>   number_of_replicas = 2
>>> }
>>>
>>> Any ideas?
>>>
>

Re: Concurrency problem when automatically creating an index

[Tom]

4

El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió:
>
> How many nodes do you have in the cluster?
>
> Jörg
>
> On Thu, Jan 8, 2015 at 6:57 PM, Tom > 
> wrote:
>
>> Hi, we'd been using ES for a while now. Specifically version 0.90.3. A 
>> couple of months ago we decided to migrate to the latest version which was 
>> finally frozen to be 1.4.1. No data migration was necessary because we have 
>> a redundant MongoDB, but yesterday we enabled data writing to the new ES 
>> cluster. All was running smoothly when we noticed that at o'clock times 
>> there were bursts of four or five log messages of the following kinds:
>>
>> Error indexing None into index ind-analytics-2015.01.08. Total elapsed 
>> time: 1065 ms. 
>> org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: 
>> failed to process cluster event (acquire index lock) within 1s
>> at 
>> org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.run(MetaDataCreateIndexService.java:148)
>>  
>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
>> at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>  
>> ~[na:1.7.0_17]
>> at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>  
>> ~[na:1.7.0_17]
>> at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0_17]
>>
>> [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - 
>> Scattering to failed in 1043ms 
>> org.elasticsearch.action.UnavailableShardsException: [ind-2015.01.08.00][0] 
>> Not enough active copies to meet write consistency of [QUORUM] (have 1, 
>> needed 2). Timeout: [1s], request: index {[ind-2015.01.08.00][search][...]}
>> at 
>> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.retryBecauseUnavailable(TransportShardReplicationOperationAction.java:784)
>>  
>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
>> at 
>> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseFailureIfHaveNotEnoughActiveShardCopies(TransportShardReplicationOperationAction.java:776)
>>  
>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
>> at 
>> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:507)
>>  
>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
>> at 
>> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)
>>  
>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
>> at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>  
>> ~[na:1.7.0_17]
>> at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>  
>> ~[na:1.7.0_17]
>> at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0_17]
>>
>> This occurs at o'clock times because we write over hour-based indices. 
>> For example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to 
>> ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and 
>> so on.
>>
>> With 0.90.3 version of ES, automatic index creation was working 
>> flawlessly (with no complaints) but the new version doesn't seem to handle 
>> that feature very well. It looks like, when all those concurrent writes 
>> competes to be the first to create the index, all but one fails. Of course 
>> we could just create such indices manually to avoid this situation 
>> altogether, but this would only be a workaround for a feature that 
>> previously worked.
>>
>> Also, we use ES through the native Java client and the configuration for 
>> all our indices is 
>>
>> settings = {
>>   number_of_shards = 5,
>>   number_of_replicas = 2
>> }
>>
>> Any ideas?
>>
>> Thanks in advance,
>> Tom;
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/61a15d1a-02f6-484d-8ce7-862bfe427f17%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Concurrency problem when automatically creating an index

[Tom]

Hi, we'd been using ES for a while now. Specifically version 0.90.3. A 
couple of months ago we decided to migrate to the latest version which was 
finally frozen to be 1.4.1. No data migration was necessary because we have 
a redundant MongoDB, but yesterday we enabled data writing to the new ES 
cluster. All was running smoothly when we noticed that at o'clock times 
there were bursts of four or five log messages of the following kinds:

Error indexing None into index ind-analytics-2015.01.08. Total elapsed 
time: 1065 ms. 
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: 
failed to process cluster event (acquire index lock) within 1s
at 
org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.run(MetaDataCreateIndexService.java:148)
 
~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
~[na:1.7.0_17]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
~[na:1.7.0_17]
at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0_17]

[ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - Scattering 
to failed in 1043ms org.elasticsearch.action.UnavailableShardsException: 
[ind-2015.01.08.00][0] Not enough active copies to meet write consistency 
of [QUORUM] (have 1, needed 2). Timeout: [1s], request: index 
{[ind-2015.01.08.00][search][...]}
at 
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.retryBecauseUnavailable(TransportShardReplicationOperationAction.java:784)
 
~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
at 
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseFailureIfHaveNotEnoughActiveShardCopies(TransportShardReplicationOperationAction.java:776)
 
~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
at 
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:507)
 
~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
at 
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)
 
~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
~[na:1.7.0_17]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
~[na:1.7.0_17]
at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0_17]

This occurs at o'clock times because we write over hour-based indices. For 
example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to 
ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and 
so on.

With 0.90.3 version of ES, automatic index creation was working flawlessly 
(with no complaints) but the new version doesn't seem to handle that 
feature very well. It looks like, when all those concurrent writes competes 
to be the first to create the index, all but one fails. Of course we could 
just create such indices manually to avoid this situation altogether, but 
this would only be a workaround for a feature that previously worked.

Also, we use ES through the native Java client and the configuration for 
all our indices is 

settings = {
  number_of_shards = 5,
  number_of_replicas = 2
}

Any ideas?

Thanks in advance,
Tom;

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregegation buckets count

[Tom]

Hi Rich,

perfect, that's it, thx a lot.

Cheers, Tom

Am Dienstag, 16. Dezember 2014 11:02:04 UTC+1 schrieb Rich Somerfield:
>
> Hi Tom,
>
> I think the "Cardinality" aggregation is what you want.
>
> e.g. :
>
> {
>   ...query...
> },
> "aggregations": {
>   "totalUniqueUsers": {
> "cardinality": {
>   "field": "username"
> }
>   }
> }
>
> -Rich
>
> On Tuesday, December 16, 2014 8:48:51 AM UTC, Tom wrote:
>>
>> Hi,
>>
>> is there a way to get just the count of buckets (not the count of docs, 
>> which works i know) of an aggregation without receiving the whole buckets 
>> content?
>>
>> thx, Tom
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a36ad00b-5f7e-4e36-9c75-928befb63fea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Aggregegation buckets count

[Tom]

Hi,

is there a way to get just the count of buckets (not the count of docs, 
which works i know) of an aggregation without receiving the whole buckets 
content?

thx, Tom


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/99c0776d-8059-432d-b2a7-5ce115dae9d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: AutoCompletion Suggester - Duplicate record in suggestion return

[Tom]

Hi,

i still have same problems with completion suggest duplicates of old and 
updates data using ES 1.4.x
Only way that fixed it so far was using _optimize?max_num_segments=1 which 
has performance and maybe other impacts i assume.

Is there another solution than _optimize?max_num_segments=1?

Thx
Tom


Am Montag, 28. April 2014 11:21:49 UTC+2 schrieb Kaspars Sprogis:
>
> Hi,
>
> I have exactly same problem.
> I resolved duplicates and appearance of deleted items by running 
> "_optimize?only_expunge_deletes=true" on daily basis.
>
> However i have still problem with updates. Even if data have been updated 
> for items in index, they still show old data when searching.
> Only solution i found is running: "_optimize?max_num_segments=1'".
>
> However i have quite a lot of updates and this get me worried, because of 
> Alexander's note:
>
>> if you do regular updates on that index, you should not update down to 
>> one single segment
>
>
> Alexander Reelsen, could you please advise, why we shouldn't do that or 
> does this create some permanent damage for optimization logic or does it 
> affects performance for search later?
> What should we do instead?
>
> Thank you.
>
> On Wednesday, April 23, 2014 1:52:16 AM UTC+3, Alexander Reelsen wrote:
>>
>> Hey
>>
>> if you do regular updates on that index, you should not update down to 
>> one single segment. Do you have an example, where the updates do not work 
>> as expected?
>>
>>
>> --Alex
>>
>>
>> On Tue, Apr 22, 2014 at 10:51 AM, kidkid  wrote:
>>
>>> Hey Alexander,
>>> Thanks for your reply.
>>> Currently I also manual do optimize by running optimize: 
>>> host:9200/completion_index/_optimize?max_num_segments=1 
>>> I think it's a work around solution. I would like to make it better.
>>>
>>> Sometime, I also have problem with update payload, when I change 
>>> payload, the completion is not change too.
>>>
>>>
>>> On Monday, April 21, 2014 7:26:13 PM UTC+7, Alexander Reelsen wrote:
>>>
>>>> Hey,
>>>>
>>>> the output is used to unify the search results, otherwise the input is 
>>>> used. The payload itself is just meta information.
>>>> The main reason, why you see the suggestion twice is, that even though 
>>>> a document is deleted and cannot be found anymore, the suggest data 
>>>> structures are only cleaned up during merges/optimizations. Running 
>>>> optimize should fix this.
>>>>
>>>> Makes sense?
>>>>
>>>>
>>>> --Alex
>>>>
>>>>
>>>>
>>>> On Sun, Apr 13, 2014 at 12:49 PM, kidkid  wrote:
>>>>
>>>>> I have figure out the problem.
>>>>> The main problem is I have used the same output for all input then ES 
>>>>> have been wrong in this case.
>>>>>
>>>>> I still trying to improve the performance. I am just test on 64Gb Ram 
>>>>> server (32Gb for ES 1.0.1) 24 core.
>>>>> Have only 2 record but it took me 3ms to suggest.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sunday, April 13, 2014 4:53:21 PM UTC+7, kidkid wrote:
>>>>>>
>>>>>> There are something really strange.
>>>>>> I don't know whether anyone have worked with this such feature or 
>>>>>> it's just not-stable feature.
>>>>>> If we do index same input, and different output,payload, then only 
>>>>>> one result found.
>>>>>>
>>>>>> Do anyone tell me how could I fix it ?
>>>>>>
>>>>>>
>>>>>>  -- 
>>>>> You received this message because you are subscribed to the Google 
>>>>> Groups "elasticsearch" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send 
>>>>> an email to elasticsearc...@googlegroups.com.
>>>>> To view this discussion on the web visit https://groups.google.com/d/
>>>>> msgid/elasticsearch/f6547a58-c002-4ff3-80c9-2052e1d14ddd%
>>>>> 40googlegroups.com 
>>>>> <https://groups.google.com/d/msgid/elasticsearch/f6547a58-c002-4ff3-80c9-2052e1d14ddd%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>>  -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/b3bf74ad-c729-4e78-9ebd-364c6c000819%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/elasticsearch/b3bf74ad-c729-4e78-9ebd-364c6c000819%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/bf7ccf72-d65c-4e1c-a6ec-b3a11c56f834%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: completion suggester caching

[Tom]

Answer: On the failing cluster we had accidently same node names for two 
nodes. After fixing that everything is fine.
I am wondering: Shouldn't ES come up with some kind of warning when same 
node name is used twice in a cluster or do names haven't to be unique 
actually?

Am Montag, 27. Oktober 2014 15:13:30 UTC+1 schrieb Tom:
>
> Addendum: And against a 1.3.2 cluster with 2 nodes it works as expected, 
> but not in 1.3.4  ?
>
> Am Montag, 27. Oktober 2014 15:09:10 UTC+1 schrieb Tom:
>>
>> I ran into same "caching" problem with latest ES 1.3.4.
>> I updated documents which just hold completion field to use with 
>> completion suggester, even if i deleted docs via id and they can't be found 
>> via _search endpoint anymore i still get suggests for those deleted 
>> documents.
>> I use a cluster with 2 nodes. Any suggestions?! 
>>
>> Am Freitag, 7. Februar 2014 16:04:01 UTC+1 schrieb Jorge Sanchez:
>>>
>>> Hello,
>>>
>>> I am using completion suggester for user search on my website. I 
>>> basically followed the howto from elasticsearch webpage. 
>>>
>>> Created the mapping:
>>> post /gruppu/user/_mapping
>>> {
>>>   "user" : {
>>> "properties" : {
>>> "name" : { "type" : "string" },
>>> "suggest" : { "type" : "completion",
>>>   "index_analyzer" : "simple",
>>>   "search_analyzer" : "simple",
>>>   "payloads" : true
>>> }
>>> }
>>> }
>>> }
>>>
>>> And I put something to the ES:
>>> post /gruppu/user/10001?refresh=true
>>> {
>>> "name" : "jorge",
>>> "suggest" : {
>>> "input": [ "jjorge"],
>>> "payload" : { "userid" : 10001, "grouporuser": "group" }
>>> }
>>> }'
>>>
>>> *The problem is that when I delete or update above document,  the 
>>> suggester still seems to cache the data. My question is what is the correct 
>>> way to update the documents in ES to have the suggester working correctly? *
>>>
>>> For example the above document, after updating the document and changing 
>>> the "input" , when I search for it I get the latest version: 
>>> {"_index":"gruppu","_type":"user","_id":"10001","_version":4,"exists":true,
>>>  
>>> "_source" : {
>>> "name" : "jorge",
>>> "suggest" : {
>>> "input": [ "*jorge group*"],
>>> "payload" : { "userid" : 10001, "grouporuser": "group" }
>>> }
>>> }'
>>> }
>>>
>>> When I run the suggester I get following:
>>> "user-suggest": [
>>>   {
>>>  "text": "j",
>>>  "offset": 0,
>>>  "length": 1,
>>>  "options": [
>>> {
>>>"text": "jorge group",
>>>"score": 2,
>>>"payload": {
>>>   "userid": 10001,
>>>   "grouporuser": "group"
>>>}
>>> },
>>> {
>>>"text": "*jjorge*",
>>>"score": 1,
>>>"payload": {
>>>   "userid": 10001,
>>>   "grouporuser": "group"
>>>}
>>> },
>>>
>>>
>>> So what is the correct way of updating the ES documents when using the 
>>> suggester ? 
>>>
>>> Thanks,
>>>
>>> Jorge Sanchez
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0209a805-5150-4564-b53e-88d113b14348%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: completion suggester caching

[Tom]

Addendum: And against a 1.3.2 cluster with 2 nodes it works as expected, 
but not in 1.3.4  ?

Am Montag, 27. Oktober 2014 15:09:10 UTC+1 schrieb Tom:
>
> I ran into same "caching" problem with latest ES 1.3.4.
> I updated documents which just hold completion field to use with 
> completion suggester, even if i deleted docs via id and they can't be found 
> via _search endpoint anymore i still get suggests for those deleted 
> documents.
> I use a cluster with 2 nodes. Any suggestions?! 
>
> Am Freitag, 7. Februar 2014 16:04:01 UTC+1 schrieb Jorge Sanchez:
>>
>> Hello,
>>
>> I am using completion suggester for user search on my website. I 
>> basically followed the howto from elasticsearch webpage. 
>>
>> Created the mapping:
>> post /gruppu/user/_mapping
>> {
>>   "user" : {
>> "properties" : {
>> "name" : { "type" : "string" },
>> "suggest" : { "type" : "completion",
>>   "index_analyzer" : "simple",
>>   "search_analyzer" : "simple",
>>   "payloads" : true
>> }
>> }
>> }
>> }
>>
>> And I put something to the ES:
>> post /gruppu/user/10001?refresh=true
>> {
>> "name" : "jorge",
>> "suggest" : {
>> "input": [ "jjorge"],
>> "payload" : { "userid" : 10001, "grouporuser": "group" }
>> }
>> }'
>>
>> *The problem is that when I delete or update above document,  the 
>> suggester still seems to cache the data. My question is what is the correct 
>> way to update the documents in ES to have the suggester working correctly? *
>>
>> For example the above document, after updating the document and changing 
>> the "input" , when I search for it I get the latest version: 
>> {"_index":"gruppu","_type":"user","_id":"10001","_version":4,"exists":true,
>>  
>> "_source" : {
>> "name" : "jorge",
>> "suggest" : {
>> "input": [ "*jorge group*"],
>> "payload" : { "userid" : 10001, "grouporuser": "group" }
>> }
>> }'
>> }
>>
>> When I run the suggester I get following:
>> "user-suggest": [
>>   {
>>  "text": "j",
>>  "offset": 0,
>>  "length": 1,
>>  "options": [
>> {
>>"text": "jorge group",
>>"score": 2,
>>"payload": {
>>   "userid": 10001,
>>   "grouporuser": "group"
>>}
>> },
>> {
>>"text": "*jjorge*",
>>"score": 1,
>>"payload": {
>>   "userid": 10001,
>>   "grouporuser": "group"
>>}
>> },
>>
>>
>> So what is the correct way of updating the ES documents when using the 
>> suggester ? 
>>
>> Thanks,
>>
>> Jorge Sanchez
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/54e3a018-c903-47dd-9e4a-4e420ea37964%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: completion suggester caching

[Tom]

I ran into same "caching" problem with latest ES 1.3.4.
I updated documents which just hold completion field to use with completion 
suggester, even if i deleted docs via id and they can't be found via 
_search endpoint anymore i still get suggests for those deleted documents.
I use a cluster with 2 nodes. Any suggestions?! 

Am Freitag, 7. Februar 2014 16:04:01 UTC+1 schrieb Jorge Sanchez:
>
> Hello,
>
> I am using completion suggester for user search on my website. I basically 
> followed the howto from elasticsearch webpage. 
>
> Created the mapping:
> post /gruppu/user/_mapping
> {
>   "user" : {
> "properties" : {
> "name" : { "type" : "string" },
> "suggest" : { "type" : "completion",
>   "index_analyzer" : "simple",
>   "search_analyzer" : "simple",
>   "payloads" : true
> }
> }
> }
> }
>
> And I put something to the ES:
> post /gruppu/user/10001?refresh=true
> {
> "name" : "jorge",
> "suggest" : {
> "input": [ "jjorge"],
> "payload" : { "userid" : 10001, "grouporuser": "group" }
> }
> }'
>
> *The problem is that when I delete or update above document,  the 
> suggester still seems to cache the data. My question is what is the correct 
> way to update the documents in ES to have the suggester working correctly? *
>
> For example the above document, after updating the document and changing 
> the "input" , when I search for it I get the latest version: 
> {"_index":"gruppu","_type":"user","_id":"10001","_version":4,"exists":true,
>  
> "_source" : {
> "name" : "jorge",
> "suggest" : {
> "input": [ "*jorge group*"],
> "payload" : { "userid" : 10001, "grouporuser": "group" }
> }
> }'
> }
>
> When I run the suggester I get following:
> "user-suggest": [
>   {
>  "text": "j",
>  "offset": 0,
>  "length": 1,
>  "options": [
> {
>"text": "jorge group",
>"score": 2,
>"payload": {
>   "userid": 10001,
>   "grouporuser": "group"
>}
> },
> {
>"text": "*jjorge*",
>"score": 1,
>"payload": {
>   "userid": 10001,
>   "grouporuser": "group"
>}
> },
>
>
> So what is the correct way of updating the ES documents when using the 
> suggester ? 
>
> Thanks,
>
> Jorge Sanchez
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f4bcada2-f0a4-4ea3-a2d7-5de54920fda1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch “Query string” query on Geo?

[tom]

I'm trying to do a query on Elasticsearch using the query string interface 
on a Geolocation. Specifically, I am trying to return all documents within 
a circle (point with radius). This is clearly possible using the full query 
DSL based on JSON. However, I can't find the syntax for the same search 
using query strings in the docs here:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax

Does this mean geo queries are not supported by query string queries?

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0e76efcc-9d0c-4795-822b-c03ad1c0e74c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

GeoDistance search with dynamic distance

[Tom Miller]

I have a product database, where products have a set delivery zones, 
represented in elasticsearch as an array: 

[
  {
distance: double (in miles),
location: geohash (source location of product/seller),
deliveryprice: double
  }
]

does anyone have any idea how I can filter using the distance within the 
document?

I know the co-ordinates of my users, and I like to filter out products that 
don't cannot be delivered to their location - in other words, where the 
distance "double" field is greater than the distance between their 
co-ordinates and the source co-ordinates of the product.

All the examples of goedistance filtering I've seen require the distance to 
be provided in the query - not analysed from the document itself.

I'm not sure if I can use scripting to do this - if so, how?

Any help would be hugely appreciated!

Tom.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7cc7b619-f06e-4bd4-9da0-03dbf26e3b5c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: has_parent filter acting strange

[Tom Miller]

Martin,

Wow I feel like an idiot! You're totally right!

Doh!

Thanks,

Tom.


On 2 October 2014 21:34, Martijn v Groningen 
wrote:

> A child document can point to an id of a parent document that doesn't
> exist (yet). So a child document that just has a _parent field can be
> returned if you wrap a has_parent in a must_not clause.
>
> If you try to perform this request:
> GET /[your-index]/psa/6_905
>
> then I expect no parent document to be returned.
>
> On 2 October 2014 22:14, Tom Miller  wrote:
>
>> Martin,
>>
>> I'm running this on v1.3.
>>
>> The mapping for "document" only has one parent type: "psa", so the fact
>> both documents have a parent means they must be the same type. I'm using
>> using routing, so the parents and children are on the same shard.
>>
>> I'm not sure exactly what extra info I can give? My data is huge...
>>
>> I'll try fiddling a bit more to see if I can replicate it further.
>>
>> I guess the question is, how can it return a result which clearly
>> contains a "_parent" field, when I'm doing a "must_not" "has_parent"
>> filter? That should never be possible.
>>
>> On 2 October 2014 20:54, Martijn v Groningen <
>> martijn.v.gronin...@gmail.com> wrote:
>>
>>> Can you share a reproduction of the mismatches you're experiencing? (in
>>> a gist or something like that, so that I can see on what data you're
>>> running these queries)
>>>
>>> The has_parent query in a must clause should return all child documents
>>> with a parent and a has_parent in must_not should return all child docs
>>> with no parent.
>>>
>>> Also on what ES version are you running this?
>>>
>>> On 2 October 2014 20:27, Tom Miller  wrote:
>>>
>>>> I have two documents, both (should) have a parent (different parents,
>>>> but the same type).
>>>>
>>>> However, I don't get the result I expect - especially trying to return
>>>> documents *without* a parent.
>>>>
>>>> For example:
>>>>
>>>> {
>>>>   "fields":["_parent"],
>>>>   "query": {
>>>> "filtered": {
>>>>   "query": {
>>>> "query_string": {
>>>>   "query": "kittens*"
>>>> }
>>>>   },
>>>>   "filter": {
>>>> "bool": {
>>>>   "must_not": [
>>>> {
>>>>   "has_parent": {
>>>> "parent_type": "psa",
>>>> "query": {
>>>>   "match_all": {}
>>>> }
>>>>   }
>>>> }
>>>>   ]
>>>> }
>>>>   }
>>>> }
>>>>   }
>>>> }
>>>>
>>>> I run that, and I get this back:
>>>>
>>>> {
>>>>"took": 287,
>>>>"timed_out": false,
>>>>"_shards": {
>>>>   "total": 1,
>>>>   "successful": 1,
>>>>   "failed": 0
>>>>},
>>>>"hits": {
>>>>   "total": 1,
>>>>   "max_score": 1,
>>>>   "hits": [
>>>>  {
>>>> "_index": "egroup",
>>>> "_type": "document",
>>>> "_id": "6_9104",
>>>> "_score": 1,
>>>> "fields": {
>>>>"_parent": "6_905"
>>>> }
>>>>  }
>>>>   ]
>>>>}
>>>> }
>>>>
>>>>
>>>> Clearly it DOES have a parent because it shows in the results!
>>>>
>>>> If I change the initial query from "must_not" to "must", I get the
>>>> other document in the result:
>>>>
>>>> {
>>>>"took": 75,
>>>>"timed_out": false,
>>>>"_shards": {
>>>>

Re: has_parent filter acting strange

[Tom Miller]

Martin,

I'm running this on v1.3.

The mapping for "document" only has one parent type: "psa", so the fact
both documents have a parent means they must be the same type. I'm using
using routing, so the parents and children are on the same shard.

I'm not sure exactly what extra info I can give? My data is huge...

I'll try fiddling a bit more to see if I can replicate it further.

I guess the question is, how can it return a result which clearly contains
a "_parent" field, when I'm doing a "must_not" "has_parent" filter? That
should never be possible.

On 2 October 2014 20:54, Martijn v Groningen 
wrote:

> Can you share a reproduction of the mismatches you're experiencing? (in a
> gist or something like that, so that I can see on what data you're running
> these queries)
>
> The has_parent query in a must clause should return all child documents
> with a parent and a has_parent in must_not should return all child docs
> with no parent.
>
> Also on what ES version are you running this?
>
> On 2 October 2014 20:27, Tom Miller  wrote:
>
>> I have two documents, both (should) have a parent (different parents, but
>> the same type).
>>
>> However, I don't get the result I expect - especially trying to return
>> documents *without* a parent.
>>
>> For example:
>>
>> {
>>   "fields":["_parent"],
>>   "query": {
>> "filtered": {
>>   "query": {
>> "query_string": {
>>   "query": "kittens*"
>> }
>>   },
>>   "filter": {
>> "bool": {
>>   "must_not": [
>> {
>>   "has_parent": {
>> "parent_type": "psa",
>> "query": {
>>   "match_all": {}
>> }
>>   }
>> }
>>   ]
>> }
>>   }
>> }
>>   }
>> }
>>
>> I run that, and I get this back:
>>
>> {
>>"took": 287,
>>"timed_out": false,
>>"_shards": {
>>   "total": 1,
>>   "successful": 1,
>>   "failed": 0
>>},
>>"hits": {
>>   "total": 1,
>>   "max_score": 1,
>>   "hits": [
>>  {
>> "_index": "egroup",
>> "_type": "document",
>> "_id": "6_9104",
>> "_score": 1,
>> "fields": {
>>"_parent": "6_905"
>> }
>>  }
>>   ]
>>}
>> }
>>
>>
>> Clearly it DOES have a parent because it shows in the results!
>>
>> If I change the initial query from "must_not" to "must", I get the other
>> document in the result:
>>
>> {
>>"took": 75,
>>"timed_out": false,
>>"_shards": {
>>   "total": 1,
>>   "successful": 1,
>>   "failed": 0
>>},
>>"hits": {
>>   "total": 1,
>>   "max_score": 1,
>>   "hits": [
>>  {
>> "_index": "egroup",
>> "_type": "document",
>> "_id": "6_9106",
>> "_score": 1,
>> "fields": {
>>"_parent": "6_847"
>> }
>>  }
>>   ]
>>}
>> }
>>
>> One query should return zero results, and the other should return 2.
>> However each are returning 1.
>>
>> Am I being thick, or is something amiss here - and how can I troubleshoot
>> it?
>>
>>
>> Tom.
>>
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/c8beeddf-2cfb-4af7-8836-6779baf85933%40googlegroups.com
>> 

has_parent filter acting strange

[Tom Miller]

I have two documents, both (should) have a parent (different parents, but 
the same type).

However, I don't get the result I expect - especially trying to return 
documents *without* a parent.

For example:

{
  "fields":["_parent"],
  "query": {
"filtered": {
  "query": {
"query_string": {
  "query": "kittens*"
}
  },
  "filter": {
"bool": {
  "must_not": [
{
  "has_parent": {
"parent_type": "psa",
"query": {
  "match_all": {}
}
  }
}
  ]
}
  }
}
  }
}

I run that, and I get this back:

{
   "took": 287,
   "timed_out": false,
   "_shards": {
  "total": 1,
  "successful": 1,
  "failed": 0
   },
   "hits": {
  "total": 1,
  "max_score": 1,
  "hits": [
 {
"_index": "egroup",
"_type": "document",
"_id": "6_9104",
"_score": 1,
"fields": {
   "_parent": "6_905"
}
 }
  ]
   }
}


Clearly it DOES have a parent because it shows in the results!

If I change the initial query from "must_not" to "must", I get the other 
document in the result:

{
   "took": 75,
   "timed_out": false,
   "_shards": {
  "total": 1,
  "successful": 1,
  "failed": 0
   },
   "hits": {
  "total": 1,
  "max_score": 1,
  "hits": [
 {
"_index": "egroup",
"_type": "document",
"_id": "6_9106",
"_score": 1,
"fields": {
   "_parent": "6_847"
}
 }
  ]
   }
}

One query should return zero results, and the other should return 2. 
However each are returning 1.

Am I being thick, or is something amiss here - and how can I troubleshoot 
it?


Tom.


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c8beeddf-2cfb-4af7-8836-6779baf85933%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch 1.4.0 release data?

[Tom Miller]

I'm in the same boat as Dan. Desperate for child aggregation!

Looks like the label has changed 
too: https://github.com/elasticsearch/elasticsearch/labels/v1.4.0.Beta1

Tom.


On Wednesday, September 10, 2014 6:02:27 PM UTC+1, Ivan Brusic wrote:
>
> I think this release might be their biggest one since 1.0. Lots of big 
> changes including a change in the consensus algorithm. It might take time, 
> but that is only a guess.
>
> -- 
> Ivan
>
> On Wed, Sep 10, 2014 at 2:57 AM, joerg...@gmail.com  <
> joerg...@gmail.com > wrote:
>
>> I use the Github issue tracker to watch the progress of the fabulous ES 
>> dev team
>>
>> https://github.com/elasticsearch/elasticsearch/labels/v1.4.0
>>
>> Today: 20 issues left, 4 blockers. Looks like it will still take some 
>> days.
>>
>> Jörg
>>
>>
>> On Wed, Sep 10, 2014 at 11:39 AM, Dan Tuffery > > wrote:
>>
>>> Is there are release date scheduled for ES 1.4.0? I need the child 
>>> aggregation for the project I'm working on at the moment.
>>>
>>> https://github.com/elasticsearch/elasticsearch/pull/6936
>>>
>>> Dan
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com .
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/0238c4fd-a702-4fca-8bcc-3dab6d71bc6f%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/elasticsearch/0238c4fd-a702-4fca-8bcc-3dab6d71bc6f%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGP%2Bq64F5FVAfjym9SvO6RM5dHOzuJMe7L8xFL4ekut%3Dg%40mail.gmail.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGP%2Bq64F5FVAfjym9SvO6RM5dHOzuJMe7L8xFL4ekut%3Dg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/43241323-6d39-4549-836a-a629694a2523%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Completion suggester: Problems getting suggests of middle words using recommended disabling of preserve_position_increments

[Tom]

Hi,

referencing 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.x/search-suggesters-completion.html
 
and 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.x/search-suggesters-completion.html
 
i tried to get suggests from middlewords without success.
Setup and Request:

#!/bin/bash

DOMAIN='127.0.0.1'
PORT='9200'
INDEX='music'
TYPE='song'

curl -X PUT $DOMAIN:$PORT/hotels -d '
{
  "mappings": {
"hotel" : {
  "properties" : {
"name" : { "type" : "string" },
"city" : { "type" : "string" },
"name_suggest" : {
  "type" :"completion",
  "index_analyzer" :  "stop", # also tried standard, simple ...
  "search_analyzer" : "stop", # also tried standard, simple ...

  "preserve_position_increments": false,
  "preserve_separators": false
}
  }
}
  }
}'

curl -X PUT $DOMAIN:$PORT/hotels/hotel/1 -d '
{
  "name" : "Mercure Hotel Munich",
  "city" : "Munich",
  "name_suggest" : {
"input" :  [
  "Mercure Hotel Munich",
  "Mercure Munich"
]
  }
}'

curl -X PUT $DOMAIN:$PORT/hotels/hotel/2 -d '
{
  "name" : "Hotel Monaco",
  "city" : "Munich",
  "name_suggest" : {
"input" :  [
  "Monaco Munich",
  "Hotel Monaco"
]
  }
}'

curl -X PUT $DOMAIN:$PORT/hotels/hotel/3 -d '
{
  "name" : "Courtyard by Marriot Munich City",
  "city" : "Munich",
  "name_suggest" : {
"input" :  [
  "Courtyard by Marriot Munich City",
  "Marriot Munich City"
]
  }
}'

curl -XPOST $DOMAIN:$PORT/hotels/_refresh

curl -X POST $DOMAIN:$PORT/hotels/_suggest?pretty -d '
{
  "hotels" : {
"text" : "Munich",
"completion" : {
  "field" : "name_suggest"
}
  }
}'

Response:
{
  "_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
  },
  "hotels" : [ {
"text" : "Munich",
"offset" : 0,
"length" : 6,
"options" : [ ]
  } ]
}

Any suggestions? Checked this against ES 1.3.2.

Thx in advance,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/72f73b37-6432-4b61-82b9-2b078d956d3f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Memory Explosion: Heap Dump in less than one minute

[Tom Wilson]

What exactly do I need to delete and how do I do it?

On Wednesday, July 30, 2014 5:45:03 PM UTC-7, Mark Walkom wrote:
>
> Unless you are attached to the stats you have in the marvel index for 
> today it might be easier to delete them than try to recover the unavailable 
> shards.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>  
>
> On 31 July 2014 10:36, Tom Wilson > 
> wrote:
>
>> Upping to 1GB, memory usage seems to level off at 750MB, but there's a 
>> problem in there somewhere. I'm getting a failure message, and the marvel 
>> dashboard isn't able to fetch.
>>
>>
>> C:\elasticsearch-1.1.1\bin>elasticsearch
>> Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
>> [2014-07-30 17:33:27,138][INFO ][node ] [Mondo] 
>> version[1.1.1], pid[10864], build[f1585f0/2014-04-16
>> T14:27:12Z]
>> [2014-07-30 17:33:27,139][INFO ][node ] [Mondo] 
>> initializing ...
>> [2014-07-30 17:33:27,163][INFO ][plugins  ] [Mondo] 
>> loaded [ldap-river, marvel], sites [marvel]
>> [2014-07-30 17:33:30,731][INFO ][node ] [Mondo] 
>> initialized
>> [2014-07-30 17:33:30,731][INFO ][node ] [Mondo] 
>> starting ...
>> [2014-07-30 17:33:31,027][INFO ][transport] [Mondo] 
>> bound_address {inet[/0.0.0.0:9300]}, publish_address
>>  {inet[/192.168.0.6:9300]}
>> [2014-07-30 17:33:34,202][INFO ][cluster.service  ] [Mondo] 
>> new_master [Mondo][liyNQAHAS0-8f-qDDqa5Rg][twilson-T
>> HINK][inet[/192.168.0.6:9300]], reason: zen-disco-join (elected_as_master)
>> [2014-07-30 17:33:34,239][INFO ][discovery] [Mondo] 
>> elasticsearch/liyNQAHAS0-8f-qDDqa5Rg
>>  [2014-07-30 17:33:34,600][INFO ][http ] [Mondo] 
>> bound_address {inet[/0.0.0.0:9200]}, publish_address
>>  {inet[/192.168.0.6:9200]}
>> [2014-07-30 17:33:35,799][INFO ][gateway  ] [Mondo] 
>> recovered [66] indices into cluster_state
>> [2014-07-30 17:33:35,815][INFO ][node ] [Mondo] 
>> started
>> [2014-07-30 17:33:39,823][DEBUG][action.search.type   ] [Mondo] All 
>> shards failed for phase: [query_fetch]
>> [2014-07-30 17:33:39,830][DEBUG][action.search.type   ] [Mondo] All 
>> shards failed for phase: [query_fetch]
>> [2014-07-30 17:33:39,837][DEBUG][action.search.type   ] [Mondo] All 
>> shards failed for phase: [query_fetch]
>> [2014-07-30 17:33:39,838][DEBUG][action.search.type   ] [Mondo] All 
>> shards failed for phase: [query_fetch]
>> [2014-07-30 17:33:43,973][DEBUG][action.search.type   ] [Mondo] All 
>> shards failed for phase: [query_fetch]
>> [2014-07-30 17:33:44,212][DEBUG][action.search.type   ] [Mondo] All 
>> shards failed for phase: [query_fetch]
>> [2014-07-30 17:33:44,357][DEBUG][action.search.type   ] [Mondo] All 
>> shards failed for phase: [query_fetch]
>> [2014-07-30 17:33:44,501][DEBUG][action.search.type   ] [Mondo] All 
>> shards failed for phase: [query_fetch]
>> [2014-07-30 17:33:53,294][DEBUG][action.search.type   ] [Mondo] All 
>> shards failed for phase: [query_fetch]
>> [2014-07-30 17:33:53,309][DEBUG][action.search.type   ] [Mondo] All 
>> shards failed for phase: [query_fetch]
>> [2014-07-30 17:33:53,310][DEBUG][action.search.type   ] [Mondo] All 
>> shards failed for phase: [query_fetch]
>> [2014-07-30 17:33:53,310][DEBUG][action.search.type   ] [Mondo] All 
>> shards failed for phase: [query_fetch]
>> [2014-07-30 17:34:03,281][DEBUG][action.search.type   ] [Mondo] All 
>> shards failed for phase: [query_fetch]
>> [2014-07-30 17:34:03,283][DEBUG][action.search.type   ] [Mondo] All 
>> shards failed for phase: [query_fetch]
>> [2014-07-30 17:34:03,286][DEBUG][action.search.type   ] [Mondo] All 
>> shards failed for phase: [query_fetch]
>> [2014-07-30 17:34:45,662][ERROR][marvel.agent.exporter] [Mondo] 
>> create failure (index:[.marvel-2014.07.31] type: [no
>> de_stats]): UnavailableShardsException[[.marvel-2014.07.31][0] [2] 
>> shardIt, [0] active : Timeout waiting for [1m], reque
>> st: org.elasticsearch.action.bulk.BulkShardRequest@39b65640]
>>
>>
>>
>> On Wednesday, July 30, 2014 5:30:29 PM UTC-7, Mark Walkom wrote:
>>
>>> Up that to 1GB and see if it starts.
>>> 512MB is pretty tiny, you're better off starting at 1/2GB if you can.
>>>
>>> Regards,
&g

Re: Memory Explosion: Heap Dump in less than one minute

[Tom Wilson]

Upping to 1GB, memory usage seems to level off at 750MB, but there's a 
problem in there somewhere. I'm getting a failure message, and the marvel 
dashboard isn't able to fetch.


C:\elasticsearch-1.1.1\bin>elasticsearch
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
[2014-07-30 17:33:27,138][INFO ][node ] [Mondo] 
version[1.1.1], pid[10864], build[f1585f0/2014-04-16
T14:27:12Z]
[2014-07-30 17:33:27,139][INFO ][node ] [Mondo] 
initializing ...
[2014-07-30 17:33:27,163][INFO ][plugins  ] [Mondo] loaded 
[ldap-river, marvel], sites [marvel]
[2014-07-30 17:33:30,731][INFO ][node ] [Mondo] 
initialized
[2014-07-30 17:33:30,731][INFO ][node ] [Mondo] 
starting ...
[2014-07-30 17:33:31,027][INFO ][transport] [Mondo] 
bound_address {inet[/0.0.0.0:9300]}, publish_address
 {inet[/192.168.0.6:9300]}
[2014-07-30 17:33:34,202][INFO ][cluster.service  ] [Mondo] 
new_master [Mondo][liyNQAHAS0-8f-qDDqa5Rg][twilson-T
HINK][inet[/192.168.0.6:9300]], reason: zen-disco-join (elected_as_master)
[2014-07-30 17:33:34,239][INFO ][discovery] [Mondo] 
elasticsearch/liyNQAHAS0-8f-qDDqa5Rg
[2014-07-30 17:33:34,600][INFO ][http ] [Mondo] 
bound_address {inet[/0.0.0.0:9200]}, publish_address
 {inet[/192.168.0.6:9200]}
[2014-07-30 17:33:35,799][INFO ][gateway  ] [Mondo] 
recovered [66] indices into cluster_state
[2014-07-30 17:33:35,815][INFO ][node ] [Mondo] started
[2014-07-30 17:33:39,823][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:39,830][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:39,837][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:39,838][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:43,973][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:44,212][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:44,357][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:44,501][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:53,294][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:53,309][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:53,310][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:33:53,310][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:34:03,281][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:34:03,283][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:34:03,286][DEBUG][action.search.type   ] [Mondo] All 
shards failed for phase: [query_fetch]
[2014-07-30 17:34:45,662][ERROR][marvel.agent.exporter] [Mondo] create 
failure (index:[.marvel-2014.07.31] type: [no
de_stats]): UnavailableShardsException[[.marvel-2014.07.31][0] [2] shardIt, 
[0] active : Timeout waiting for [1m], reque
st: org.elasticsearch.action.bulk.BulkShardRequest@39b65640]



On Wednesday, July 30, 2014 5:30:29 PM UTC-7, Mark Walkom wrote:
>
> Up that to 1GB and see if it starts.
> 512MB is pretty tiny, you're better off starting at 1/2GB if you can.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>  
>
> On 31 July 2014 10:28, Tom Wilson > 
> wrote:
>
>> JDK 1.7.0_51
>>
>> It has 512MB of heap, which was enough -- I've been running it like that 
>> for the past few months, and I only have two indexes and around 300-400 
>> documents. This is a development instance I'm running on my local machine. 
>> This only happened when I started it today. 
>>
>> -tom
>>
>>
>> On Wednesday, July 30, 2014 5:16:11 PM UTC-7, Mark Walkom wrote:
>>
>>> What java version? How much heap have you allocated and how much RAM on 
>>> the server?
>>>
>>> Basically you have too much data for the heap size, so increasing it 
>>> will help.
>>>  
>>> Regards,
>>> Mark Walkom
>>>
>>> Infrastructure Engineer
>>> Campaign Monitor
>>> email: ma...@campaignmonitor.com
>>> web: www.campaignmonitor.com
>>>
>>>
>>> On 31 July 2014 10:11, Tom W

Re: Memory Explosion: Heap Dump in less than one minute

[Tom Wilson]

JDK 1.7.0_51

It has 512MB of heap, which was enough -- I've been running it like that 
for the past few months, and I only have two indexes and around 300-400 
documents. This is a development instance I'm running on my local machine. 
This only happened when I started it today. 

-tom

On Wednesday, July 30, 2014 5:16:11 PM UTC-7, Mark Walkom wrote:
>
> What java version? How much heap have you allocated and how much RAM on 
> the server?
>
> Basically you have too much data for the heap size, so increasing it will 
> help.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
>
> On 31 July 2014 10:11, Tom Wilson > 
> wrote:
>
>> Help! Elasticsearch was working fine, but now it's using up all its heap 
>> space in the matter of a few minutes. I uninstalled the river and am 
>> performing no queries. How do I diagnose the problem? 2-3 minutes after 
>> starting, it runs out of heap space, and I'm not sure how to find out why.
>>
>> Here is the profile of memory usage:
>>
>>
>> <https://lh6.googleusercontent.com/-La0i_IrQBLA/U9mIyZZDYLI/Fx0/SfnYVdKvFAw/s1600/elasticsearch-memory.png>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> And  here is the console output. You can see it takes less than a minute 
>> after starting to run out of memory. This isn't even enough time to examine 
>> the indices through marvel.
>>
>> C:\elasticsearch-1.1.1\bin>elasticsearch
>> Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
>> [2014-07-30 16:59:02,579][INFO ][node ] [Texas 
>> Twister] version[1.1.1], pid[8572], build[f1585f0/201
>> 4-04-16T14:27:12Z]
>> [2014-07-30 16:59:02,580][INFO ][node ] [Texas 
>> Twister] initializing ...
>> [2014-07-30 16:59:02,600][INFO ][plugins  ] [Texas 
>> Twister] loaded [marvel], sites [marvel]
>> [2014-07-30 16:59:06,437][INFO ][node ] [Texas 
>> Twister] initialized
>> [2014-07-30 16:59:06,437][INFO ][node ] [Texas 
>> Twister] starting ...
>> [2014-07-30 16:59:06,691][INFO ][transport] [Texas 
>> Twister] bound_address {inet[/0.0.0.0:9300]}, publish
>> _address {inet[/192.168.0.6:9300]}
>> [2014-07-30 16:59:09,862][INFO ][cluster.service  ] [Texas 
>> Twister] new_master [Texas Twister][ShQRhZRFQnuZMTRCu
>> vY9XQ][twilson-THINK][inet[/192.168.0.6:9300]], reason: zen-disco-join 
>> (elected_as_master)
>> [2014-07-30 16:59:09,902][INFO ][discovery] [Texas 
>> Twister] elasticsearch/ShQRhZRFQnuZMTRCuvY9XQ
>> [2014-07-30 16:59:10,213][INFO ][http ] [Texas 
>> Twister] bound_address {inet[/0.0.0.0:9200]}, publish
>> _address {inet[/192.168.0.6:9200]}
>> [2014-07-30 16:59:11,631][INFO ][gateway  ] [Texas 
>> Twister] recovered [65] indices into cluster_state
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid8572.hprof ...
>> Heap dump file created [814218130 bytes in 14.202 secs]
>> Exception in thread "elasticsearch[Texas Twister][generic][T#2]" 
>> java.lang.OutOfMemoryError: Java heap space
>> at java.lang.Class.getDeclaredFields0(Native Method)
>> at java.lang.Class.privateGetDeclaredFields(Class.java:2397)
>> at java.lang.Class.getDeclaredFields(Class.java:1806)
>> at 
>> org.apache.lucene.util.RamUsageEstimator.shallowSizeOfInstance(RamUsageEstimator.java:388)
>> at 
>> org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.(Lucene42DocValuesProducer.java:101)
>> at 
>> org.apache.lucene.codecs.lucene42.Lucene42NormsFormat.normsProducer(Lucene42NormsFormat.java:75)
>> at 
>> org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:123)
>> at 
>> org.apache.lucene.index.SegmentReader.(SegmentReader.java:96)
>> at 
>> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:141)
>> at 
>> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:235)
>> at 
>> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:101)
>> at 
>> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:382)
>> at 
>> org.apache.lucene.index.Directory

Memory Explosion: Heap Dump in less than one minute

[Tom Wilson]

Help! Elasticsearch was working fine, but now it's using up all its heap 
space in the matter of a few minutes. I uninstalled the river and am 
performing no queries. How do I diagnose the problem? 2-3 minutes after 
starting, it runs out of heap space, and I'm not sure how to find out why.

Here is the profile of memory usage:




















And  here is the console output. You can see it takes less than a minute 
after starting to run out of memory. This isn't even enough time to examine 
the indices through marvel.

C:\elasticsearch-1.1.1\bin>elasticsearch
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
[2014-07-30 16:59:02,579][INFO ][node ] [Texas Twister] 
version[1.1.1], pid[8572], build[f1585f0/201
4-04-16T14:27:12Z]
[2014-07-30 16:59:02,580][INFO ][node ] [Texas Twister] 
initializing ...
[2014-07-30 16:59:02,600][INFO ][plugins  ] [Texas Twister] 
loaded [marvel], sites [marvel]
[2014-07-30 16:59:06,437][INFO ][node ] [Texas Twister] 
initialized
[2014-07-30 16:59:06,437][INFO ][node ] [Texas Twister] 
starting ...
[2014-07-30 16:59:06,691][INFO ][transport] [Texas Twister] 
bound_address {inet[/0.0.0.0:9300]}, publish
_address {inet[/192.168.0.6:9300]}
[2014-07-30 16:59:09,862][INFO ][cluster.service  ] [Texas Twister] 
new_master [Texas Twister][ShQRhZRFQnuZMTRCu
vY9XQ][twilson-THINK][inet[/192.168.0.6:9300]], reason: zen-disco-join 
(elected_as_master)
[2014-07-30 16:59:09,902][INFO ][discovery] [Texas Twister] 
elasticsearch/ShQRhZRFQnuZMTRCuvY9XQ
[2014-07-30 16:59:10,213][INFO ][http ] [Texas Twister] 
bound_address {inet[/0.0.0.0:9200]}, publish
_address {inet[/192.168.0.6:9200]}
[2014-07-30 16:59:11,631][INFO ][gateway  ] [Texas Twister] 
recovered [65] indices into cluster_state
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid8572.hprof ...
Heap dump file created [814218130 bytes in 14.202 secs]
Exception in thread "elasticsearch[Texas Twister][generic][T#2]" 
java.lang.OutOfMemoryError: Java heap space
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2397)
at java.lang.Class.getDeclaredFields(Class.java:1806)
at 
org.apache.lucene.util.RamUsageEstimator.shallowSizeOfInstance(RamUsageEstimator.java:388)
at 
org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.(Lucene42DocValuesProducer.java:101)
at 
org.apache.lucene.codecs.lucene42.Lucene42NormsFormat.normsProducer(Lucene42NormsFormat.java:75)
at 
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:123)
at 
org.apache.lucene.index.SegmentReader.(SegmentReader.java:96)
at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:141)
at 
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:235)
at 
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:101)
at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:382)
at 
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:111)
at 
org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)
at 
org.elasticsearch.index.engine.internal.InternalEngine.buildSearchManager(InternalEngine.java:1471)
at 
org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:279)
at 
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard
.java:699)
at 
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:205)
at 
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:197)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/21e9cc63-0c5d-4ea0-96a2-78d817b89236%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Time range filter

[Tom Miller]

https://github.com/elasticsearch/elasticsearch/issues/6785


On 8 July 2014 22:06, vineeth mohan  wrote:

> Hello Tom ,
>
> Please paste the link to the issue.
> I am seeing more of such request in the forum.
>
> Thanks
>   Vineeth
>
>
> On Wed, Jul 9, 2014 at 1:06 AM, Tom Miller  wrote:
>
>> Thanks guys - I've created a ticket in github. I'll store the time
>> separately for now as Vineeth suggested.
>>
>> Thanks again.
>>
>>
>> Tom.
>>
>>
>>
>> On 8 July 2014 19:54, Shay Banon  wrote:
>>
>>>  Aye, make sense to add a dedicated filter for this, care to open an
>>> issue?
>>>
>>> On Jul 8, 2014, at 6:06, vineeth mohan 
>>> wrote:
>>>
>>> Hello Tom ,
>>>
>>> At this point , i can think of 2 approaches -
>>>
>>>
>>>1. Store an additioanl field with just the time and not the date
>>>information. Do a normal range query here.
>>>2. Create script filters - In the filter , take the time out and
>>>check the range.
>>>
>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-script-filter.html
>>>
>>>
>>> But then this is a common use case and some elegant way to do it should
>>> exist.
>>> If not , I will create a bug.
>>>
>>> Thanks
>>> Vineeth
>>>
>>>
>>>
>>> On Tue, Jul 8, 2014 at 7:19 AM, Tom Miller 
>>> wrote:
>>>
>>>> All of the examples I can find on the web relate to date-range
>>>> filtering. What I need is a time-range filter: i,e
>>>> 19:00 - 23:30.
>>>>
>>>> So, in this example, I want all hits between 7PM and 11:30, regardless
>>>> of the day...
>>>>
>>>> I'd do this in SQL by doing "Where TIME(column) BETWEEN x and y".
>>>>
>>>> Is this possible in elasticsearch?
>>>>
>>>> My only solution thus far is to date_histogram by hour, and then filter
>>>> on the client and add them up, which is kinda lame...
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "elasticsearch" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/elasticsearch/943a4cca-ee2c-497a-840e-be39ad821a0f%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/elasticsearch/943a4cca-ee2c-497a-840e-be39ad821a0f%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/CAGdPd5kFZMt1nuUfBJkwPuFkBCGN4ZUHXESxPn6Ccy9F0QL5xA%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/elasticsearch/CAGdPd5kFZMt1nuUfBJkwPuFkBCGN4ZUHXESxPn6Ccy9F0QL5xA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>  --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "elasticsearch" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/elasticsearch/KQiqJ-ZiIlw/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/D18BC2AF-20BF-4CC1-94B7-B189E9193607%40gmail.com
>>> <https://groups.google.com/d/msgid/elasticsearch/D18BC2AF-20BF-4CC1-94B7-B189E9193607%40gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>>
>> Tom Miller
>>
>> Managing Director
>>
>>
>> ===

Re: Time range filter

[Tom Miller]

Thanks guys - I've created a ticket in github. I'll store the time
separately for now as Vineeth suggested.

Thanks again.


Tom.



On 8 July 2014 19:54, Shay Banon  wrote:

> Aye, make sense to add a dedicated filter for this, care to open an issue?
>
> On Jul 8, 2014, at 6:06, vineeth mohan  wrote:
>
> Hello Tom ,
>
> At this point , i can think of 2 approaches -
>
>
>1. Store an additioanl field with just the time and not the date
>information. Do a normal range query here.
>2. Create script filters - In the filter , take the time out and check
>the range.
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-script-filter.html
>
>
> But then this is a common use case and some elegant way to do it should
> exist.
> If not , I will create a bug.
>
> Thanks
> Vineeth
>
>
>
> On Tue, Jul 8, 2014 at 7:19 AM, Tom Miller  wrote:
>
>> All of the examples I can find on the web relate to date-range filtering.
>> What I need is a time-range filter: i,e
>> 19:00 - 23:30.
>>
>> So, in this example, I want all hits between 7PM and 11:30, regardless of
>> the day...
>>
>> I'd do this in SQL by doing "Where TIME(column) BETWEEN x and y".
>>
>> Is this possible in elasticsearch?
>>
>> My only solution thus far is to date_histogram by hour, and then filter
>> on the client and add them up, which is kinda lame...
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/943a4cca-ee2c-497a-840e-be39ad821a0f%40googlegroups.com
>> <https://groups.google.com/d/msgid/elasticsearch/943a4cca-ee2c-497a-840e-be39ad821a0f%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAGdPd5kFZMt1nuUfBJkwPuFkBCGN4ZUHXESxPn6Ccy9F0QL5xA%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAGdPd5kFZMt1nuUfBJkwPuFkBCGN4ZUHXESxPn6Ccy9F0QL5xA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/KQiqJ-ZiIlw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/D18BC2AF-20BF-4CC1-94B7-B189E9193607%40gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/D18BC2AF-20BF-4CC1-94B7-B189E9193607%40gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 

Tom Miller

Managing Director


===



eBiz

Centurion House, London Road, Staines, TW18 4AX

Tel: 08448 045046

email: tom.miller@ebiz. uk  |
web: ebiz.uk | www.facebook.com/eBizUK | www.twitter.com/eBizUK





Information contained in this communication may be confidential and/or
legally privileged. It is intended solely for the use of the addressee and
others authorized to receive it. If you are not the intended recipient any
disclosure, copying, distribution or action taken in reliance on its
contents is prohibited and may be unlawful. Neither eBiz | interactive
business solutions nor the author accepts legal liability for the contents
of this message. If you receive this communication in error please advise
us at priv...@ebiz.co.uk



Whilst we run anti-virus software we are not liable for any loss/damage
sustained as a result of software viruses. The recipients are advised to
run their own anti-virus software

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAM9sqB25Y3o86_y8N7eGo8tBpBaub9kY9zjwZOdozymfLP1qqw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Time range filter

[Tom Miller]

All of the examples I can find on the web relate to date-range filtering. 
What I need is a time-range filter: i,e
19:00 - 23:30.

So, in this example, I want all hits between 7PM and 11:30, regardless of 
the day...

I'd do this in SQL by doing "Where TIME(column) BETWEEN x and y".

Is this possible in elasticsearch?

My only solution thus far is to date_histogram by hour, and then filter on 
the client and add them up, which is kinda lame...

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/943a4cca-ee2c-497a-840e-be39ad821a0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How to retrieve just certain amount of docs from a larger query?

[Tom]

Hi,

i need to fire a query against larger amount of docs (100.000)
I just need to get back _id under a certain soerting and therefore using 
_source filter false, so there is no overhead by gettings _source data.
Nevertheless the response and response processing of course takes a while 
caused by the large amount.

Question: Is there a way to to do something like "execute the query, but 
just give me the 10 docs before and 10 docs after a certain doc identified 
by id" ?

Thx in advance,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/086989ae-5568-4d3c-9c09-a449f4d7a7ce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to get just certain fields on query time?

[Tom]

Perfect, thx guys !!

Am Samstag, 24. Mai 2014 20:57:49 UTC+2 schrieb Nikolas Everett:
>
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-source-filtering.html
>
>
> On Sat, May 24, 2014 at 8:54 PM, Tom  >wrote:
>
>> Hi,
>>
>> is there a way to get just parts of _source on query time?
>>
>> Thx
>> Tom
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/7f9524bf-5899-4857-aa86-fce7f987fc38%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/7f9524bf-5899-4857-aa86-fce7f987fc38%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/83c8188a-e285-46ce-96ca-b83fc4becc55%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How to get just certain fields on query time?

[Tom]

Hi,

is there a way to get just parts of _source on query time?

Thx
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7f9524bf-5899-4857-aa86-fce7f987fc38%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Update on synonyms file requires full index recreation?

[Tom]

Read about file wathcer service 
here https://github.com/elasticsearch/elasticsearch/issues/4063

Anything happening according this or am i monitoring the wrong path?

Am Freitag, 16. Mai 2014 13:42:19 UTC+2 schrieb Tom:
>
> Hi,
>
> ES 1.1.1
> when i use the synonyms filter with a synonyms_path file i have to create 
> whole index again to take new synonyms in effect.
> It is not enough to update the index data, in need to remove, create and 
> update whole index.
>
> Is that expected behaviour? If so what is ur strategy to get this done on 
> a productive system?
>
>
> Thx & regards
> Tom
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c9eb7ca7-328b-4829-b636-3a69f714f226%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Cross Fields w/ Fuzziness

[Tom]

+1 fuzziness would be great when using cross_fields

Am Mittwoch, 7. Mai 2014 22:00:25 UTC+2 schrieb Ryan Tanner:
>
> Any update to this?
>
> On Monday, April 7, 2014 7:59:54 AM UTC-6, Elliott Bradshaw wrote:
>>
>> Hi Elasticsearch,
>>
>> I've been playing with the new cross_fields multi match type, and I've 
>> got to say that I love it.  It's a great way to search complex data without 
>> doing a lot of memory killing denormalization.  That said, is there any 
>> plan to implement a fuzziness option with this type?  That would certainly 
>> be very valuable.
>>
>> - Elliott
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/119435db-2635-4866-90de-762685e22d41%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Update on synonyms file requires full index recreation?

[Tom]

Hi,

ES 1.1.1
when i use the synonyms filter with a synonyms_path file i have to create 
whole index again to take new synonyms in effect.
It is not enough to update the index data, in need to remove, create and 
update whole index.

Is that expected behaviour? If so what is ur strategy to get this done on a 
productive system?


Thx & regards
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0d8448bb-92e2-4748-a624-29fd95ea4ba1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Error installing ldap river plugin

[Tom Wilson]

I was able to install the plugin by building it from source locally and 
specifying the JAR file.

-tom


On Friday, April 18, 2014 10:50:54 AM UTC-7, Tom Wilson wrote:
>
> I'm completely new to elasticsearch and am trying to put together a 
> proof-of-concept using LDAP as a data store.
>
> However, I came across a problem right out of the starting gate, 
> attempting to install the ldap river plugin, according to the instructions 
> here:
>
> https://github.com/tlrx/elasticsearch-river-ldap
>
> I got this output. What went wrong, and how do I fix it?
>
> -tom
>
> C:\Users\twilson\Downloads\elasticsearch-1.1.1\elasticsearch-1.1.1\bin>plugin 
> -install tlrx/elasticsearch-river-ldap/0.0
> .2
> -> Installing tlrx/elasticsearch-river-ldap/0.0.2...
> Trying 
> http://download.elasticsearch.org/tlrx/elasticsearch-river-ldap/elasticsearch-river-ldap-0.0.2.zip.
> ..
> Trying 
> http://search.maven.org/remotecontent?filepath=tlrx/elasticsearch-river-ldap/0.0.2/elasticsearch-river-ldap-0.0.2
> .zip...
> Trying 
> https://oss.sonatype.org/service/local/repositories/releases/content/tlrx/elasticsearch-river-ldap/0.0.2/elastics
> earch-river-ldap-0.0.2.zip...
> Trying 
> https://github.com/tlrx/elasticsearch-river-ldap/archive/v0.0.2.zip...
> Trying 
> https://github.com/tlrx/elasticsearch-river-ldap/archive/master.zip...
> Downloading DONE
> Installed tlrx/elasticsearch-river-ldap/0.0.2 into 
> C:\Users\twilson\Downloads\elasticsearch-1.1.1\elasticsearch-1.1.1\pl
> ugins\river-ldap
> Usage:
> -u, --url [plugin location]   : Set exact URL to download the 
> plugin from
> -i, --install [plugin name]   : Downloads and installs listed 
> plugins [*]
> -t, --timeout [duration]  : Timeout setting: 30s, 1m, 1h... 
> (infinite by default)
> -r, --remove  [plugin name]   : Removes listed plugins
> -l, --list: List installed plugins
> -v, --verbose : Prints verbose messages
> -s, --silent  : Run in silent mode
> -h, --help: Prints this help message
>
>  [*] Plugin name could be:
>  elasticsearch/plugin/version for official elasticsearch plugins 
> (download from download.elasticsearch.org)
>  groupId/artifactId/version   for community plugins (download from 
> maven central or oss sonatype)
>  username/repository  for site plugins (download from github 
> master)
>
> Message:
>Error while installing plugin, reason: IllegalArgumentException: Plugin 
> installation assumed to be site plugin, but c
> ontains source code, aborting installation.
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a9b28a82-b096-4893-b9f2-6e0cd95956f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

LDAP plugin not populating

[Tom Wilson]

I'm trying to set up search of LDAP objects  using the ldap river plugin. I 
managed to install the plugin and set up my new river, but all searches are 
coming up empty. The elasticsearch stdout says:

[2014-04-18 15:00:16,904][INFO ][river.ldap   ] [Silver 
Scorpion] [ldap][hpd] now, ldap river null waiting for 1m ms

Why is my ldap river "null?" Maybe someone can look at this and tell me 
what I'm doing wrong.

I am trying to index one LDAP object (objectClass=HCProfessional), which 
resides in the container ou=HCProfessional,o=testhie,dc=hpdtest

I included a list of basic attributes, and am authenticating using the 
default admin account. Here is the REST payload I sent the server

PUT http://localhost:9200/_river/hpd/_meta
{
"type" : "ldap",
"ldap" : {
"host" : "localhost",
"port" : "10389",
"ssl"  : false,
"userDn" : "uid=admin,ou=users,ou=system",
"credentials" : "secret",
"baseDn" : "ou=HCProfessional,o=testhie,dc=hpdtest",
"filter" : "(objectClass=HCProfessional)",
"scope" : "subtree",
"attributes" : [
"uid",
"sn", 
"cn", 
"description",
"facsimileTelephoneNumber",
"gender",
"givenName",
"hcSpecialization",
"hpdMedicalRecordsDeliveryEmail",
"hpdProviderLanguageSupported",
"hpdProviderMailingAddress",
"mail",
"telephoneNumber"
],
"fields" : [
"_id",
"sn", 
"cn", 
"description",
"facsimileTelephoneNumber",
"gender",
"givenName",
"hcSpecialization",
"hpdMedicalRecordsDeliveryEmail",
"hpdProviderLanguageSupported",
"hpdProviderMailingAddress",
"mail",
"telephoneNumber"
],
"poll" : 6
},
"index" : {
"index" : "hpd",
"type" : "HCProfessional"
}
}


Now, when I send what I think is a simple search command:

GET http://localhost:9200/hpd/_search

I get back this:


   1. {
   2. "took": 1,
   3. "timed_out": false,
   4. "_shards": {
   5. "total": 5,
   6. "successful": 5,
   7. "failed": 0
   8. },
   9. "hits": {
   10. "total": 0,
   11. "max_score": null,
   12. "hits": []
   13. }
   14. }

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5b4ee277-2eee-4100-a74c-67c858d0e907%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Error installing ldap river plugin

[Tom Wilson]

I'm completely new to elasticsearch and am trying to put together a 
proof-of-concept using LDAP as a data store.

However, I came across a problem right out of the starting gate, attempting 
to install the ldap river plugin, according to the instructions here:

https://github.com/tlrx/elasticsearch-river-ldap

I got this output. What went wrong, and how do I fix it?

-tom

C:\Users\twilson\Downloads\elasticsearch-1.1.1\elasticsearch-1.1.1\bin>plugin 
-install tlrx/elasticsearch-river-ldap/0.0
.2
-> Installing tlrx/elasticsearch-river-ldap/0.0.2...
Trying 
http://download.elasticsearch.org/tlrx/elasticsearch-river-ldap/elasticsearch-river-ldap-0.0.2.zip...
Trying 
http://search.maven.org/remotecontent?filepath=tlrx/elasticsearch-river-ldap/0.0.2/elasticsearch-river-ldap-0.0.2
.zip...
Trying 
https://oss.sonatype.org/service/local/repositories/releases/content/tlrx/elasticsearch-river-ldap/0.0.2/elastics
earch-river-ldap-0.0.2.zip...
Trying 
https://github.com/tlrx/elasticsearch-river-ldap/archive/v0.0.2.zip...
Trying 
https://github.com/tlrx/elasticsearch-river-ldap/archive/master.zip...
Downloading DONE
Installed tlrx/elasticsearch-river-ldap/0.0.2 into 
C:\Users\twilson\Downloads\elasticsearch-1.1.1\elasticsearch-1.1.1\pl
ugins\river-ldap
Usage:
-u, --url [plugin location]   : Set exact URL to download the 
plugin from
-i, --install [plugin name]   : Downloads and installs listed 
plugins [*]
-t, --timeout [duration]  : Timeout setting: 30s, 1m, 1h... 
(infinite by default)
-r, --remove  [plugin name]   : Removes listed plugins
-l, --list: List installed plugins
-v, --verbose : Prints verbose messages
-s, --silent  : Run in silent mode
-h, --help: Prints this help message

 [*] Plugin name could be:
 elasticsearch/plugin/version for official elasticsearch plugins 
(download from download.elasticsearch.org)
 groupId/artifactId/version   for community plugins (download from 
maven central or oss sonatype)
 username/repository  for site plugins (download from github 
master)

Message:
   Error while installing plugin, reason: IllegalArgumentException: Plugin 
installation assumed to be site plugin, but c
ontains source code, aborting installation.



-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/187f5738-2d27-4d8c-842d-d521934a94f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Accessing non-stored fields

[Tom Verschueren]

Hi

I am new to elasticsearch and am trying out the attachement plugin. I'm a 
bit confused on how to handle the meta-data from the attachements.

I have created a simple mapping as example. I explicitly store the 'title' 
field, other fields are by default not stored.
PUT /test/file/_mapping
{
  "random" : {
  "properties": {
  "content" : { 
   "type" : "attachment",
"fields" : {
   
"title" : {
"index": "analyzed", 
"store" : "yes"
},

"content_type" : {
"store" : "no"
}
}
  }
  }
  }
}

This is the mapping as given by elasticsearch

{
   "test": {
  "mappings": {
 "file": {
"properties": {
   "content": {
  "type": "attachment",
  "path": "full",
  "fields": {
 "content": {
"type": "string"
 },
 "author": {
"type": "string"
 },
 "title": {
"type": "string",
"store": true
 },
 "name": {
"type": "string"
 },
 "date": {
"type": "date",
"format": "dateOptionalTime"
 },
 "keywords": {
"type": "string"
 },
 "content_type": {
"type": "string"
 },
 "content_length": {
"type": "integer"
 }
  }
   }
}
 }
  }
   }
}


Example query:

GET /test/file/_search
{
 "fields": [
"*", "content.content_type" 
 ],  
"query": {
"match": {   
   "content.content_type": "xhtml test document"
}
}
} 

response:
{
   "took": 13,
   "timed_out": false,
   "_shards": {
  "total": 5,
  "successful": 5,
  "failed": 0
   },
   "hits": {
  "total": 1,
  "max_score": 0.26574233,
  "hits": [
 {
"_index": "test",
"_type": "file",
"_id": "3SmwWJe6TtiP0nheD6pFCg",
"_score": 0.26574233,
"fields": {
   "content.content_type": [
  "...PCEtLQogTGljZW5zZWQgdG8gdGhlI..."
   ],
   "content.title": [
  "XHTML test document"
   ]
}
 }
  ]
   }
}


So I am able to query on the "content_type" field, but in the response I 
get the base64 representation of the attachement, instead of 
""application/xhtml+xml".
Do I really need to store each meta-data field for my attachement? I was 
under the impression that elasticsearch would extract the field from the 
_source at runtime (or would this cause to much overhead?)

Thx,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/461e4ba9-cdab-4c76-a915-c8e1f8b7ae22%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Strange issue with "ghost" percolators

[Tom F]

I'm sorry, I restarted ES and it stopped happening. I will try and
reproduce it and email again if successful.
On 6 Mar 2014 22:32, "Binh Ly"  wrote:

> This is interesting. Can you try adding a match_all filter or query to the
> percolate call and see if that makes any difference?
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/P3XsQG9PsnE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/75f70463-38e7-4830-860b-3745dc12ac65%40googlegroups.com
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAFNZOJPoaffUY8ADsoEB9wOodit6vsBx7TeU-epx6z597d-CNg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Strange issue with "ghost" percolators

[Tom F]

Hello,
I am writing an app that makes heavy use of percolators (the es 1.0 
variety). I've found that sometimes I get hits back from percolate() that 
don't actually exist anymore. In the paste below I search the logs index 
for any percolators, but find none, then run an existing document through 
percolate(). It returns 3 matches:

http://bpaste.net/show/zeRg1kMWgX7e2mbFy3bz/

I was wondering if this was expected in certain conditions? My app does 
create and delete percolators quite fast.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/92d139ef-4985-4ca4-8fec-62d29f9db560%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Elasticsearch returning blank response

[Tom Pryor]

Not tried it on a clean index but found a way around the problem. If I tell 
my client library to not use persistent (i.e keep alive) connections then 
this issue goes away.

On Monday, March 3, 2014 2:15:43 PM UTC, David Pilato wrote:
>
> Weird.
>
> Can you reproduce it on a clean index?
>
> -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
> @dadoonet <https://twitter.com/dadoonet> | 
> @elasticsearchfr<https://twitter.com/elasticsearchfr>
>
>
> Le 3 mars 2014 à 15:14:19, Tom Pryor (tomw...@gmail.com ) a 
> écrit:
>
> _source is enabled, if I fetch other documents, or fetch one of the 
> documents after update & data loss, the _source is present. 
>
> The response I'm occasionally seeing is completely blank (no json or 
> anything), just an empty string.
>
> On Monday, March 3, 2014 2:02:31 PM UTC, David Pilato wrote: 
>>
>>  Did you disable _source?
>>
>>  -- 
>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com* 
>>  @dadoonet <https://twitter.com/dadoonet> | 
>> @elasticsearchfr<https://twitter.com/elasticsearchfr>
>>  
>>
>> Le 3 mars 2014 à 14:44:58, Tom Pryor (tomw...@gmail.com) a écrit:
>>
>>  Hi, 
>>
>> Having a strange issue when trying to get a document by id, i.e GET to 
>> http://localhost:9200/index/type/
>>
>> Sometimes Elasticsearch returns a HTTP 200 but a totally empty response. 
>> The other strange thing is that the content-type of the response 
>> is text/plain; charset=UTF-8 rather than the expected JSON.
>>
>> This is causing major problems because when it occurs when I'm trying to 
>> update a document my client library thinks there is no data for the 
>> document and just indexes the document with purely the updated data, 
>> causing data loss.
>>
>> Anyone got any ideas?
>>
>> Thanks.
>>  --
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/98d31ba6-c7fb-4151-ac6f-1f273338aa5b%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>  
>>   --
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/ae6ec54c-9fc4-4a2e-b171-91586cf73cb7%40googlegroups.com
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/74493cd0-b282-4d9f-bb0e-8d223ef61416%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Elasticsearch returning blank response

[Tom Pryor]

_source is enabled, if I fetch other documents, or fetch one of the 
documents after update & data loss, the _source is present.

The response I'm occasionally seeing is completely blank (no json or 
anything), just an empty string.

On Monday, March 3, 2014 2:02:31 PM UTC, David Pilato wrote:
>
> Did you disable _source?
>
> -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
> @dadoonet <https://twitter.com/dadoonet> | 
> @elasticsearchfr<https://twitter.com/elasticsearchfr>
>
>
> Le 3 mars 2014 à 14:44:58, Tom Pryor (tomw...@gmail.com ) a 
> écrit:
>
> Hi, 
>
> Having a strange issue when trying to get a document by id, i.e GET to 
> http://localhost:9200/index/type/
>
> Sometimes Elasticsearch returns a HTTP 200 but a totally empty response. 
> The other strange thing is that the content-type of the response 
> is text/plain; charset=UTF-8 rather than the expected JSON.
>
> This is causing major problems because when it occurs when I'm trying to 
> update a document my client library thinks there is no data for the 
> document and just indexes the document with purely the updated data, 
> causing data loss.
>
> Anyone got any ideas?
>
> Thanks.
>  --
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/98d31ba6-c7fb-4151-ac6f-1f273338aa5b%40googlegroups.com
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ae6ec54c-9fc4-4a2e-b171-91586cf73cb7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Elasticsearch returning blank response

[Tom Pryor]

Hi,

Having a strange issue when trying to get a document by id, i.e GET 
to http://localhost:9200/index/type/

Sometimes Elasticsearch returns a HTTP 200 but a totally empty response. 
The other strange thing is that the content-type of the response 
is text/plain; charset=UTF-8 rather than the expected JSON.

This is causing major problems because when it occurs when I'm trying to 
update a document my client library thinks there is no data for the 
document and just indexes the document with purely the updated data, 
causing data loss.

Anyone got any ideas?

Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/98d31ba6-c7fb-4151-ac6f-1f273338aa5b%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Kibana 3 index settings

[tom rkba]

I have it working now.  Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/26448f0a-0caf-46ef-91d8-5237434cda64%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Kibana 3 index settings

[tom rkba]

Thanks!  

On Friday, February 28, 2014 1:28:59 PM UTC-5, spezam . wrote:
>
> Tom,
> as Binh wrote, those are indexes name of my three ES cluster nodes. 
> Indexes that I want to query all at once via Kibana 3.
>
>
>
>
>
> On Friday, February 28, 2014 5:16:10 PM UTC+1, Binh Ly wrote:
>>
>> Tom, that is a name mask to tell Kibana which indexes it should be 
>> looking at. That setting is accessible from your dashboard - upper right - 
>> Configure Dashboard | Index | Default Index.
>>
>> On Friday, February 28, 2014 10:22:15 AM UTC-5, tom rkba wrote:
>>>
>>>
>>> ""
>>>>
>>>> No, wildcards are not supported in the pattern, you can however supply 
>>>> multiple patterns:
>>>>
>>>> "pattern": "[logstash-1-].MM.DD,[logstash-2-].MM.DD,",
>>>>
>>>> ""
>>>>
>>>>
>>> I do not understand what you are doing here.  Is this in a querystring 
>>> field?  What exactly are you entering?
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/79c18594-3657-407f-96e5-f522b151d5cd%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Logstash type disappeared from Elasticsearch

[tom rkba]

I think this had something to do with the Marvel beta code.  ES came back 
up properly with the index once I uninstalled the marvel beta plugin and 
cleaned the ES config file.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/544e0298-41c1-4e36-a472-04ff5c352d0b%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Kibana 3 index settings

[tom rkba]

> ""
>
> No, wildcards are not supported in the pattern, you can however supply 
> multiple patterns:
>
> "pattern": "[logstash-1-].MM.DD,[logstash-2-].MM.DD,",
>
> ""
>
>
I do not understand what you are doing here.  Is this in a querystring 
field?  What exactly are you entering?

For example, I have an index called "test-2014.02.27" that contains one 
event:

_index
  _type

   _id_score   
   message
@timestamp
  @version
type
  host


test-2014.02.27
test_log
1EIFYscWTaKYltB4e_AF3g
1   
 hello
   2014-02-27T23:03:24.141Z
 1   
   test_log
   mytesthost

What query do I need to construct in Kibana to pull this information?  The 
following failed (using custom time search):

index:"test-2014.02.27" 

_index:"test-2014.02.27"

index:"test-2014.02.27" AND hello

_index:"test-2014.02.27" AND hello


















-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/75129b0f-a4ab-4644-b9f6-ddf2da3fc8d1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ElasticSearch while browsing an index that is updated every couple of minutes

[Tom K]

Good idea w/ the facets - I'll definitely be looking into that more! Thanks!

On Friday, February 7, 2014 1:54:33 PM UTC-8, Binh Ly wrote:
>
> You have sorting and paging which are very straightforward to implement in 
> ES. You can also do full text search using ES easily - for example, if I 
> type like "club snap" into the search box, you can run a single query that 
> looks for that text across multiple fields in ES easily. You can also 
> further enhance the search experience using ES facets so for example, give 
> hints to the user while he is searching, here are the top items by 
> popularity, or by price ranges, or by brand or something along those lines. 
> :)
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/571e1787-6158-4478-a64f-718f170826ce%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ElasticSearch while browsing an index that is updated every couple of minutes

[Tom K]

A number of categories (i.e. watches) has 100's of thousands of possible 
results that ultimately need to be sorted - price/brand/time. The default 
sorting will be by time. Are you saying that this type of sorting may be 
difficult/slow? Nothing bugs me more than a slow site - that's one of the 
main reasons I'm looking at ES.

On Friday, February 7, 2014 1:51:49 PM UTC-8, Nikolas Everett wrote:
>
> In past lives I was involved in integrating Solr to power all kinds of 
> funky little "shelves" on a page.  Elasticsearch can handle it.  The trick 
> is to make sure that you don't do too many slow thing against too many 
> documents.  If the shelf is built by matches and sorting (by a field or 
> relevance) you are fine.  Just don't run script scoring on tens of 
> thousands of results on every page hit.
>
> Nik
>
>
>
> On Fri, Feb 7, 2014 at 4:33 PM, Tom K >wrote:
>
>> Awesome - I could definitely see using the script for calculating the 
>> adjustments. Thanks!
>>
>> Any other ideas where ES should/should not be used on the site? 
>>
>>
>> On Friday, February 7, 2014 5:10:36 AM UTC-8, Binh Ly wrote:
>>>
>>> Tom, 
>>>
>>> You might be interested in the script_fields functionality:
>>>
>>> http://www.elasticsearch.org/guide/en/elasticsearch/
>>> reference/current/search-request-script-fields.html
>>>
>>> It allows you to introduce a dynamically computed field at query-time 
>>> where you can script the logic on how the field value is computed - for 
>>> example like doing time offset adjustments.
>>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/6d1d74e0-6341-4892-9bf4-e05bece740c9%40googlegroups.com
>> .
>>
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/34d82190-618d-4849-bb74-d804cea7c066%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ElasticSearch while browsing an index that is updated every couple of minutes

[Tom K]

Awesome - I could definitely see using the script for calculating the 
adjustments. Thanks!

Any other ideas where ES should/should not be used on the site? 

On Friday, February 7, 2014 5:10:36 AM UTC-8, Binh Ly wrote:
>
> Tom, 
>
> You might be interested in the script_fields functionality:
>
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-script-fields.html
>
> It allows you to introduce a dynamically computed field at query-time 
> where you can script the logic on how the field value is computed - for 
> example like doing time offset adjustments.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6d1d74e0-6341-4892-9bf4-e05bece740c9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

ElasticSearch while browsing an index that is updated every couple of minutes

[Tom K]

Hello,

To be clear and transparent - I know very little about ES. I would normally 
lurk around, but figure I might as well ask.

I have a project that I think would be a good fit for ES, but wondering to 
what extent. I'd like to use ES for what it is really good at vs trying to 
make it work in other situations.

The project is www.lionseek.com

One part of me thinks that ES would be best if I used it only when a 
visitor conducts a text search and thereby replacing a fulltext search via 
mysql. The search query would tap into the ES index which updates every 
minute or so and the results would be displayed.

The other part of me thinks that ES can be used for much more. Basically 
every time a page is rendered, ES is involved in one way or another 
regardless if a search term/parameter is entered.

A concern I do have (with my very very limited knowledge of ES) is how 
relative timestamps would be used. 

For a specific example:

1) let's say that Lionseek indexes content from a forum
2) it knows that the time of the posting is 9am today
3) the timezone of the forum is in San Francisco (UTC-08:00)
4) a time offset is entered
5) a relative time is shown to the visitor - i.e. 'updated 2 hours ago'

In MySQL we would do a sum() and join. We'd sum up the offset and the 
posted_time column as normalized_time.

In ES, it seems that can't be done? ES uses UTC as the timezone and that 
seems unchangeable. Maybe a new field needs to be added?

Thanks for taking the time to read through this and my ultimate question in 
the end is to what extent do you see ES being integrated into a project 
like Lionseek? I know it's asking a lot, but I'm hoping the gurus here can 
point to things they see and say... 'Well, I think ES would be an excellent 
component to help you in [this] regard or [that] regard.'

-Tom
 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/39943803-1918-4843-97ca-6c532925f47d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Logstash type disappeared from Elasticsearch

[tom rkba]

Something happened around 0700 EST today where suddenly one of our data types 
dropped off our Kibana graphs.  I am trying to figure out how to troubleshoot 
it and have some ideas.

First, redis-cli monitor shows events landing in redis.  However, the file DB 
is filling up and is 10-30 GB depending upon the server.  I am not sure how to 
see what is in the redis db.

Second, logstash does appear to be sending events and the missing type is 
configured (it worked until this morning).  

I restarted everything (redis, logstash, elasticsearch and all shippers) and 
waited for the 2014-01-30 index.  It is still not appearing.  I do think the 
events are landing in ES based upon the traffic volume, but are somehow no 
longer searchable.  How would I confirm that?

How do I determine the most recent event for that type using a curl query 
against ES?  If the last event is many hours old, how do I get ES to tell me 
why it is failing? 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/df0babe9-fc91-442c-8320-813b37f00156%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Completion suggester: How to get hits per output for matching documents?

[Tom]

Hi,

is there a way to get the count of matching documents for a certain output 
when using completion suggester?
There is nothing in the response as i can see, but a very usual requirement 
to show how many documents match a certain suggestion.

Thanks,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2460b511-2e3d-4068-addf-ef655e88abc4%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.