Re: ElasticsearchIllegalStateException: Maximum number of bytes allocated exceeded

[Alexander Reelsen]

Hey,

do you have a full stack trace in one of your log files? Also, are you
creating lots of new connections all the time and then stop them abruptly
(just a wild guess here)?


--Alex


On Mon, Jun 16, 2014 at 11:18 PM, Sriram Kannan  wrote:

> I am getting this error constantly and the 5 nodes in the cluster goes
> down one by one. Is there a way to fix the issue? Could it be a corrupt
> data issue?
>
>
> [2014-06-16 13:13:34,422][WARN ][discovery.zen.ping.multicast] [nw-es-201]
> failed to receive confirmation on sent ping response to
> [[nw-es-202][xOi0MSQBQ4OoQjpuIDUjtA][nw-es-202.prod.pcln.com
> ][inet[/172.21.88.113:9320]]{master=true}]
> org.elasticsearch.transport.SendRequestTransportException:
> [nw-es-202][inet[/172.21.88.113:9320]][discovery/zen/multicast]
> at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:219)
> at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:190)
> at
> org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.handleNodePingRequest(MulticastZenPing.java:462)
> at
> org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.onMessage(MulticastZenPing.java:334)
> at
> org.elasticsearch.common.network.MulticastChannel$Plain$Receiver.run(MulticastChannel.java:363)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.elasticsearch.ElasticsearchIllegalStateException: Maximum
> number of bytes allocated exceeded: [836334432] (> 836318003)
> at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:384)
> at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:398)
> at
> org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:423)
> at
> org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:431)
> at
> org.elasticsearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:60)
> at
> org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:37)
> at
> org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:558)
> at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:206)
> ... 5 more
> [2014-06-16 13:13:34,601][WARN ][discovery.zen.ping.multicast] [nw-es-201]
> failed to receive confirmation on sent ping response to
> [[nw-es-205][p9b19RfCT1uF7Ry2_eQXQw][nw-es-205.prod.pcln.com
> ][inet[/172.21.88.116:9320]]{master=true}]
> org.elasticsearch.transport.SendRequestTransportException:
> [nw-es-205][inet[/172.21.88.116:9320]][discovery/zen/multicast]
> at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:219)
> at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:190)
> at
> org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.handleNodePingRequest(MulticastZenPing.java:462)
> at
> org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.onMessage(MulticastZenPing.java:334)
> at
> org.elasticsearch.common.network.MulticastChannel$Plain$Receiver.run(MulticastChannel.java:363)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.elasticsearch.ElasticsearchIllegalStateException: Maximum
> number of bytes allocated exceeded: [836334432] (> 836318003)
> at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:384)
> at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:398)
> at
> org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:423)
> at
> org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:431)
> at
> org.elasticsearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:60)
> at
> org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:37)
> at
> org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:558)
> at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:206)
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/3769ecc9-1119-4f3e-8131-76d4cfef24bb%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch

Re: Creating a browse interface from ES

[Robin Sheat]

joergpra...@gmail.com schreef op ma 16-06-2014 om 13:12 [+0200]:


> This is how I implement "register search"

This is interesting. It could work for me.

Though, I'm not sure I totally understand it. To find, say "Smith", I'd
search for it, get its index, and then use the from/size stuff to bring
up the list in that area. Is that essentially what you're using?

If so, that seems like what I need. The only issue is that it'll require
a total reindex every time something is added. But, I don't see a way
around that even with some other ideas I'm exploring.

-- 
Robin Sheat
Catalyst IT Ltd.
✆ +64 4 803 2204
GPG: 5FA7 4B49 1E4D CAA4 4C38  8505 77F5 B724 F871 3BDF

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1402982936.19820.60.camel%40zarathud.wgtn.cat-it.co.nz.
For more options, visit https://groups.google.com/d/optout.

Boost field does not work..

[Felix Xu]

Hi Guys,
I have two types in a index, one is used for indexing topics and another 
one is used for indexing comments.
Here is the sample mapping:

*Topic*:

curl -XPUT http://localhost:9200/bbs/topic/_mapping -d'
{
"topic": {
"_timestamp": {
"enabled": true
},
"properties": {
"title": {
"type": "string",
"store": true,
"term_vector": "with_positions_offsets",
"indexAnalyzer": "ik",
"searchAnalyzer": "ik",
"include_in_all": true,
*"boost": 8*
},
"content": {
"type": "string",
"store": true,
"term_vector": "with_positions_offsets",
"indexAnalyzer": "ik",
"searchAnalyzer": "ik",
"include_in_all": true,
*"boost": 4*
}
}
}


*Comment*:

curl -XPUT http://localhost:9200/bbs/comment/_mapping -d'
{
"comment": {
"_timestamp": {
"enabled": true
},
"properties": {
"title": {
"type": "string",
"store": true,
"term_vector": "with_positions_offsets",
"indexAnalyzer": "ik",
"searchAnalyzer": "ik",
"include_in_all": true,
*"boost": 0*
},
"content": {
"type": "string",
"store": true,
"term_vector": "with_positions_offsets",
"indexAnalyzer": "ik",
"searchAnalyzer": "ik",
"include_in_all": true,
*"boost": 4*
}
}
}


I want to search theses two types(title,content of Topic and only content 
of Comment) at the same time, however, I do not want to match the "title" 
field of comments, since the title is the same with its corresponding 
topic, matching the title field of a comment does not make any sense.
I have tried to set the boost value of Comment's title field to zero but it 
seems does not work..
I think a simple solution is to set the "title" of Comment with 
"not_analyzed", but I also want to highlight the matching words in the 
title, so it's better to also index the title field but let it have little 
effects on scoring.. 
Could someone please give me some hints? Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a635b0ca-d669-49c1-9263-1e5a682e63f9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Boost field does not work..

[Felix Xu]

Hi Guys,
I have two types in a index, one is used for indexing topics and another 
one is used for indexing comments.
Here is the sample mapping:

*Topic*:

curl -XPUT http://localhost:9200/bbs/topic/_mapping -d'
{
"topic": {
"_timestamp": {
"enabled": true
},
"properties": {
"title": {
"type": "string",
"store": true,
"term_vector": "with_positions_offsets",
"indexAnalyzer": "ik",
"searchAnalyzer": "ik",
"include_in_all": true,
*"boost": 8*
},
"content": {
"type": "string",
"store": true,
"term_vector": "with_positions_offsets",
"indexAnalyzer": "ik",
"searchAnalyzer": "ik",
"include_in_all": true,
*"boost": 4*
}
}
}


*Comment*:

curl -XPUT http://localhost:9200/bbs/comment/_mapping -d'
{
"comment": {
"_timestamp": {
"enabled": true
},
"properties": {
"title": {
"type": "string",
"store": true,
"term_vector": "with_positions_offsets",
"indexAnalyzer": "ik",
"searchAnalyzer": "ik",
"include_in_all": true,
*"boost": 0*
},
"content": {
"type": "string",
"store": true,
"term_vector": "with_positions_offsets",
"indexAnalyzer": "ik",
"searchAnalyzer": "ik",
"include_in_all": true,
*"boost": 4*
}
}
}


I want to search theses two types(title,content of Topic and only content 
of Comment) at the same time, however, I do not want to match the "title" 
field of comments, since the title is the same with its corresponding 
topic, matching the title field of a comment does not make any sense.
I have tried to set the boost value of Comment's title field to zero but it 
seems does not work..
I think a simple solution is to set the "title" of Comment with 
"not_analyzed", but I also want to highlight the matching words in the 
title, so it's better to also index the title field but let it have little 
effects on scoring.. 
Could someone please give me some hints? Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4dac461c-03ab-4014-ab27-53b50b01e8b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Kibana chart data view understanding

[fred . grummit]

I have a problem trying to visualise the data below in Kibana.
Each document describes a test run audit entry with passing, failing and 
pending tests along with a timestamp, project identifier and host name.
The curls below setup four documents and they are correctly returned if I 
do http://localhost:9200/someaudits/_search?pretty=true

I would like to use kibana to display a single graph with:
The X axis using @timestamp
The Y axis showing four separate lines for passed, failed, pending and 
(passed + failed + pending)
Each document (and its timestamp value) should contain a tag that 
references the document itself.
Documents and their pass/fail/pending values should not be totalised, so 
they remain distinct on the graph.

However the sticking point is that I'm cannot see what to click (and in 
what order) to setup the graph view from a blank Kibana instance located at 
http://localhost:9200/_plugin/kibana/
I've read the kibana related tutorials but I'm just not groking it.



# Delete the whole index:
curl -XDELETE http://localhost:9200/someaudits

# Create the index:
curl -XPOST 'localhost:9200/someaudits/'

# Use this mapping:
curl -XPUT http://localhost:9200/someaudits/testaudit/_mapping -d '
{
  "testaudit" : {
   "properties" : {
   "@timestamp" : {"format" : "dateOptionalTime", "type" : "date" },
"project" : {"type": "string" },
"host" : {"type": "string" },
"passed" : { "type" : "integer" },
"failed" : { "type" : "integer" },
"pending" : { "type" : "integer" }
   }
  }
 }
'

# Add some data:
curl -XPUT 'http://localhost:9200/someaudits/testaudit/1' -d '
{
"@timestamp" : "2014-06-17T02:10:08.593Z",
"project" : "test",
"host" : "mymachine",
"passed" : 10,
"failed" : 20,
"pending" : 1
}'

curl -XPUT 'http://localhost:9200/someaudits/testaudit/2' -d '
{
"@timestamp" : "2014-06-17T02:15:08.593Z",
"project" : "test",
"host" : "mymachine",
"passed" : 0,
"failed" : 30,
"pending" : 0
}'

curl -XPUT 'http://localhost:9200/someaudits/testaudit/3' -d '
{
"@timestamp" : "2014-06-17T02:20:08.593Z",
"project" : "test",
"host" : "mymachine",
"passed" : 50,
"failed" : 0,
"pending" : 1
}'

curl -XPUT 'http://localhost:9200/someaudits/testaudit/4' -d '
{
"@timestamp" : "2014-06-17T02:10:18.593Z",
"project" : "another test",
"host" : "mymachine",
"passed" : 0,
"failed" : 1,
"pending" : 0
}'

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/86f13f44-868a-49b8-991d-64138c602f15%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Tribe node data compliance across regions

[Mark Walkom]

It's best explained in the docs
http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.x/modules-tribe.html

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 17 June 2014 05:24, Kyle House  wrote:

> We are staging a rollout of multiple ES clusters across multiple regions,
> and we have a requirement to keep data contained inside the predefined
> regions. I wanted to check on how a tribe node would impact this
> requirement. My understanding is this node would be attached to multiple
> clusters for quick searching across regions, but would not pull any data
> locally thus violating this data requirement. Any information would be
> helpful, thanks!
>
> --Kyle
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/f81460ac-d2db-4ae4-a014-29ed86c0a5de%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624a8NDBzjrUjeMMoSH7kfuGQ0u8V-z0Hh4vbFVKqmf_sCw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: IllegalArgumentException[No type mapped for [43]], version 1.2.1

[Brooke Babcock]

Thank you for the suggestions. A couple clarifications and additional 
information:

- We do use bulk indexing, as you inferred, using the Java TransportClient 
directly from our Java application.

- This index, however, does not use dynamic mapping. I included the mapping 
in my first post and we don't send any other fields to ES for this index. 
It acts as a type of object storage for us and we do not query against it. 
(I understand this is an atypical use case and not exactly what ES is 
designed for. But we were so impressed by the feature set of ES, we are 
trying to use it for more than just searching. Outside of this one issue, 
it has performed excellently. And, as stated, we did not see this issue in 
earlier versions.)

- We have 20 nodes in our cluster with 2 replicas. When we first 
experienced this issue, we only had 2 or 3 clients doing bulk indexing into 
the cluster. Each client is single-threaded and waits for each bulk 
operation to finish before issuing the next one.

I will try making the following changes to see what effect it may have:

- Reduce the number of replicas from 2 to 1.

- Disable dynamic mapping (This *should* have no net effect. But, it 
shouldn't hurt either, since we don't require this functionality)

Lastly, to test, I will reduce to 1 client doing bulk indexing to see if 
that helps narrow down the problem. But, it is not a long term solution for 
us since our steady flow of new data is so high, we would not be able to 
keep up in production with only one indexing thread.


On Monday, June 16, 2014 4:15:20 PM UTC-5, Jörg Prante wrote:
>
> I guess you hit the following condition:
>
> - you insert data with bulk indexing
>
> - your index has dynamic mapping and already has huge field mappings
>
> - bulk requests span over many nodes / shards / replicas and introduce 
> tons of new fields into the dynamic mapping
>
> - you do not wait for bulk responses before sending new bulk requests
>
> That is, ES tries heavily to create the new field mappings but the result 
> of the new mapping does not make it to the other node in time before new 
> bulks arrive at the other node. The node just sees there must be a mapping 
> for a new field, but the cluster state has none to present although the 
> field was being mapped.
>
> Maybe the cluster state is not sent at all, or it could not be read fully 
> from disk, or it is "stuck" somewhere else.
>
> ES tries hard to prevent such conditions by assigning high priority to 
> cluster state messages that are sent throughout the cluster. Also, ES 
> avoids flooding of such messages.
>
> Your observation is correct: the longer you execute bulk indexing with the 
> same type of data (except random data), the number of new field mappings 
> decreases over time, so the number of new ES cluster state promotions.
>
> You can try the following to tackle this challenge:
>
> - pre-create the field mappings for your indexes, or even better, 
> pre-create indices and disable dynamic mapping, so no cluster state changes 
> have to be promoted
>
> - switch to synchronous bulk requests, or reduce concurrency in your bulk 
> requests. So you let the bulk indexing routine wait for the cluster state 
> changes to be consistent at all nodes.
>
> - reduce the (perhaps huge) number of field mappings (more a question 
> about the type of data you index)
>
> - reduce number of nodes (obviously an anti-pattern) 
>
> - or reduce replica level (always a good thing for efficiency while using 
> bulk indexing), to give the cluster some breath to broadcast the new 
> cluster states in shorter time to the corresponding nodes
>
> Jörg
>
>
>
> On Mon, Jun 16, 2014 at 10:34 PM, Brooke Babcock  > wrote:
>
>> Thanks for the reply. 
>> We've checked the log files on all the nodes - no errors or warnings. 
>> Disks were practically empty - it was a fresh cluster, fresh index.
>>
>> We have noticed that the problem occurs less frequently the more data we 
>> send to the cluster. Our latest theory is that it "corrects itself" 
>> (meaning, we are able to get by _id again) once a flush occurs. So by 
>> sending it more data, we are ensuring that flushes happen more often.
>>
>>
>> On Monday, June 16, 2014 8:05:15 AM UTC-5, Alexander Reelsen wrote:
>>
>>> Hey,
>>>
>>> it seems, as if writing into the translog fails at some stage (from a 
>>> complete birds eye view). Can you check your logfiles, if you ran into some 
>>> weird exceptions before that happens? Also, you did not run out of disk 
>>> space at any time when this has happened?
>>>
>>>
>>> --Alex
>>>
>>>
>>> On Fri, Jun 6, 2014 at 8:39 PM, Brooke Babcock  
>>> wrote:
>>>
 In one part of our application we use Elasticsearch as an object store. 
 Therefore, when indexing, we supply our own _id. Likewise, when accessing 
 a 
 document we use the simple GET method to fetch by _id. This has worked 
 well 
 for us, up until recently. Normally, this is what we get:

 curl -XGET 'http://127.

Re: node failures

[Mark Walkom]

TTL does use a lot of resources as it constantly scans for expired docs.
It'd be more efficient to switch to daily indexes and then drop them,
though that might not fit your business requirements.

You can try forcing an optimise on an index,
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-optimize.html,
it's very resource intensive though but it if reduces your segment count
then it may allude to where the problem lies.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 17 June 2014 07:07, Kireet Reddy  wrote:

> java version is 1.7.0_55. the servers have a 32GB heap, 96GB of memory, 12
> logical cores, and 4 spinning disks.
>
> Currently we have about 450GB of data on each machine, average doc size is
> about 1.5KB. We create an index (4 shards, 1 replica) every N days. Right
> now we have 12 indices, meaning about 24 shards/node (12*4*2 / 4).
>
> Looking at ElasticHQ, I noticed some warnings around documents deleted.
> Our percentages are in the 70s and the pass level is 10% (!). Due to our
> business requirements, we have to use TTL. My understanding is this leads
> to a lot of document deletions and increased merge activity. However it
> seems that maybe segments with lots of deletes aren’t being merged? We
> stopped indexing temporarily and there are no merges occurring anywhere in
> the system so it’s not a throttling issue. We are using almost all default
> settings, but is there some setting in particular I should look at?
>
> On Jun 10, 2014, at 3:41 PM, Mark Walkom 
> wrote:
>
> Are you using a monitoring plugin such as marvel or elastichq? If not then
> installing those will give you a better insight into your cluster.
> You can also check the hot threads end point to check each node -
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html
>
> Providing a bit more info on your cluster setup may help as well, index
> size and count, server specs, java version, that sort of thing.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
>
>
> On 11 June 2014 00:41, Kireet Reddy  wrote:
>
>> On our 4 node test cluster (1.1.2), seemingly out of the blue we had one
>> node experience very high cpu usage and become unresponsive and then after
>> about 8 hours another node experienced the same issue. The processes
>> themselves stayed alive, gc activity was normal, they didn't experience an
>> OutOfMemoryError. The nodes left the cluster though, perhaps due to the
>> unresponsiveness. The only errors in the log files were a bunch of messages
>> like:
>>
>> org.elasticsearch.search.SearchContextMissingException: No search context
>> found for id ...
>>
>> and errors about the search queue being full. We see the
>> SearchContextMissingException occasionally during normal operation, but
>> during the high cpu period it happened quite a bit.
>>
>> I don't think we had an unusually high number of queries during that time
>> because the other 2 nodes had normal cpu usage and for the prior week
>> things ran smoothly.
>>
>> We are going to restart testing, but is there anything we can do to
>> better understand what happened? Maybe change a particular log level or do
>> something while the problem is happening, assuming we can reproduce the
>> issue?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/58351342-da89-43ad-a1be-194d8b608457%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/6ze7e1TVM8A/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAEM624bNyfbBkLZbeGpz8v%2Bq8VOPOLmAeGmWf%2BNQrEar2owLoQ%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email t

ElasticsearchIllegalStateException: Maximum number of bytes allocated exceeded

[Sriram Kannan]

I am getting this error constantly and the 5 nodes in the cluster goes down 
one by one. Is there a way to fix the issue? Could it be a corrupt data 
issue? 


[2014-06-16 13:13:34,422][WARN ][discovery.zen.ping.multicast] [nw-es-201] 
failed to receive confirmation on sent ping response to 
[[nw-es-202][xOi0MSQBQ4OoQjpuIDUjtA][nw-es-202.prod.pcln.com][inet[/172.21.88.113:9320]]{master=true}]
org.elasticsearch.transport.SendRequestTransportException: 
[nw-es-202][inet[/172.21.88.113:9320]][discovery/zen/multicast]
at 
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:219)
at 
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:190)
at 
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.handleNodePingRequest(MulticastZenPing.java:462)
at 
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.onMessage(MulticastZenPing.java:334)
at 
org.elasticsearch.common.network.MulticastChannel$Plain$Receiver.run(MulticastChannel.java:363)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.ElasticsearchIllegalStateException: Maximum 
number of bytes allocated exceeded: [836334432] (> 836318003)
at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:384)
at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:398)
at 
org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:423)
at 
org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:431)
at 
org.elasticsearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:60)
at 
org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:37)
at 
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:558)
at 
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:206)
... 5 more
[2014-06-16 13:13:34,601][WARN ][discovery.zen.ping.multicast] [nw-es-201] 
failed to receive confirmation on sent ping response to 
[[nw-es-205][p9b19RfCT1uF7Ry2_eQXQw][nw-es-205.prod.pcln.com][inet[/172.21.88.116:9320]]{master=true}]
org.elasticsearch.transport.SendRequestTransportException: 
[nw-es-205][inet[/172.21.88.116:9320]][discovery/zen/multicast]
at 
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:219)
at 
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:190)
at 
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.handleNodePingRequest(MulticastZenPing.java:462)
at 
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.onMessage(MulticastZenPing.java:334)
at 
org.elasticsearch.common.network.MulticastChannel$Plain$Receiver.run(MulticastChannel.java:363)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.ElasticsearchIllegalStateException: Maximum 
number of bytes allocated exceeded: [836334432] (> 836318003)
at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:384)
at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:398)
at 
org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:423)
at 
org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:431)
at 
org.elasticsearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:60)
at 
org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:37)
at 
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:558)
at 
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:206)

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3769ecc9-1119-4f3e-8131-76d4cfef24bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: IllegalArgumentException[No type mapped for [43]], version 1.2.1

[joergpra...@gmail.com]

I guess you hit the following condition:

- you insert data with bulk indexing

- your index has dynamic mapping and already has huge field mappings

- bulk requests span over many nodes / shards / replicas and introduce tons
of new fields into the dynamic mapping

- you do not wait for bulk responses before sending new bulk requests

That is, ES tries heavily to create the new field mappings but the result
of the new mapping does not make it to the other node in time before new
bulks arrive at the other node. The node just sees there must be a mapping
for a new field, but the cluster state has none to present although the
field was being mapped.

Maybe the cluster state is not sent at all, or it could not be read fully
from disk, or it is "stuck" somewhere else.

ES tries hard to prevent such conditions by assigning high priority to
cluster state messages that are sent throughout the cluster. Also, ES
avoids flooding of such messages.

Your observation is correct: the longer you execute bulk indexing with the
same type of data (except random data), the number of new field mappings
decreases over time, so the number of new ES cluster state promotions.

You can try the following to tackle this challenge:

- pre-create the field mappings for your indexes, or even better,
pre-create indices and disable dynamic mapping, so no cluster state changes
have to be promoted

- switch to synchronous bulk requests, or reduce concurrency in your bulk
requests. So you let the bulk indexing routine wait for the cluster state
changes to be consistent at all nodes.

- reduce the (perhaps huge) number of field mappings (more a question about
the type of data you index)

- reduce number of nodes (obviously an anti-pattern)

- or reduce replica level (always a good thing for efficiency while using
bulk indexing), to give the cluster some breath to broadcast the new
cluster states in shorter time to the corresponding nodes

Jörg



On Mon, Jun 16, 2014 at 10:34 PM, Brooke Babcock 
wrote:

> Thanks for the reply.
> We've checked the log files on all the nodes - no errors or warnings.
> Disks were practically empty - it was a fresh cluster, fresh index.
>
> We have noticed that the problem occurs less frequently the more data we
> send to the cluster. Our latest theory is that it "corrects itself"
> (meaning, we are able to get by _id again) once a flush occurs. So by
> sending it more data, we are ensuring that flushes happen more often.
>
>
> On Monday, June 16, 2014 8:05:15 AM UTC-5, Alexander Reelsen wrote:
>
>> Hey,
>>
>> it seems, as if writing into the translog fails at some stage (from a
>> complete birds eye view). Can you check your logfiles, if you ran into some
>> weird exceptions before that happens? Also, you did not run out of disk
>> space at any time when this has happened?
>>
>>
>> --Alex
>>
>>
>> On Fri, Jun 6, 2014 at 8:39 PM, Brooke Babcock 
>> wrote:
>>
>>> In one part of our application we use Elasticsearch as an object store.
>>> Therefore, when indexing, we supply our own _id. Likewise, when accessing a
>>> document we use the simple GET method to fetch by _id. This has worked well
>>> for us, up until recently. Normally, this is what we get:
>>>
>>> curl -XGET 'http://127.0.0.1:9200/data-2014.06.06/key/test1?pretty=true'
>>> {
>>>   "_index" : "data-2014.06.06",
>>>   "_type" : "key",
>>>   "_id" : "test1",
>>>   "_version" : 1,
>>>   "found" : true,
>>>   "_source":{"sData":"test data 1"}
>>> }
>>>
>>>
>>> Now, we often encounter a recently indexed document that throws the
>>> following error when we try to fetch it:
>>>
>>> curl -XGET 'http://127.0.0.1:9200/data-2014.06.06/key/test2?pretty=true'
>>> {
>>>   "error":"IllegalArgumentException[No type mapped for [43]]",
>>>   "status":500
>>> }
>>>
>>>
>>>
>>> This condition persists anywhere from 1 to 25 minutes or so, at which
>>> point we no longer receive the error for that document and the GET succeeds
>>> as normal. From that point on, we are able to consistently retrieve that
>>> document by _id without issue. But, soon after, we will find a different
>>> newly indexed document caught in the same bad state.
>>>
>>> We know the documents are successfully indexed. Our bulk sender (which
>>> uses the Java transport client) indicates no error during indexing and
>>> we are still able to locate the document by doing an ids query, such as:
>>>
>>> curl -XPOST "http://127.0.0.1:9200/data-2014.06.06/key/_search?pretty=
>>> true" -d '
>>> {
>>>   "query": {
>>> "ids": {
>>>   "values": ["test2"]
>>> }
>>>   }
>>> }'
>>>
>>> Which responds:
>>> {
>>>"took": 543,
>>>"timed_out": false,
>>>"_shards": {
>>>   "total": 10,
>>>   "successful": 10,
>>>   "failed": 0
>>>},
>>>"hits": {
>>>   "total": 1,
>>>   "max_score": 1.0,
>>>   "hits": [ {
>>>  "_index": "data-2014.06.06",
>>>  "_type": "key",
>>>  "_id": "test2",
>>>  "_score": 1.0,
>>>  "_source":{"sData": "test d

Re: Share a document across multiple indices

[Martin Angers]

Thanks, I'll explore your suggestion.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4fae0814-4577-44ae-b4ac-e8938a4b87ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: node failures

[Kireet Reddy]

java version is 1.7.0_55. the servers have a 32GB heap, 96GB of memory, 12 
logical cores, and 4 spinning disks.

Currently we have about 450GB of data on each machine, average doc size is 
about 1.5KB. We create an index (4 shards, 1 replica) every N days. Right now 
we have 12 indices, meaning about 24 shards/node (12*4*2 / 4). 

Looking at ElasticHQ, I noticed some warnings around documents deleted. Our 
percentages are in the 70s and the pass level is 10% (!). Due to our business 
requirements, we have to use TTL. My understanding is this leads to a lot of 
document deletions and increased merge activity. However it seems that maybe 
segments with lots of deletes aren't being merged? We stopped indexing 
temporarily and there are no merges occurring anywhere in the system so it's 
not a throttling issue. We are using almost all default settings, but is there 
some setting in particular I should look at?

On Jun 10, 2014, at 3:41 PM, Mark Walkom  wrote:

> Are you using a monitoring plugin such as marvel or elastichq? If not then 
> installing those will give you a better insight into your cluster.
> You can also check the hot threads end point to check each node - 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html
> 
> Providing a bit more info on your cluster setup may help as well, index size 
> and count, server specs, java version, that sort of thing.
> 
> Regards,
> Mark Walkom
> 
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
> 
> 
> On 11 June 2014 00:41, Kireet Reddy  wrote:
> On our 4 node test cluster (1.1.2), seemingly out of the blue we had one node 
> experience very high cpu usage and become unresponsive and then after about 8 
> hours another node experienced the same issue. The processes themselves 
> stayed alive, gc activity was normal, they didn't experience an 
> OutOfMemoryError. The nodes left the cluster though, perhaps due to the 
> unresponsiveness. The only errors in the log files were a bunch of messages 
> like:
> 
> org.elasticsearch.search.SearchContextMissingException: No search context 
> found for id ...
> 
> and errors about the search queue being full. We see the 
> SearchContextMissingException occasionally during normal operation, but 
> during the high cpu period it happened quite a bit.
> 
> I don't think we had an unusually high number of queries during that time 
> because the other 2 nodes had normal cpu usage and for the prior week things 
> ran smoothly.
> 
> We are going to restart testing, but is there anything we can do to better 
> understand what happened? Maybe change a particular log level or do something 
> while the problem is happening, assuming we can reproduce the issue?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/58351342-da89-43ad-a1be-194d8b608457%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to a topic in the Google 
> Groups "elasticsearch" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/elasticsearch/6ze7e1TVM8A/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/CAEM624bNyfbBkLZbeGpz8v%2Bq8VOPOLmAeGmWf%2BNQrEar2owLoQ%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5E198314-7782-4987-81B8-D7A37684C249%40feedly.com.
For more options, visit https://groups.google.com/d/optout.

Re: better places to store es.nodes and es.port in ES Hive integration?

[Costin Leau]

Thanks for sharing - can you also give an example of the table initialization 
in init.hive vs myscript.hql?

Cheers!

On 6/16/14 11:19 PM, Jinyuan Zhou wrote:

Just share a solution  I learned  hive side.

hive cli has an -i option that takes a  file of hive commands to initilize the 
session.
so I can put a list of set comand as well as add jar ... command in one file, 
say inithive
then run the cli as this:  hive -i init.hive -f myscript.hql.  Note table 
creation hql inside myscript.hql don't have to
set es.* properties as long as it appears in init.hive file  This solves my 
problem.
Thanks,


Jinyuan (Jack) Zhou


On Sun, Jun 15, 2014 at 10:24 AM, Jinyuan Zhou mailto:zhou.jiny...@gmail.com>> wrote:

Thanks Costin,
I am aiming at modifying  the existing hadoop cluster and hive installation 
and also modularizing   some common es.*
properies in a separate common place.  I know the first goal can be 
achieved with hive cli  --auxpath option  and
hive table's TBLPROPERTERTIES. For the secon goal, I am able to move  some 
es.* settings from TBLPROPERTIES
declaration to hive's set statments. For example, I can put

set es.nodes=my.domain.com 

in the same hql file  then skip es.nodes setting in TBLPROPERTIES in the 
external table delcarations in the SAME
hql. But I wish  I can move the set statetemnt in a separate file. I now 
realize this is rather a  hive question.
Regards,
Jack


On Sun, Jun 15, 2014 at 2:19 AM, Costin Leau mailto:costin.l...@gmail.com>> wrote:

Could you please raise an issue with some type of example? Due to the 
way Hadoop (and Hive) works,
things tend to be tricky in terms of configuring a job.

The configuration needs to be created before a job is submitted which in practice 
means "dynamic configurations"
are basically impossible (this also has some security implications 
which are simply avoided this way).
Thus either one specifies the configuration manually or loads a known 
location file (hive-site.xml,
core-site.xml...)
upfront, before the job is submitted.
This means when dealing with Hive, Pig, Cascading, etc... unless one 
adds a pre-processor to the job content
(script, flow, etc...)
by the time es-hadoop kicks in, the job is already running and thus its 
changes discarded.

Cheers,

On 6/14/14 1:57 AM, Jinyuan Zhou wrote:

Hi,
I am playing with elasticsearch and hive integration. The 
documentation says
to set configuration like es.nodes, es.port  in TBLPROPERTIES. It 
works.
But it can cause many reduntant codes. If I have ten data set to 
index to the same es cluster,
   I would have to repeat this information ten times in 
TBLPROPERTIES. Even if
   I use var substitution I still have to rwrite this subtititiov 
var for  each table definition.
What I am looking for is to put these info in say one file and  
pass the location, in some way, to hive cli
so hive elasticsearch will get these settings when trying to find 
es server to talk to.
I am not looking into put these info into files like  hive-site.xml.

Thanks,

Jack

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, 
send an email to
elasticsearch+unsubscribe@__googlegroups.com 

>.
To view this discussion on the web visit

https://groups.google.com/d/__msgid/elasticsearch/7040c805-__e845-4b3d-a9fe-5e18d8445f7f%__40googlegroups.com
 


>.
For more options, visit https://groups.google.com/d/__optout 
.


--
Costin

--
You received this message because you are subscribed to a topic in the Google 
Groups "elasticsearch" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/__topic/elasticsearch/__1WH7kOD3uKs/unsubscribe

.
To unsubscribe from this group and all its topics, send an email to 
elasticsearch+unsubscribe@__googlegroups.com

Creating sub-objects and nested objects in Elasticsearch plugin

[tnadeau]

Hello. I'm creating an Elasticsearch plugin that creates a new field type 
(similarly to how the elastic-mapper-attachments plugin does here 
).
 
However, I need to store sub-objects and several instances of a nested 
object. For instance, for a type mapping like this:

{
"my_elastic_type": {
"properties": {
"my_field": {
"type": "my_new_field_type",
"stuff": {
"reference": {
"index": "myIndex",
"type": "ref",
}
},
"stuff2": {
"reference": {
"index": "myOtherIndex",
"type": "ref",
}
}
}
}
}
}

and indexing a document like this:
{
"my_field": "text here"
}

I would like the store the equivalent to the following in the index:
{
"stuff": {
"calcField1": "foo",
"calcField2": "baz",
"nestedObjs": [
{
 "nestedField1": "info",
 "nestedField2": "data"
},
{
 "nestedField1": "info",
 "nestedField2": "data"
}
]
},
"stuff2": {
"calcField1": "foo2",
"calcField2": "baz2",
"nestedObjs": [
{
 "nestedField1": "info2",
 "nestedField2": "data2"
},
{
 "nestedField1": "info2",
 "nestedField2": "data2"
}
]
},
}

I'm having a lot of trouble figuring out how to add non-simple fields in my 
plugin. I can't find any documentation of how to develop an Elasticsearch 
plugin.

Thanks for any help.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9854898a-93ab-46f0-a52c-4dc6070aa35b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Index template requires settings object even if its value is empty

[Brian]

Alex,

I am running ES version 1.2.1.

It seemed to work (no errors in the logs), but I did it as an on-disk 
template and not via PUT. And without the settings, it behaved as if it 
wasn't there.

The question is now moot, because I actually need the following setting:

"settings" : {
  "index.mapping.ignore_malformed" : true,
  "index.query.default_field" : "message"
},

I don't have a problem fiddling with local files; Elasticsearch, the 
wrapper script, and everything else I need is stored in a single zip 
archive that our operations team can easily install. So once I install it 
on my laptop and verify that it's working, it's 100.% repeatable when 
installed on any QA or production server.

I also configure logstash's elasticsearch_http as follows:

manage_template => false

That way, I don't have to depend on logstash (or anything else) doing that 
for me. It's already done by the base ES install package.

Brian


On Monday, June 16, 2014 8:03:33 AM UTC-4, Alexander Reelsen wrote:
>
> Hey,
>
> which ES version are you using? Seems to work with the latest version. You 
> can also use the index template API, so you do not have to fiddle with 
> local files (and copy them when adding new nodes).
>
> PUT _template/automap
> {
>   "template": "*",
>   "mappings": {
> "_default_": {
>   "numeric_detection": true,
>   "properties": {
> "message": {
>   "type": "string"
> },
> "host": {
>   "type": "string"
> },
> "@version": {
>   "type": "string"
> }
>   }
> }
>   }
> }
>
>
>
> --Alex
>
>
> On Tue, Jun 3, 2014 at 5:57 PM, Brian > 
> wrote:
>
>> I am not sure if this is a problem or if it's OK.
>>
>> Working with the ELK stack I have switched direction, and instead of 
>> locking down the Elasticsearch mappings I am now using its automatic 
>> mapping functions. And by adding the following JSON template definition to 
>> the /*path.to.config*/templates/automap.json file I can get numeric 
>> fields automatically correctly mapped even though logstash always emits 
>> their values as strings ("45.6" instead of 45.6). Very nice!
>>
>> {
>>   "automap" : {
>> "template" : "*",
>> *"settings" : { },*
>> "mappings" : {
>>   "_default_" : {
>> "numeric_detection" : true,
>> "properties" : {
>>   "message" : {"type" : "string"},
>>   "host" : {"type" : "string"},
>>   "@version" : {"type" : "string"}
>> }
>>   }
>> }
>>   }
>> }
>>
>> When I removed the *"settings":{}* entirely, it was as if the template 
>> did not exist; the numeric detection was not enabled and all string values 
>> were seen as strings even if they contained numbers. Because all of the 
>> settings are being controlled within elasticsearch.yml and not the template 
>> (e.g. number of shards, number of replicas, and so on), eliminating the 
>> settings from the template is desired, even if I have to leave it in but 
>> set its value to the empty JSON object.
>>
>> If this is the way it's supposed to work, that's OK. But I couldn't find 
>> anything in the documentation about it, and just wanted to get a 
>> verification either way.
>>
>> Thanks!
>>
>> Brian
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/ff4afb8e-c3e4-4772-aa48-bd6a651c78e8%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0ffa60d5-92a1-462f-b335-de83907060eb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: IllegalArgumentException[No type mapped for [43]], version 1.2.1

[Brooke Babcock]

Thanks for the reply. 
We've checked the log files on all the nodes - no errors or warnings. Disks 
were practically empty - it was a fresh cluster, fresh index.

We have noticed that the problem occurs less frequently the more data we 
send to the cluster. Our latest theory is that it "corrects itself" 
(meaning, we are able to get by _id again) once a flush occurs. So by 
sending it more data, we are ensuring that flushes happen more often.


On Monday, June 16, 2014 8:05:15 AM UTC-5, Alexander Reelsen wrote:
>
> Hey,
>
> it seems, as if writing into the translog fails at some stage (from a 
> complete birds eye view). Can you check your logfiles, if you ran into some 
> weird exceptions before that happens? Also, you did not run out of disk 
> space at any time when this has happened?
>
>
> --Alex
>
>
> On Fri, Jun 6, 2014 at 8:39 PM, Brooke Babcock  > wrote:
>
>> In one part of our application we use Elasticsearch as an object store. 
>> Therefore, when indexing, we supply our own _id. Likewise, when accessing a 
>> document we use the simple GET method to fetch by _id. This has worked well 
>> for us, up until recently. Normally, this is what we get:
>>
>> curl -XGET 'http://127.0.0.1:9200/data-2014.06.06/key/test1?pretty=true'
>> {
>>   "_index" : "data-2014.06.06",
>>   "_type" : "key",
>>   "_id" : "test1",
>>   "_version" : 1,
>>   "found" : true,
>>   "_source":{"sData":"test data 1"}
>> }
>>
>>
>> Now, we often encounter a recently indexed document that throws the 
>> following error when we try to fetch it:
>>
>> curl -XGET 'http://127.0.0.1:9200/data-2014.06.06/key/test2?pretty=true'
>> {
>>   "error":"IllegalArgumentException[No type mapped for [43]]",
>>   "status":500
>> }
>>
>>
>>
>> This condition persists anywhere from 1 to 25 minutes or so, at which 
>> point we no longer receive the error for that document and the GET succeeds 
>> as normal. From that point on, we are able to consistently retrieve that 
>> document by _id without issue. But, soon after, we will find a different 
>> newly indexed document caught in the same bad state.
>>
>> We know the documents are successfully indexed. Our bulk sender (which 
>> uses the Java transport client) indicates no error during indexing and 
>> we are still able to locate the document by doing an ids query, such as:
>>
>> curl -XPOST "
>> http://127.0.0.1:9200/data-2014.06.06/key/_search?pretty=true"; -d '
>> {
>>   "query": {
>> "ids": {
>>   "values": ["test2"]
>> }
>>   }
>> }'
>>
>> Which responds:
>> {
>>"took": 543,
>>"timed_out": false,
>>"_shards": {
>>   "total": 10,
>>   "successful": 10,
>>   "failed": 0
>>},
>>"hits": {
>>   "total": 1,
>>   "max_score": 1.0,
>>   "hits": [ {
>>  "_index": "data-2014.06.06",
>>  "_type": "key",
>>  "_id": "test2",
>>  "_score": 1.0,
>>  "_source":{"sData": "test data 2"}
>>   } ]
>>}
>> }
>>
>>
>> We first noticed this behavior in version 1.2.0. When we upgraded to 
>> 1.2.1, we deleted all indexes and started with a fresh cluster. We hoped 
>> our problem would be solved by the big fix that came in 1.2.1, but we are 
>> still regularly seeing it. Although our situation may sound like the 
>> routing bug introduced in 1.2.0, we are certain that it is not. This 
>> appears to be a significant issue with the translog - we hope the 
>> developers will be able to look at what may have changed. We did not notice 
>> this problem in version 1.1.1.
>>
>> Just in case, here is the mapping being used:
>> curl -XGET '
>> http://127.0.0.1:9200/data-2014.06.06/key/_mapping?pretty=true'
>> {
>>   "data-2014.06.06" : {
>> "mappings" : {
>>   "key" : {
>> "_all" : {
>>   "enabled" : false
>> },
>> "properties" : {
>>   "sData" : {
>> "type" : "string",
>> "index" : "no"
>>   }
>> }
>>   }
>> }
>>   }
>> }
>>
>>
>> Thanks for your help.
>>
>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/20c45cf8-3459-47f5-8cc3-1e63c93b2c0c%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8449ec28-7b7f-4e8b-a3c2-6f410ef80187%40googlegroups.com.

Re: 0.90.2 _update or _bulk update causing NullPointerException in logs and I start losing shards

[Rohit Jaiswal]

Hi Boaz,
   We are using 0.90.2 and run into this issue. As i 
understand, one option is to upgrade to 0.90.3. If we continue using 0.90.2 
and use (increase) retry_on_conflict, we will not see the problem? Please 
clarify.
 
Thanks,
Rohit
On Wednesday, August 7, 2013 9:39:56 AM UTC-7, Boaz Leskes wrote:

> HI Eric,
>
> OK. Based on the gist you sent, i tracked down a problem at fixed it: 
> https://github.com/elasticsearch/elasticsearch/issues/3448 . Thanks!! The 
> fix is part of 0.90.3, so I'd recommend upgrading. This is a secondary 
> problem which occurs when two requests try to update the same document at 
> exactly the same time. One of them succeeds and the other fails with a 
> version conflict (that error was masked by the error you were seeing). You 
> can use (or increase) the retry_on_conflict parameter to make the failing 
> request try again.
>
> I'm still curious about your reporting of loosing replicas. Can you 
> elaborate more about what happens? Do you see anything in the logs?
>
> Cheers,
> Boaz
>
> On Tuesday, August 6, 2013 5:09:26 AM UTC+2, Eric Sites wrote:
>>
>> Boaz,
>>
>> Sorry but I no longer have those logs, I upgraded to 0.90.2 from 0.90.0 
>> and wiped the logs when I did.
>> I did the upgrade to use the _bulk api for my update.
>>
>> Basically the "lang", "js" was not the issue. 
>>
>> I was using different scripts with the same set of params and an upcert. 
>> The fix was to use a different param name for different scripts, about 10 
>> unique scripts in total.
>>
>> I was losing replicated shards about every 10,000 to 30,000 updates, 
>> never the primary shard.
>>
>> I have 185 million + large json documents, with 100 shards in 1 index 
>> with 1 replication, so 200 shards total over 6 servers. Each shard is about 
>> 10.4 GB in size.
>> About 2 TB of data, 1 TB primary, 1 TB replicated.
>>
>> Cheers,
>> Eric Sites
>>
>> From: Boaz Leskes 
>> Reply-To: 
>> Date: Monday, August 5, 2013 5:38 PM
>> To: 
>> Subject: Re: 0.90.2 _update or _bulk update causing NullPointerException 
>> in logs and I start losing shards
>>
>> Hi Eric,
>>
>> Glad to hear you solved it. It would be great if you can share the failed 
>> logs from the _update (non bulk call). A failed script shouldn't cause 
>> shards to drop so I would like to research it some more.
>>
>> Cheers,
>> Boaz
>>
>>
>> On Mon, Aug 5, 2013 at 6:40 PM, Eric Sites  wrote:
>>
>>> Boaz,
>>>
>>> I found and fixed the problem.
>>>
>>> I added the "lang", "js" to the update json, that was not needed before 
>>> in es 0.90.0.
>>> I also changed the name of new_tracking to match the name of the action 
>>> in the params section.
>>> So for example the script now looks like this:
>>>
>>> if (ctx._source['tracking'] != null) {
>>> if (ctx._source.tracking['some_action'] != null) {
>>> ctx._source.tracking.some_action += param1;
>>> } else {
>>> ctx._source.tracking['some_action'] = 1;
>>> }
>>> } else {
>>> ctx._source.tracking = new_some_action;
>>> }
>>>
>>> "params" : { "param1" : 1, "new_some_action" : { "some_action" : 1 } }
>>>
>>> Cheers,
>>> Eric Sites
>>>
>>> From: Boaz Leskes 
>>> Reply-To: 
>>> Date: Monday, August 5, 2013 10:35 AM
>>> To: 
>>> Subject: Re: 0.90.2 _update or _bulk update causing 
>>> NullPointerException in logs and I start losing shards
>>>
>>> Hi Eric,
>>>
>>> This is interesting. The log stack trace from the gist comes from the 
>>> bulk calls. Can you also post one from a failed _update? Cross checking 
>>> them might help pin pointing the issue.
>>>
>>> Cheers,
>>> Boaz
>>>
>>> On Monday, August 5, 2013 1:34:16 AM UTC+2, eric_...@mac.com wrote:

 I am getting java.lang.NullPointerException exception in my 
 ElasticSearch cluster logs when I am doing a _bulk update or just an 
 _update.
 I am sending a lot of data to my clusters. After I get this error I 
 lose a shard and it has to be recreated.

 version 0.90.2

 gist: https://gist.github.com/EricSites/6152468

 I get this using the _bulk api or just normal _update api.

 My update script is a little complicated.
 I am adding a tracking object to my document if it does not exists. 
 There should only be one of these and it should not be an array of these.
 If the object does exists, I am trying to add a new field to the 
 tracking object to keep track on counts.
 So if the field does not exists I create it, else just += to it.

 if (ctx._source['tracking'] != null) {
 if (ctx._source.tracking['some_action'] != null) {
 ctx._source.tracking.some_action += param1;
 } else {
 ctx._source.tracking['some_action'] = 1;
 }
 } else {
 ctx._source.tracking = new_tracking;
 }


 Here is my mapping for this:
 {
"sample" : {
   "index_options" : "docs",
   "properties" : {
  "tracking" : {
  "type" : "obj

Re: better places to store es.nodes and es.port in ES Hive integration?

[Jinyuan Zhou]

Just share a solution  I learned  hive side.

hive cli has an -i option that takes a  file of hive commands to initilize
the session.
so I can put a list of set comand as well as add jar ... command in one
file, say inithive
then run the cli as this:  hive -i init.hive -f myscript.hql.  Note table
creation hql inside myscript.hql don't have to set es.* properties as long
as it appears in init.hive file  This solves my problem.
Thanks,


Jinyuan (Jack) Zhou


On Sun, Jun 15, 2014 at 10:24 AM, Jinyuan Zhou 
wrote:

> Thanks Costin,
> I am aiming at modifying  the existing hadoop cluster and hive
> installation and also modularizing   some common es.* properies in a
> separate common place.  I know the first goal can be achieved with hive cli
>  --auxpath option  and hive table's TBLPROPERTERTIES. For the secon goal, I
> am able to move  some es.* settings from TBLPROPERTIES declaration to
> hive's set statments. For example, I can put
>
>set es.nodes=my.domain.com
>
> in the same hql file  then skip es.nodes setting in TBLPROPERTIES in the
> external table delcarations in the SAME hql. But I wish  I can move the set
> statetemnt in a separate file. I now realize this is rather a  hive
> question.
> Regards,
> Jack
>
>
> On Sun, Jun 15, 2014 at 2:19 AM, Costin Leau 
> wrote:
>
>> Could you please raise an issue with some type of example? Due to the way
>> Hadoop (and Hive) works,
>> things tend to be tricky in terms of configuring a job.
>>
>> The configuration needs to be created before a job is submitted which in
>> practice means "dynamic configurations"
>> are basically impossible (this also has some security implications which
>> are simply avoided this way).
>> Thus either one specifies the configuration manually or loads a known
>> location file (hive-site.xml, core-site.xml...)
>> upfront, before the job is submitted.
>> This means when dealing with Hive, Pig, Cascading, etc... unless one adds
>> a pre-processor to the job content (script, flow, etc...)
>> by the time es-hadoop kicks in, the job is already running and thus its
>> changes discarded.
>>
>> Cheers,
>>
>> On 6/14/14 1:57 AM, Jinyuan Zhou wrote:
>>
>>> Hi,
>>> I am playing with elasticsearch and hive integration. The documentation
>>> says
>>> to set configuration like es.nodes, es.port  in TBLPROPERTIES. It works.
>>> But it can cause many reduntant codes. If I have ten data set to index
>>> to the same es cluster,
>>>   I would have to repeat this information ten times in TBLPROPERTIES.
>>> Even if
>>>   I use var substitution I still have to rwrite this subtititiov var for
>>>  each table definition.
>>> What I am looking for is to put these info in say one file and  pass the
>>> location, in some way, to hive cli
>>> so hive elasticsearch will get these settings when trying to find es
>>> server to talk to.
>>> I am not looking into put these info into files like  hive-site.xml.
>>>
>>> Thanks,
>>>
>>> Jack
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to
>>> elasticsearch+unsubscr...@googlegroups.com >> unsubscr...@googlegroups.com>.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/7040c805-
>>> e845-4b3d-a9fe-5e18d8445f7f%40googlegroups.com
>>> >> e845-4b3d-a9fe-5e18d8445f7f%40googlegroups.com?utm_medium=
>>> email&utm_source=footer>.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> Costin
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/
>> topic/elasticsearch/1WH7kOD3uKs/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/
>> msgid/elasticsearch/539D6507.3080207%40gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> -- Jinyuan (Jack) Zhou
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CANBTPCErh1M5_xNa0SE-ZShpUDuXKTPMCYqrWCB1z36%3D9vjaDQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Percolating a document with nested mapping

[Yuri M]

Hello,

I have noticed that a document with fields of nested type is not percolated 
correctly (all search IDs are returned). Once I remove those fields 
everything works just fine. Any ideas?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e9b2d287-09fb-4527-8de0-c5d8424f15c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Correlation Aggregation function

[Siva]

Hi,

We are trying to replace our home-grown relational Data Model based 
analytics engine with elastic search. In the existing implementation we 
used CORR (correlation function) in couple of charts, couldn't find this in 
elastic search. Is this something that's in the pipeline for future 
releases OR any technical limitation for not supporting it? Please comment.

Thanks in advance.

Siva

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/efa8368d-bb85-4c32-af76-c61f171b133b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Drop down box panel

[Robert Mattler]

Has anybody created a drop down aggregation panel for kibana?  I.E. a list 
of customers.  We have too many customers for the provided panels.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c50b3c62-b575-4d4e-ba29-918c117cf5c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Tribe node data compliance across regions

[Kyle House]

We are staging a rollout of multiple ES clusters across multiple regions, 
and we have a requirement to keep data contained inside the predefined 
regions. I wanted to check on how a tribe node would impact this 
requirement. My understanding is this node would be attached to multiple 
clusters for quick searching across regions, but would not pull any data 
locally thus violating this data requirement. Any information would be 
helpful, thanks!

--Kyle

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f81460ac-d2db-4ae4-a014-29ed86c0a5de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ElasticSearch coté Front

[John Bourdais]

Bonjour à tous,

Je débute avec *ElasticSearch *et j'ai une interrogation sur la mise en 
place coté front, je m'explique.

J'ai une application Java JEE d'administration qui permet de  consulter les 
Logs qui ont été enregistrés sous ES par le biais de l'API Java de 
*ElasticSearch*.

Jusque là pas de problème j'ai bien réussi à créer mon cluster et mon index 
et j'arrive bien à indexer mes logs.

*Par contre je suis en train de réaliser l'IHM de mon admin et là je me 
pose plusieurs questions* :


   - J'ai plusieurs critères de recherche :


   1. ID du log (String)
   2. Date de début de l'appel (Date ==> Utilisation de la fonction *Range* 
   de ES)
   3. Date de fin de l'appel (Date ==> Utilisation de la fonction *Range* 
   de ES)
   4. Nom du partenaire à réalisé l'appel (Liste déjà connu ==> Utilisation 
   de la fonction *Facet*)
   5. Nom du service ayant été appelé (Liste déjà connu ==> Utilisation de 
   la fonction *Facet*)
   6. Trace du logs (stacktrace)
   

   - Je souhaiterais pourvoir combiner certains critères de recherche en *Facet 
   *et d'autre par un formulaire HTML classic.

*Exemple** :* http://demo.fullscale.co/multiselect/
J'aimerais pouvoir donc avoir mes critères de recherche (1,2,3) dans un 
formulaire classic et les critères (4,5) en Facet comme dans l'exemple du 
lien du dessus.

*La question est :*  Est ce réalisable ? Si oui comment (*API Javascript* 
ou Appel Ajax avec *API Java*) ?

Merci d'avance pour vos réponses.



-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6a25-6d75-4b91-9d01-472bbff25e28%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Losing data after Elasticsearch restart

[Rohit Jaiswal]

Hi Alexander,
Thanks for your reply. We plan to upgrade in the
long run, however we need to fix the data loss problem on 0.90.2 in the
immediate term.

Here is the stack trace -


10:09:37.783 PM

[22:09:37,783][WARN ][indices.cluster  ] [Storm]
[b7a76aa06cfd4048987d1117f3e0433a][0] failed to start shard
org.elasticsearch.indices.recovery.RecoveryFailedException:
[b7a76aa06cfd4048987d1117f3e0433a][0]: Recovery failed from [Jeffrey
Mace][_jjr5BYJQjO6QzzheyDmhw][inet[/10.4.35.200:9300]] into
[Storm][FiW6mbR5ThqqSii5Wc28lQ][inet[/10.4.40.95:9300]]
at
org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:293)
at
org.elasticsearch.indices.recovery.RecoveryTarget.access$300(RecoveryTarget.java:62)
at
org.elasticsearch.indices.recovery.RecoveryTarget$2.run(RecoveryTarget.java:163)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.transport.RemoteTransportException: [Jeffrey
Mace][inet[/10.4.35.200:9300]][index/shard/recovery/startRecovery]
Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
[b7a76aa06cfd4048987d1117f3e0433a][0] Phase[2] Execution failed
at
org.elasticsearch.index.engine.robin.RobinEngine.recover(RobinEngine.java:1147)
at
org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:526)
at
org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:116)
at
org.elasticsearch.indices.recovery.RecoverySource.access$1600(RecoverySource.java:60)
at
org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:328)
at
org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:314)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:265)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.transport.RemoteTransportException:
[Storm][inet[/10.4.40.95:9300]][index/shard/recovery/translogOps]
Caused by: org.elasticsearch.indices.InvalidAliasNameException:
[b7a76aa06cfd4048987d1117f3e0433a] Invalid alias name
[1a4077872e41c0634cee780c1e5fc263bdd5f14b15ac9239480547ab2d3601eb], Unknown
alias name was passed to alias Filter
at
org.elasticsearch.index.aliases.IndexAliasesService.aliasFilter(IndexAliasesService.java:99)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareDeleteByQuery(InternalIndexShard.java:382)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:628)
at
org.elasticsearch.indices.recovery.RecoveryTarget$TranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:447)
at
org.elasticsearch.indices.recovery.RecoveryTarget$TranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:416)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:265)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
[22:09:37,799][WARN ][cluster.action.shard ] [Storm] sending failed
shard for [b7a76aa06cfd4048987d1117f3e0433a][0],
node[FiW6mbR5ThqqSii5Wc28lQ], [R], s[INITIALIZING], reason [Failed to start
shard, message
[RecoveryFailedException[[b7a76aa06cfd4048987d1117f3e0433a][0]: Recovery
failed from [Jeffrey Mace][_jjr5BYJQjO6QzzheyDmhw][inet[/10.4.35.200:9300]]
into [Storm][FiW6mbR5ThqqSii5Wc28lQ][inet[/10.4.40.95:9300]]]; nested:
RemoteTransportException[[Jeffrey
Mace][inet[/10.4.35.200:9300]][index/shard/recovery/startRecovery]];
nested: RecoveryEngineException[[b7a76aa06cfd4048987d1117f3e0433a][0]
Phase[2] Execution failed]; nested:
RemoteTransportException[[Storm][inet[/10.4.40.95:9300]][index/shard/recovery/translogOps]];
nested: InvalidAliasNameException[[b7a76aa06cfd4048987d1117f3e0433a]
Invalid alias name
[1a4077872e41c0634cee780c1e5fc263bdd5f14b15ac9239480547ab2d3601eb], Unknown
alias name was passed to alias Filter]; ]]
[22:09:38,025][WARN ][indices.cluster  ] [Storm]
[b7a76aa06cfd4048987d1117f3e0433a][0] failed to start shard
org.elasticsearch.indices.recovery.RecoveryFailedException:
[b7a76aa06cfd4048987d1117f3e0433a][0]: Recovery failed from [Jeffrey
Mace][_jjr5BYJQjO6QzzheyDmhw][inet[/10.4.35.200:9300]] into
[Storm][FiW6mbR5ThqqSii5Wc28lQ][inet[/10.4.40.95:9300]]
at
org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:293)
at
org.elasticsearch.indices.recovery.RecoveryTarget.acce

Re: does minimum_master_nodes include ones "self"?

[Alexander Reelsen]

Hey,

yes, it is the number of nodes in a cluster, which are able to become a
master node. If node A and node B are configured that way, both are
counted, possibly including  themselves. Makes sense?


--Alex


On Mon, Jun 16, 2014 at 5:11 PM,  wrote:

> running 1.2.1
>
> If a cluster has 3 master eligible nodes, and one node dies leaving nodeA
> and nodeB and minimum_master_nodes = 2
>
> Does nodeA when up, include itself when evaluating minimum_master_nodes?
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/5674f8bd-fb02-4e68-83ae-e093cc66a074%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-%3DmMMYV-%2BJ2th0UGUTDo%2BOJ7BwusX6SK58eLpW%2Bay08A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Detecting end of Scroll...

[mooky]

Just to confirm:
Is it correct that in order to automatically clean up the resources 
associated with a scroll, you must execute it until zero results are 
retrieved?
What about the case when all of the hits are returned in the first result 
(ie num hits = total hits)


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/bb66ee75-c40b-465e-96c7-6433c4c08a58%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Cannot Increase Write TPS in Elasticsearch by adding more nodes

[pranav amin]

We used Jmeter for this test.

On Friday, June 13, 2014 10:13:02 AM UTC-4, Greg Murnane wrote:
>
> I haven't seen it asked yet; what is feeding data into your elasticsearch? 
> Depending on what you're doing to get it there, a large document size could 
> easily bottleneck some feeding mechanisms. It's also noteable that some 
> "green" spinning disks top out in the realm of 72MB/s. It might be useful 
> to make sure that your feeding mechanism can handle more than 500 TPS.
>
> The information transmitted in this email is intended only for the 
> person(s) or entity to which it is addressed and may contain confidential 
> and/or privileged material. Any review, retransmission, dissemination or 
> other use of, or taking of any action in reliance upon, this information by 
> persons or entities other than the intended recipient is prohibited. If you 
> received this email in error, please contact the sender and permanently 
> delete the email from any computer.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/78489304-73b9-42d7-a8c3-c1ceb58fe84a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

spark and elasticsearch-hadoop-2.0.0

[Ignacio Mulas]

Hi! 
I am trying to setup a spark + elasticsearch solution. I am trying the 
simple example done by Barnaby Gray here [1], trying to adapt it to the 
elasticsearch-hadoop 2.0.0. I get some connectivity error that I am not 
able to overcome. I get the following error:  

14/06/16 17:11:13 INFO HttpMethodDirector: I/O exception 
(java.net.ConnectException) caught when processing request: Connection 
timed out
14/06/16 17:11:13 INFO HttpMethodDirector: Retrying request

For the complete log: https://gist.github.com/sallum/6792a0505d107ea1e47f

I have the following scala code: 
https://gist.github.com/sallum/020f7847917fbadde123

I am sure that there is connectivity and the port is open in the address 
used:

$machine~> telnet 172.17.10.20 9200
Trying 172.17.10.20...
Connected to 172.17.10.20.
Escape character is '^]'.

Has anyone faced the same issue? Any ideas what can it be?

Thanks in advance!
Ignacio

[1] http://loads.pickle.me.uk/2013/11/12/spark-and-elasticsearch.html

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9a3ef325-b2f0-46b6-92af-e243932935ab%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: date_histogram not returning key_as_string

[Tim Heikell]

Thank you Alexander, but I got it figured out. I was storing 
sessionStartTime as a Unix timestamp but did not provide any mapping, so 
the field type was set to be a double. Once I explicitly mapped 
sessionStartTime to be a date my query results started including the 
key_as_string field.

Tim


On Monday, June 16, 2014 5:38:42 AM UTC-7, Alexander Reelsen wrote:
>
> Hey,
>
> can you create a full gist to reproduce this? Should work out of the box 
> (dont see anything wrong here), also on 1.1.0
>
>
> --Alex
>
>
> On Thu, Jun 5, 2014 at 8:06 PM, Tim Heikell  > wrote:
>
>> Sorry for the noob question, but is there some setting I am missing? It's 
>> not clear to me why I'm not getting a key_as_string field in my results. 
>> I'm running v1.1.0, here is my search:
>>
>> GET /_all/_search
>> {
>>   "aggs": {
>> "totalsByHour": {
>>   "date_histogram": {
>> "field": "sessionStartTime",
>> "interval": "day",
>> "format": "-MM-dd"
>>   }
>> }
>>   }
>> }
>>
>> and here are my results:
>>
>>"aggregations": {
>>   "totalsByHour": {
>>  "buckets": [
>> {
>>"key": 140166720,
>>"doc_count": 15973
>> },
>> {
>>"key": 140175360,
>>"doc_count": 11996
>> },
>> {
>>"key": 140184000,
>>"doc_count": 14
>> },
>> {
>>"key": 140192640,
>>"doc_count": 4031
>> }
>>  ]
>>   }
>>}
>>  
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/0acca250-ab7f-48a8-a73a-648f7f544dcd%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2fbead94-8471-43e9-9544-9e23f3d2d1b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Detecting end of Scroll...

[mooky]

Oh, I see.
Can we make any assumption about the scrollId not changing when we reach 
the end of the results?




On Monday, 16 June 2014 15:58:20 UTC+1, Ralph Meijer wrote:
>
> On Mon, Jun 16, 2014 at 07:45:00AM -0700, mooky wrote: 
> > [..] 
> > That said, I do notice that when the end of the results are reached, the 
> > scrollId stops changing - and its also a lot shorter - and also it seems 
> to 
> > always be the same value (c2NhbjswOzE7dG90YWxfaGl0czoxOw==) 
>
> That scroll id is the base64 encoding of the following string: 
>
> scan;0;1;total_hits:1; 
>
> -- 
> Groetjes, 
>
> ralphm 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/34e73a88-285e-4ea7-8468-5dc11b7ee9bd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

does minimum_master_nodes include ones "self"?

[bitsofinfo . g]

running 1.2.1

If a cluster has 3 master eligible nodes, and one node dies leaving nodeA 
and nodeB and minimum_master_nodes = 2

Does nodeA when up, include itself when evaluating minimum_master_nodes?


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5674f8bd-fb02-4e68-83ae-e093cc66a074%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Detecting end of Scroll...

[Ralph Meijer]

On Mon, Jun 16, 2014 at 07:45:00AM -0700, mooky wrote:
> [..]
> That said, I do notice that when the end of the results are reached, the
> scrollId stops changing - and its also a lot shorter - and also it seems to
> always be the same value (c2NhbjswOzE7dG90YWxfaGl0czoxOw==)

That scroll id is the base64 encoding of the following string:

scan;0;1;total_hits:1;

-- 
Groetjes,

ralphm

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/20140616145804.GB25839%40ik.nu.
For more options, visit https://groups.google.com/d/optout.

Re: Percollation limits

[Luca Cavanna]

It is something we might look into but there's no concrete plan for now. On
the other hand metadata filtering (or eventually routing) allows to reduce
the number of queries that need to be run, only problem is that users need
to do it themselves.


On Mon, Jun 16, 2014 at 4:53 PM, Ralph Meijer  wrote:

> On Mon, Jun 16, 2014 at 06:19:23AM -0700, Luca Cavanna wrote:
> > Hi Maciej,
> > what you describe doesn't sound insane, just make sure you use proper
> filtering
> > as much as you can to limit the number of queries you execute when
> percolating
> > each document.
> > Also, with the percolator available since 1.0 you can scale out just by
> adding
> > more nodes and have the percolator queries distributed over multiple
> shards.
> > That means that if you were to reach the limit of a single shard you
> could
> > always scale out.
>
> If I remember correctly from Martijn's presentation, each percolator
> query is matched against the document sequentially. Are there
> plans for using commonality between queries to do more efficient
> matching, maybe using decision trees or somesuch?
>
> --
> Cheers,
>
> ralphm
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/BQbBFTyx31g/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/20140616145342.GA25839%40ik.nu
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CADdZ9MXrQyjNpfT%3D8HcvVtROduhqUtu8-CQOX6DGUPFBOx%3DPuA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Percollation limits

[Ralph Meijer]

On Mon, Jun 16, 2014 at 06:19:23AM -0700, Luca Cavanna wrote:
> Hi Maciej,
> what you describe doesn't sound insane, just make sure you use proper 
> filtering
> as much as you can to limit the number of queries you execute when percolating
> each document.
> Also, with the percolator available since 1.0 you can scale out just by adding
> more nodes and have the percolator queries distributed over multiple shards.
> That means that if you were to reach the limit of a single shard you could
> always scale out.

If I remember correctly from Martijn's presentation, each percolator
query is matched against the document sequentially. Are there
plans for using commonality between queries to do more efficient
matching, maybe using decision trees or somesuch?

-- 
Cheers,

ralphm

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/20140616145342.GA25839%40ik.nu.
For more options, visit https://groups.google.com/d/optout.

Detecting end of Scroll...

[mooky]

According to the documentation, you know when you have reached the end of 
the scroll results when you receive zero hits in the response.
However, if your SearchType is SCAN, then you receive zero hits in the 
*first* result,
The question is how to avoid this ambiguity. 

My code that handles the scrolling results doesn't know what the SearchType 
is - so it doesn't know how to differentiate between the two scenarios of 
in which we receive zero hits.

Can the end of the results not be signalled through the scrollId?
I kind of expected the scrollId to be null when we reached the end of the 
results (but I can see its not).

That said, I do notice that when the end of the results are reached, the 
scrollId stops changing - and its also a lot shorter - and also it seems to 
always be the same value (c2NhbjswOzE7dG90YWxfaGl0czoxOw==)


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6a9da841-1c70-4179-a0ba-b6f145363e23%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Accessing Search Templates via Rest

[Alexander Reelsen]

Hey,

no, this is not yet possible, but this will be added sooner or later as the
search template API should behave like any other API.


--Alex


On Fri, Jun 13, 2014 at 9:51 AM, Sebastian Gräser 
wrote:

> so i guess its not possible?
>
> Am Dienstag, 10. Juni 2014 16:58:31 UTC+2 schrieb Sebastian Gräser:
>
>> Hello,
>>
>> maybe someone can help me. Is there a way to get the available search
>> templates via rest api? havent found a way yet, hope you can help me.
>>
>> Best regards
>> Sebastian
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/ae1fedb0-4c74-4407-9532-fe7ad705ceb0%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-7y7s8NhRS5P7-C2MxnfnwnF4PZDywP9qfkKCK6NVFfQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch field mapping, dynamic_templates

[Alexander Reelsen]

Hey,

using your curl call you only set the mapping for existing indices, but not
for newly created ones. You may want to consider using index templates for
this specific field.


--Alex


On Mon, Jun 9, 2014 at 4:45 PM, sirkubax 
wrote:

> As I did mention here
> https://groups.google.com/forum/#!topic/elasticsearch/7Bn5Pc6TSgs
> I have some sorting issue when date is stored in string format.
>
>
>
> *I did try to set field format as date, but it is overide when date is
> inserted into index.The mapping was:*
>
> curl -XPUT localhost:9200/*/_mapping/loglog -d '
> {
> "loglog" : {
> "properties" : {
> "testdate7" : {"type" : "date", "format" : "-MM-dd
> HH:mm:ss.SS",  "store" : true }
> }
> }
> }
> '
>
> *but after while it become:*
>
> "properties" : {
>   "testdate7" : {
> "type" : "string",
> "norms" : {
>   "enabled" : false
> },
>
>
> *I guess it may be caused by *
>
> "dynamic_templates" : [ {
>   "*string*_*fields*" : {
> "mapping" : {
>   "type" : "multi_field",
>   "fields" : {
> "raw" : {
>   "index" : "not_analyzed",
>   "ignore_above" : 256,
>   "type" : "string"
> },
> "{name}" : {
>   "index" : "analyzed",
>   "omit_norms" : true,
>   "type" : "string"
> }
>   }
> },
> "match" : "*",
> "match_mapping_type" : "string"
>   }
> } ],
>
>
>
>
>
>
>
> *So the question os, how create mappint that would cover that issue.The
> message that is insterted looks like:*
> "ANYTHING (pid: 23291, thread: 4131280592) ] [* aa.xx:555: MSG(3)
> 2014-06-09 10:50:08.255111 ... "
>
>
> *I did try to set*
>  "dynamic_date_formats" : ["date_optional_time", "-MM-dd
> HH:mm:ss.SS"],
> *but I think I did not set it properly.*
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/bf31ea3f-292b-451b-aa14-0ef2f3632f44%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8cXo_Hx7-UszUWcWsE51k0q4AMKNamr-6Zq1%2BLAVMMzg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Find all the geoshapes that insersects with a given latitude/longitude

[Alexander Reelsen]

Hey,

this looks like a problem, because you are using an _id object, I filed an
issue here: https://github.com/elasticsearch/elasticsearch/issues/6514
You can just specify ids manually or make sure, that you _id field is empty
for now.


--Alex



On Sun, Jun 8, 2014 at 9:42 PM, Vidal Chriqui 
wrote:

> Hi
>
> My goal is to find for a given latitude/longitude all the indexed
> documents (circular geoshapes with specific radius for each) that contains
> this lat/lon.
> If necessary i'm ok to transform the circular zones with envelope, but
> this does not seem to be the issue.
>
> I need help to find the way to write the search query.
> In this gist, you can find the document mappings, 3 sample docs et 2
> attempts to write the query, but, those queries do not return anything
> whereas it should (obviously the query is not the correct one).
>
> https://gist.github.com/anonymous/3e6aa70bf8b31e8eb345
>
> Thanks for your help to write the correct query.
>
> Best regards
> Vidal
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/2ba3bd12-8f75-4584-9d89-a90fc8af0f53%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9niKkctJhwi%3D8aYrEYfBya%3D-e%3D4_ShMvfpzSPJwnzL%2Bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I need to query parent which has not child attach to it

[Ayache Khettar]

Hi 

The below query search for all entries in a index which has child with the 
given stepUUID. I am interested to search for all entries which don't have 
child item populated yet. I was looking at using 'must_not" but not sure 
how to construct the query. Your help is very much appreciated.

Regards,

akhettar



{
"query": {
"match": {
"stepUUID": "fd7a5c5d-5254-4941-9c8a-e19a39be86b0"
}
},
"filter": {
"has_child": {
"type": "redelivery",
"query" : {
"match": {
"stepUUID": {
"query" : "fd7a5c5d-5254-4941-9c8a-e19a39be86b0",
"operator" : "and"
}
}
}
}
}
}

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b02a052c-bb25-432a-abd1-aae8f77cac15%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Percollation limits

[Luca Cavanna]

Hi Maciej,
what you describe doesn't sound insane, just make sure you use proper 
filtering as much as you can to limit the number of queries you execute 
when percolating each document.
Also, with the percolator available since 1.0 you can scale out just by 
adding more nodes and have the percolator queries distributed over multiple 
shards. That means that if you were to reach the limit of a single shard 
you could always scale out.

On Friday, June 13, 2014 5:15:05 PM UTC+2, Maciej Dziardziel wrote:
>
> Hi
>
> I wanted to ask those who use percollation: how many queries are you 
> percollating?
>
> I need to set up some equivalent of percollation for about 100k queries. 
> With some filtering
> probably up to 10k would actually had to be checked for each new document.
> Is the idea of using ES percollations for that insane?
>
> Thanks
> Maciej Dziardziel
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e81b15ba-f9c0-4ffa-bbb8-f644823c8367%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Losing data after Elasticsearch restart

[Alexander Reelsen]

Hey,

without stack traces it is pretty hard to see the actual problem, do you
have them around (on one node this exception has happened, so it should
have been logged into the elasticsearch logfile as well). Also, you should
really upgrade if possible, as releases after 0.90.2 have seen many many
improvements.


--Alex


On Mon, Jun 9, 2014 at 4:15 AM, Rohit Jaiswal 
wrote:

> Hello Everyone,
>  We lost data after restarting Elasticsearch
> cluster. Restarting is a part of deploying our software stack.
>
>  We have a 20-node cluster running 0.90.2 and we
> have Splunk configured to index ES logs.
>
>  Looking at the Splunk logs, we could find the
> following *error a day before the deployment* (restart) -
>
> [cluster.action.shard ] [Rictor] sending failed shard for 
> [c0a71ddaa70b463a9a179c36c7fc26e3][2], node[nJvnclczRNaLbETunjlcWw], [R], 
> s[STARTED], reason
> [Failed to perform [bulk/shard] on replica, message 
> [RemoteTransportException; nested: ResponseHandlerFailureTransportException; 
> nested: NullPointerException; ]]
>
> [cluster.action.shard ] [Kiss] received shard failed for 
> [c0a71ddaa70b463a9a179c36c7fc26e3][2], node[nJvnclczRNaLbETunjlcWw], [R], 
> s[STARTED], reason
> [Failed to perform [bulk/shard] on replica, message 
> [RemoteTransportException; nested: ResponseHandlerFailureTransportException; 
> nested: NullPointerException; ]]
>
>
>
>   Further,* a day after the deploy,* we see the
> same errors on another node -
>
>
>
> [cluster.action.shard ] [Contrary] received shard failed 
> for [a58f9413315048ecb0abea48f5f6aae7][1], node[3UbHwVCkQvO3XroIl-awPw], [R], 
> s[STARTED], reason
> [Failed to perform [bulk/shard] on replica, message 
> [RemoteTransportException; nested: ResponseHandlerFailureTransportException; 
> nested: NullPointerException; ]]
>
>
>  *Immediately next, the following error is seen*. This error is 
> seen repeatedly on a couple of other nodes as well -
>
>  failed to start shard
>
>  [cluster.action.shard ] [Copperhead] sending failed 
> shard for [a58f9413315048ecb0abea48f5f6aae7][0], 
> node[EuRzr3MLQiSS6lzTZJbiKw], [R], s[INITIALIZING],
>  reason [Failed to start shard, message 
> [RecoveryFailedException[[a58f9413315048ecb0abea48f5f6aae7][0]: Recovery 
> failed from [Frank Castle][dlv2mPypQaOxLPQhHQ67Fw]
>  [inet[/10.2.136.81:9300]] into 
> [Copperhead][EuRzr3MLQiSS6lzTZJbiKw][inet[/10.3.207.55:9300]]]; nested: 
> RemoteTransportException[[Frank Castle]
>  
> [inet[/10.2.136.81:9300]][index/shard/recovery/startRecovery]]; nested: 
> RecoveryEngineException[[a58f9413315048ecb0abea48f5f6aae7][0] Phase[2] 
> Execution failed];
>  nested: 
> RemoteTransportException[[Copperhead][inet[/10.3.207.55:9300]][index/shard/recovery/translogOps]];
>  nested: InvalidAliasNameException[[a58f9413315048ecb0abea48f5f6aae7]
>
> * Invalid alias name 
> [fbf1e55418a2327d308e7632911f9bb8bfed58059dd7f1e4abd3467c5f8519c3], Unknown 
> alias name was passed to alias Filter]; ]]*
>
>
> *During this time, we could not access previously indexed documents.*
>  I looked up the alias error, looks like it is related to 
> https://github.com/elasticsearch/elasticsearch/issues/1198 (Delete By Query 
> wrongly persisted to translog # 1198),
>  but this should be fixed in ES 0.18.0 and, we are using 0.90.2, 
> so why is ES encountering this issue?
>
>  What do we need to do to set this right and get back lost data? 
> Please help.
>
> Thanks.
>
>
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/00e54753-ab89-4f63-a39e-0931e8f7e2f0%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8yrprZNCpzNqOiDzaoFwqh6Dth23OSc1byZe81P7Ba9w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: IllegalArgumentException[No type mapped for [43]], version 1.2.1

[Alexander Reelsen]

Hey,

it seems, as if writing into the translog fails at some stage (from a
complete birds eye view). Can you check your logfiles, if you ran into some
weird exceptions before that happens? Also, you did not run out of disk
space at any time when this has happened?


--Alex


On Fri, Jun 6, 2014 at 8:39 PM, Brooke Babcock 
wrote:

> In one part of our application we use Elasticsearch as an object store.
> Therefore, when indexing, we supply our own _id. Likewise, when accessing a
> document we use the simple GET method to fetch by _id. This has worked well
> for us, up until recently. Normally, this is what we get:
>
> curl -XGET 'http://127.0.0.1:9200/data-2014.06.06/key/test1?pretty=true'
> {
>   "_index" : "data-2014.06.06",
>   "_type" : "key",
>   "_id" : "test1",
>   "_version" : 1,
>   "found" : true,
>   "_source":{"sData":"test data 1"}
> }
>
>
> Now, we often encounter a recently indexed document that throws the
> following error when we try to fetch it:
>
> curl -XGET 'http://127.0.0.1:9200/data-2014.06.06/key/test2?pretty=true'
> {
>   "error":"IllegalArgumentException[No type mapped for [43]]",
>   "status":500
> }
>
>
>
> This condition persists anywhere from 1 to 25 minutes or so, at which
> point we no longer receive the error for that document and the GET succeeds
> as normal. From that point on, we are able to consistently retrieve that
> document by _id without issue. But, soon after, we will find a different
> newly indexed document caught in the same bad state.
>
> We know the documents are successfully indexed. Our bulk sender (which
> uses the Java transport client) indicates no error during indexing and
> we are still able to locate the document by doing an ids query, such as:
>
> curl -XPOST "http://127.0.0.1:9200/data-2014.06.06/key/_search?pretty=true
> " -d '
> {
>   "query": {
> "ids": {
>   "values": ["test2"]
> }
>   }
> }'
>
> Which responds:
> {
>"took": 543,
>"timed_out": false,
>"_shards": {
>   "total": 10,
>   "successful": 10,
>   "failed": 0
>},
>"hits": {
>   "total": 1,
>   "max_score": 1.0,
>   "hits": [ {
>  "_index": "data-2014.06.06",
>  "_type": "key",
>  "_id": "test2",
>  "_score": 1.0,
>  "_source":{"sData": "test data 2"}
>   } ]
>}
> }
>
>
> We first noticed this behavior in version 1.2.0. When we upgraded to
> 1.2.1, we deleted all indexes and started with a fresh cluster. We hoped
> our problem would be solved by the big fix that came in 1.2.1, but we are
> still regularly seeing it. Although our situation may sound like the
> routing bug introduced in 1.2.0, we are certain that it is not. This
> appears to be a significant issue with the translog - we hope the
> developers will be able to look at what may have changed. We did not notice
> this problem in version 1.1.1.
>
> Just in case, here is the mapping being used:
> curl -XGET 'http://127.0.0.1:9200/data-2014.06.06/key/_mapping?pretty=true
> '
> {
>   "data-2014.06.06" : {
> "mappings" : {
>   "key" : {
> "_all" : {
>   "enabled" : false
> },
> "properties" : {
>   "sData" : {
> "type" : "string",
> "index" : "no"
>   }
> }
>   }
> }
>   }
> }
>
>
> Thanks for your help.
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/20c45cf8-3459-47f5-8cc3-1e63c93b2c0c%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8-hdJrpjXpH3dRLzsvLLoT-75O5DCO_tm5SFe9tEvXFw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Bug in context suggester?

[Alexander Reelsen]

Hey,

the problem is, that the mapping of the context variable 'network_id' is
not a string. Thinking if we either should support this or reject it
directly and only support string fields in the suggester.

Added github issue
https://github.com/elasticsearch/elasticsearch/issues/6512


--Alex


On Fri, Jun 6, 2014 at 8:41 PM, Gilbert Chang 
wrote:

> I am observing an issue with field reference in context suggester.
> Consider the following use case:
>
> PUT /services/service/_mapping
> {
> "service": {
> "properties": {
> "name": {
> "type" : "string"
> },
> "network_id": {
> "type" : "long"
> },
> "suggest_field": {
> "type": "completion",
> "context": {
> "network": {
> "type": "category",
> "path": "network_id"
> }
> }
> }
> }
> }
> }
>
> PUT /services/service/1
> {
> "name": "knapsack",
> "network_id": 1,
> "suggest_field": {
> "input": ["knacksack", "backpack", "daypack"]
> }
> }
>
> POST services/_suggest?pretty'
> {
> "suggest" : {
> "text" : "k",
> "completion" : {
> "field" : "suggest_field",
> "size": 10,
> "context": {
> "network": 1
> }
> }
> }
> }
>
> The suggest query does not return any option. However, if I change the
> field type of network_id to 'string' (and not_analyzed), the suggest query
> works. Explicitly indexing the network_id as the context in the
> suggest_field works as well:
>
> PUT /services/service/1
> {
> "name": "knapsack",
> "network_id": 1,
> "suggest_field": {
> "input": ["knacksack", "backpack", "daypack"],
> "context": {"network_id": 1}
> }
> }
>
> Seems like a bug?
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/214f059d-e2e6-4a28-aaaf-7dc6c1e1bcad%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-oAgFRo8sRkNCD2X4jo6ZUdevPHmyHS1nzd7JTeK0mgA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Help needed: CentOS ES-Logstash-Kibana running, but 0 docs, 0 indices

['Robert AMS' via elasticsearch]

After renaming my cluster from 'elastic search-demo' to the default 
'elasticsearch' the connection between ES and Logstash automatically was 
fixed after a restart of ES.

Next question: Where do I have to change the name of the cluster, other 
than /etc/elasticsearch/elasticsearch.yml ?

Op zaterdag 14 juni 2014 10:34:42 UTC+2 schreef Robert AMS:
>
> Als POC I've setup a single machine with Logstash, Elasticsearch and 
> Kibana, But I am stuck: It is running, but no docs are getting in ES and 
> Kibana stays only a dark page. Can someone help me how to get the logs into 
> ES en searchable in Kibana?
>
> This is my machine. Just install a CentOS 6.5 (virtual) server, su to root 
> and copy-paste this to the command line and all will be installed:
>
> Pastebin: Install_ES_Logstash_Kibana_for_CentOS6.5 
>  ( http://pastebin.com/fpz5AQzd )
>
> Wat is missing?
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/199085c2-3a61-4455-b738-65e00fcb669c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: date_histogram not returning key_as_string

[Alexander Reelsen]

Hey,

can you create a full gist to reproduce this? Should work out of the box
(dont see anything wrong here), also on 1.1.0


--Alex


On Thu, Jun 5, 2014 at 8:06 PM, Tim Heikell 
wrote:

> Sorry for the noob question, but is there some setting I am missing? It's
> not clear to me why I'm not getting a key_as_string field in my results.
> I'm running v1.1.0, here is my search:
>
> GET /_all/_search
> {
>   "aggs": {
> "totalsByHour": {
>   "date_histogram": {
> "field": "sessionStartTime",
> "interval": "day",
> "format": "-MM-dd"
>   }
> }
>   }
> }
>
> and here are my results:
>
>"aggregations": {
>   "totalsByHour": {
>  "buckets": [
> {
>"key": 140166720,
>"doc_count": 15973
> },
> {
>"key": 140175360,
>"doc_count": 11996
> },
> {
>"key": 140184000,
>"doc_count": 14
> },
> {
>"key": 140192640,
>"doc_count": 4031
> }
>  ]
>   }
>}
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/0acca250-ab7f-48a8-a73a-648f7f544dcd%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_aj0JhVsn6e7qxSF6VuLG8%2BnE3x5RxONWWFuLstz8GkQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Logash 1.4.1, Windows Server 2008, Tomcat 7.0.30, stuck with localhost_access_log.2014-06-16.txt

[Антон Мацюк]

Done! Damn slashes! It should be as this, then glob search works fine: "/"

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9d2fb117-41ac-4616-9011-600c1d80c02e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation average value is not coming correct

[Alexander Reelsen]

Hey,

you are setting a post filter, which means, that the aggregations will work
without the range filter applied. You may want to use a filtered query and
move the filter inside the filter part of that particular query.


--Alex


On Thu, Jun 5, 2014 at 12:38 PM, Subhadip Bagui  wrote:

> Hi,
>
> I'm using the below code to get the average value of cpu_usage using
> aggregation. When I checked the output of cpu value individually and
> calculate the avg, it is not matching with the aggregation avg value. I'm
> using a boolquery along with rangeFilter here to get the data.
>
> Please help to identify the issue.
>
> *Code :*
> public static SearchResponse searchResultWithAggregation(String es_index,
> String es_type, List ipList) {
> logger.debug("inside method searchResultWithAggregation...");
> Client client = ESClientFactory.getInstance();
> logger.debug("got the elasticsearch client connection");
>
> BoolQueryBuilder bqb = QueryBuilders.boolQuery()
> .mustNot(QueryBuilders.termQuery("address", "10.203.238.140"));
>
> Iterator i = ipList.iterator();
> logger.debug("got the ip list as :" + ipList);
>
> while (i.hasNext()) {
> bqb.should(QueryBuilders.termQuery("address", i.next()));
> }
>
> String time = "now-30m";
> FilterBuilder fb = FilterBuilders.rangeFilter("@timestamp").from(time)
> .to("now");
>
> SearchResponse response = client
> .prepareSearch(es_index)
> .setTypes(es_type)
> .setQuery(bqb)
> .setPostFilter(fb)
> .addAggregation(
> AggregationBuilders.avg("cpu_average").field("value"))
> .setSize(100).execute().actionGet();
>
> System.out.println(response.toString());
>
> return response;
> }
>
> *Output :*
> {
>   "took" : 31,
>   "timed_out" : false,
>   "_shards" : {
> "total" : 5,
> "successful" : 5,
> "failed" : 0
>   },
>   "hits" : {
> "total" : 15,
> "max_score" : 1.7314732,
> "hits" : [ {
>   "_index" : "cpu_usage_metrics",
>   "_type" : "cpu_usage_metrics",
>   "_id" : "UQ9vquDGTQO8WedjgCcESA",
>   "_score" : 1.7314732, "_source" :
> {"status":0,"occurrences":1,"value":"1","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:23:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
> -s Aricloud.`hostname -s`.cpu_usage"}
> }, {
>   "_index" : "cpu_usage_metrics",
>   "_type" : "cpu_usage_metrics",
>   "_id" : "EMT85ZKcS3OuoDmHgcSEjw",
>   "_score" : 1.7314732, "_source" :
> {"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:25:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
> -s Aricloud.`hostname -s`.cpu_usage"}
> }, {
>   "_index" : "cpu_usage_metrics",
>   "_type" : "cpu_usage_metrics",
>   "_id" : "0Pf-XKZmTI-wpADuIVToFA",
>   "_score" : 1.7314714, "_source" :
> {"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:21:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
> -s Aricloud.`hostname -s`.cpu_usage"}
> }, {
>   "_index" : "cpu_usage_metrics",
>   "_type" : "cpu_usage_metrics",
>   "_id" : "Pdn5h2gGRsK0hL2DKj0ZjA",
>   "_score" : 1.7314714, "_source" :
> {"status":0,"occurrences":1,"value":"2","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:27:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
> -s Aricloud.`hostname -s`.cpu_usage"}
> }, {
>   "_index" : "cpu_usage_metrics",
>   "_type" : "cpu_usage_metrics",
>   "_id" : "5_mloLYMSgKRb_lnH7pqGQ",
>   "_score" : 1.7314714, "_source" :
> {"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:33:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
> -s Aricloud.`hostname -s`.cpu_usage"}
> }, {
>   "_index" : "cpu_usage_metrics",
>   "_type" : "cpu_usage_metrics",
>   "_id" : "xjBgO2cXTH-DIQoNpIRnBA",
>   "_score" : 1.7314714, "_source" :
> {"status":0,"occurrences":1,"value":"4","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:35:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
> -s Aricloud.`hostname -s`.cpu_usage"}
> }, {
>   "_index" : "cpu_usage_metrics",
>   "_type" : "cpu_usage_metrics",
>   "_id" : "0sclBpwcRQmfyKklXPJbow",
>   "_score" : 1.7314694, "_source" :
> {"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:29:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
> -s Aricloud.`hostname -s`.

Re: Geo Distance Facet - ElasticsearchParseException

[Munjal Dhamecha]

Oops! That one was very silly overlook mistake. Thank you for pointing it 
out.

Yes, I moved to aggregations with few failed tries on facets.

On Monday, 16 June 2014 17:57:34 UTC+5:30, Alexander Reelsen wrote:
>
> Hey,
>
> your ranges specification is broken, you need to use an array of objects. 
> Also, please try to use aggregations if possible, as they are the successor 
> of facets.
>
>
> --Alex
>
>
> On Thu, Jun 5, 2014 at 10:37 AM, Munjal Dhamecha  > wrote:
>
>> Hello All,
>>
>> I've been facing problem with geo_distance facet since few hours.
>>
>> Error is: ElasticsearchParseException[field must be either 'lat', 'lon' 
>> or 'geohash']
>>
>> I am not sure, if this is bug or I am making a silly mistake here. Please 
>> guide in right direction.
>>
>> Gist: https://gist.github.com/mastermunj/a58298bc56615b54d643
>>
>> Tested On: 1.2.0, 1.2.1
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/6ec869ad-c32a-4e85-ab60-131bca8623fb%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/bf1cc3b1-f900-4e65-a113-cf22a6778f56%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Object property vs array index

[Alexander Reelsen]

Hey,

that doesnt work like you want it to work. I am not sure, if I understood
the intention of your query. Do you want to count on all first elements of
all documents or do you just want to filter for that term? If its the
first, maybe a script aggregations can help you, that just returns the
first field of that array.


--Alex


On Thu, Jun 5, 2014 at 12:07 PM,  wrote:

> When a field contains an object, in a terms aggregation I can specify a
> specific object property that contains the terms I want to use eg
>
> {
> "terms": {
> "field": "fieldName.propertyContainingTerms"
> }
> }
>
> So with a array type field that contains a list of strings ["first",
> "second", "third"] I should be able to do the same via array indices
>
> {
> "terms": {
> "field": "fieldName[0]"
> }
> }
>
> But this doesn't work. Am I using the wrong syntax or is this just not
> possible?
>
> My current solution is to use objects with integers converted to strings
> as the property keys eg "fieldName.0", it works but feels wrong.
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/002d9cd8-04b6-4dec-91a0-d49f5f187a2d%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM88H%2B9fnXiLKpx5HESfN-N2L5oU4804J8_DF75ygAfp5A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Geo Distance Facet - ElasticsearchParseException

[Alexander Reelsen]

Hey,

your ranges specification is broken, you need to use an array of objects.
Also, please try to use aggregations if possible, as they are the successor
of facets.


--Alex


On Thu, Jun 5, 2014 at 10:37 AM, Munjal Dhamecha 
wrote:

> Hello All,
>
> I've been facing problem with geo_distance facet since few hours.
>
> Error is: ElasticsearchParseException[field must be either 'lat', 'lon' or
> 'geohash']
>
> I am not sure, if this is bug or I am making a silly mistake here. Please
> guide in right direction.
>
> Gist: https://gist.github.com/mastermunj/a58298bc56615b54d643
>
> Tested On: 1.2.0, 1.2.1
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/6ec869ad-c32a-4e85-ab60-131bca8623fb%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_wd8EctDuvpA7T-89O7P5Pjhk2oYhgqh5tAcDMMJL%2BQg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch filter query with time range using java api

[Subhadip Bagui]

Hi Alex,

Yes I tried that and it's working. Thanks :)

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8addde47-a57b-4a4a-aa7a-68b419bcf90e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

What would be the best data structure for efficient search on this data?

[Beetel]

I am using elasticsearch 1.2.0.

I have around 3 million documents which can be of 2 types - 'products' or 
'categories'. Currently I am usingscript filters 

 to 
check whether category_name1 exists in 'belongs_to_categories' array.

I have to reindex the data, so I have the opportunity to modify the 
structure of the data. Any ideas how should I structure my data so that 
search based on 'belongs to this category' are fastest?

My data is currently in the form -

"_type": "product",
"_source": {
"id": "id1",
..., 
"name": "product_name",
"belongs_to_categories": [category_name1, category_name2]
}

"_type": "category",
"_source": {
"id": "1ID",
"name": "category_name1"
...
}

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/745afc0b-7629-4845-afbe-173bffeb0842%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: MapperParsingException on data that should not be parsed but caught by my plugin

[Alexander Reelsen]

Hey,

first you should really upgrade ES, 0.90.3 is not the freshest one and
there have been many improvements.
Second, logging this as DEBUG makes a lot of sense, as otherwise you
logfiles might explode, just because you are indexing a lot of bogus
documents. You source of truth should not be the logfile, but the error
message, which should have been returned upon indexing this JSON document.

It seems, as if the data you sent is not valid UTF-8, but some other
encoding...


--Alex



On Wed, Jun 4, 2014 at 6:30 PM, Laurent T.  wrote:

> Hi,
>
> I've just activated DEBUG mode on my ES logs and i'm seeing this kind of
> exception:
>
> [2014-06-04 15:50:03,539][DEBUG][action.index ] [Supercharger]
> [myplugin][0], node[drCfkhlURn2Yz_SsM6bD3w], [P], s[STARTED]: Failed to
> execute [index {[myplugin][client1][69Pb9C_kT6CF6jdH06WbJw],
> source[{"value":"Cr�py-en-valois"}]}]
> org.elasticsearch.index.mapper.MapperParsingException: failed to parse
> [value]
>  at
> org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:396)
>  at
> org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:599)
>  at
> org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:467)
>  at
> org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:515)
>  at
> org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:457)
>  at
> org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:515)
>  at
> org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:457)
>  at
> org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:507)
>  at
> org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:451)
>  at
> org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:308)
>  at
> org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:211)
>  at
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:521)
>  at
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)
>  at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  at java.lang.Thread.run(Thread.java:744)
> Caused by: org.elasticsearch.common.jackson.core.JsonParseException:
> Invalid UTF-8 middle byte 0x70
>  at [Source: [B@2d1b0cf9; line: 1, column: 11]
>  at
> org.elasticsearch.common.jackson.core.JsonParser._constructError(JsonParser.java:1369)
>  at
> org.elasticsearch.common.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:599)
>  at
> org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._reportInvalidOther(UTF8StreamJsonParser.java:3004)
>  at
> org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._reportInvalidOther(UTF8StreamJsonParser.java:3011)
>  at
> org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._decodeUtf8_3fast(UTF8StreamJsonParser.java:2833)
>  at
> org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2135)
>  at
> org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._finishString(UTF8StreamJsonParser.java:2084)
>  at
> org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:270)
>  at
> org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:85)
>  at
> org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:107)
>  at
> org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:285)
>  at
> org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:385)
>  ... 15 more
>
> I'm wondering why this is happening and why this error is shown only in
> DEBUG mode.
> This request should actually be targeting my plugin that would JSON-decode
> it using Google Gson.
>
> Is ES doing anything else before forwarding the request to the plugin ?
> We're using version 0.90.3 of ES.
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/f6afec0d-24f5-4438-b5d4-3e2d6b87c5c6%40googlegroups.com
> 
> .
> For more options, vis

Re: elasticsearch filter query with time range using java api

[Alexander Reelsen]

Hey,

did you try something along the
lines FilterBuilders.rangeFilter("@timestamp").from("now").to("now - 5d") -
did that pose problems?

make sure you are reading this paragraph about date range filters:
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/filter-caching.html


--Alex


On Wed, Jun 4, 2014 at 1:43 PM, Subhadip Bagui  wrote:

> Hi,
>
> I have a document like below
>
> {
> "_index": "cpu_usage_metrics",
> "_type": "cpu_usage_metrics",
> "_id": "CKAAs1n8TKiR6FncC5NLGA",
> "_score": 1,
> "_source": {
>"status": 0,
>"occurrences": 1,
>"value": "33",
>"key": "vm.server2.cpu.usage",
>"client": "vm.server2",
>"@timestamp": "2014-06-03T20:18:19+05:30",
>"check_name": "cpu_usage_metrics",
>"address": "10.203.238.138",
>"command": "cpu-usage-metrics.sh"
> }
>  }
>
>  I want to do a filtered query with time range using java api like below
>
> "filter": {
> "range": {
> "@timestamp": {
>  "to": "now",
> "from": "now - 5mins"
> }
> }
> }
>
>
> Please suggest how to form the Filter in java api.
>
> Thanks,
> Subhadip
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/f596658e-5ec4-42ac-abc1-4f99416be101%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_YymPLyoy2nVmk9VAeq2RFHu7mJtufhBHCGd5Q6u9ZUw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Logash 1.4.1, Windows Server 2008, Tomcat 7.0.30, stuck with localhost_access_log.2014-06-16.txt

[Антон Мацюк]

Have this in logstash log itself:

{:timestamp=>"2014-06-16T15:01:50.817000+0300", 
:message=>"_discover_file_glob: 
c:\\apache\\logs\\localhost_access_log2.*.txt: glob is: []", 
:level=>:debug, 
:file=>"/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/watch.rb",
 
:line=>"117"}

It looks like logstash can't understand asterisk...

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3ee56a55-304e-425c-9237-e46c37383ea1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Prefix search on integer field

[Alexander Reelsen]

Hey,

not sure here, whats your expectation of the data being returned? If you
search for "1", should "1", "10", "111" be returned (otherwise a range
query would make more sense)? If so, it might make more sense to have an
additional multi field as a not_analyzed string and use that one for these
kind of queries...


--Alex


On Wed, Jun 4, 2014 at 11:13 AM, Simon Cast  wrote:

> Hi,
>
> I'm trying to use a prefix search on an integer field that is stored and
> not analysed. From the documentation I would have expected that sending 1
> would return all numbers starting with 1 but that doesn't seem to be the
> case.
>
> Does the prefix search work on integer fields?
>
> Regards,
>
> Simon
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/110f1887-c6e0-4f77-9ece-689f0a42b306%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_uDVczuDkiBHD9ZZgx9iR7TQKRGWZZu1%2B73TQhg4kgNg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Index template requires settings object even if its value is empty

[Alexander Reelsen]

Hey,

which ES version are you using? Seems to work with the latest version. You
can also use the index template API, so you do not have to fiddle with
local files (and copy them when adding new nodes).

PUT _template/automap
{
  "template": "*",
  "mappings": {
"_default_": {
  "numeric_detection": true,
  "properties": {
"message": {
  "type": "string"
},
"host": {
  "type": "string"
},
"@version": {
  "type": "string"
}
  }
}
  }
}



--Alex


On Tue, Jun 3, 2014 at 5:57 PM, Brian  wrote:

> I am not sure if this is a problem or if it's OK.
>
> Working with the ELK stack I have switched direction, and instead of
> locking down the Elasticsearch mappings I am now using its automatic
> mapping functions. And by adding the following JSON template definition to
> the /*path.to.config*/templates/automap.json file I can get numeric
> fields automatically correctly mapped even though logstash always emits
> their values as strings ("45.6" instead of 45.6). Very nice!
>
> {
>   "automap" : {
> "template" : "*",
> *"settings" : { },*
> "mappings" : {
>   "_default_" : {
> "numeric_detection" : true,
> "properties" : {
>   "message" : {"type" : "string"},
>   "host" : {"type" : "string"},
>   "@version" : {"type" : "string"}
> }
>   }
> }
>   }
> }
>
> When I removed the *"settings":{}* entirely, it was as if the template
> did not exist; the numeric detection was not enabled and all string values
> were seen as strings even if they contained numbers. Because all of the
> settings are being controlled within elasticsearch.yml and not the template
> (e.g. number of shards, number of replicas, and so on), eliminating the
> settings from the template is desired, even if I have to leave it in but
> set its value to the empty JSON object.
>
> If this is the way it's supposed to work, that's OK. But I couldn't find
> anything in the documentation about it, and just wanted to get a
> verification either way.
>
> Thanks!
>
> Brian
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/ff4afb8e-c3e4-4772-aa48-bd6a651c78e8%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9g43jE8cKehZLNxBNbFZs2jrpw%3DRnbC5b%3DO10uJSr5Pg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch Node.Client Options

[Alexander Reelsen]

Hey,

client nodes do execute a reduce phase (merging all the results from a
search in order to find out the top-n results), and if you do deep
pagination this can take some memory. There is no data stored on a client
node (as node.data property is always set to false).

Do you have a stack trace of the exception, so one can see where the OOM
happens?


--Alex


On Tue, Jun 3, 2014 at 8:58 PM, VB  wrote:

> We have client, data and master node configurations.
>
> And we are seeing lots of OutOfMemory errors on node.client nodes.
>
> One of the thing which we have some doubt is, node.client caches data for
> some reason? I am hoping not.
>
> If yes then what are the ways we can control this caching so client does
> not cache any data.
>
> And in general what are setting options which we can control through yml
> for client nodes (node.client).
>
> And what all operations happens on Node.Client when search with some query.
>
> Regards,
> VB.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/03191c28-185f-4278-baaa-93e779477dd2%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9_qsrYQvrgk5dnGeyHhAL5e_a7QC%3Dd25DaYbyQwOSafA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Configure connections using TransportClient

[Alexander Reelsen]

Hey,

those configuration parameters still exist (there are more for ping and
recovery). Wondering why exactly you want to change those? Do you have any
specific problems going on?


--Alex


On Tue, Jun 3, 2014 at 7:47 AM, Srinath C  wrote:

> Hi,
> I wasn't able to figure out how to configure the number of connections
> that are opened up by a TransportClient to an ElasticSearch cluster. There
> was some information in one of the discussions
>  but
> they don't seem to work with ElasticSearch 1.1.1.
> I can see 14 connections established from one TransportClient instance
> to one ElasticSearch instance by default.
>
> Thanks,
> Srinath.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAHhx-GJZi6WeSNPTDC%2BW7jG0cKNdRuy64u5DOVA87WXnTRZAuQ%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9BF4y8tX4Dvvg0NJPCeWmX%2B2JzcGJAnS%3D9%3D-HjYmapjg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Term Suggester Usage

[Klaus Stadler]

Hi Alex,

thanks for the clarification about the term suggester.
That leaves either the phrase suggester or autocomplete suggester.
The field I want to find suggestions for usually has a lot of words.
So I'd prefer to complete only the current word, not the whole content of 
the field.
Is this possible with the autocomplete suggester?

Which analyzers are best suited for this use case?

-- Klaus

Am Montag, 16. Juni 2014 13:45:56 UTC+2 schrieb Alexander Reelsen:
>
> Hey,
>
> the term suggester takes a query like "search is awesome" and returns 
> suggestions for each of the terms, so three different suggestions are 
> returned here. You maybe want to take a look at the completion suggester, 
> see
>
> http://www.elasticsearch.org/blog/you-complete-me/
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-suggesters-completion.html
>
>
> --Alex
>
>
> On Tue, Jun 3, 2014 at 11:13 AM, Klaus Stadler <
> klaus@united-internet-media.de > wrote:
>
>> Hello,
>>
>> is it correct that Term Suggester is what is closes to Google style 
>> autocompletion?
>> If so, how do you use the term suggester?
>> Unfortunately, I can't find any examples in the ES documentation.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/d2bc5e70-37e9-41e6-bc30-bf77ae2ee4b9%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ab937435-0785-40bb-ac1e-32546cd807bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to debug aggregation problems?

[Alexander Reelsen]

Hey,

if you think this is a bug, please try to create a reproducible example and
file a github issue. See http://www.elasticsearch.org/help
If you are using the terms aggregation, you might want to read this one as
well before filing a bug:
https://github.com/elasticsearch/elasticsearch/issues/1305


--Alex


On Tue, Jun 3, 2014 at 11:51 AM, mooky  wrote:

>
> I have an aggregation (filter + daterange) that is giving me an incorrect
> result (the count for a bucket is 2 when I expect 1) - but when I search
> using the same* criteria, I get only the 1 result I expect.
> I am wondering what techniques there are for trying to debug the
> aggregation - to try figure out why I get a count of 2 instead of 1.
>
> * The criteria are almost the same - there is a process of turning one of
> the buckets of the date-range aggregation into a date range filter - but as
> it happens, this is not the part that is generating the incorrect results.
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/49b01a36-bc17-4c59-b7d6-d100fe455ec5%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8v1iPOuJRxPf%2Brq1aO6o76yNvOk6EzSKwzcVTzuhLv2g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Term Suggester Usage

[Alexander Reelsen]

Hey,

the term suggester takes a query like "search is awesome" and returns
suggestions for each of the terms, so three different suggestions are
returned here. You maybe want to take a look at the completion suggester,
see

http://www.elasticsearch.org/blog/you-complete-me/
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-suggesters-completion.html


--Alex


On Tue, Jun 3, 2014 at 11:13 AM, Klaus Stadler <
klaus.stad...@united-internet-media.de> wrote:

> Hello,
>
> is it correct that Term Suggester is what is closes to Google style
> autocompletion?
> If so, how do you use the term suggester?
> Unfortunately, I can't find any examples in the ES documentation.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/d2bc5e70-37e9-41e6-bc30-bf77ae2ee4b9%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8YvrnBkXBbbLnvGgJ9qDxno17XaxQ-LBFH1cZ5g_BkYw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Detective wanted: Green cluster but query timeout unless changing preference to local.

[Alexander Reelsen]

Hey,

you should check your logfiles on each of the nodes. Maybe they
disconnected from each other (and reconnect all the time), so that you get
problems when you execute your requests round robin. Using curl from one
node to another does not show any problems? They are reachable all the time
(at low latency)?


--Alex


On Mon, Jun 2, 2014 at 8:22 PM, arthurX  wrote:

> I had debugging this problem for so long time now. I tweaked every
> relevant setting but just cannot get it to work. I really hope someone with
> knowledge about Elasticsearch networking can give me some help.
>
> So I got two machines in the same data-center, but communicate to each
> other by public ip. (network.publish_host is set)
> The symptoms are:
> 1: Cluster health is green showing 2 nodes
> 2: When the slave node initially started, all queries are fine. After some
> time, query would not work unless adding "preference=_local". This is true
> for HTTP curl for both machine.
> 3: When I shut down the slave node, soon the master node would work fine
> without the "_local" suffix.
>
> No special error is shown.
> Please feel free to share any ideas you have about this case.
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/0a2c60d6-56d5-4d8d-8549-3f61aaa46cd2%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_YeNPm3dOwEwSHcqGwjxx-dq18KVgoF8Ayhpkv26w3gw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Could not lock IndexWriter isLocked [false]

[Alexander Reelsen]

Hey,

what elasticsearch version are you using? Judging from the directory I dont
think you are using NFS, right? Are you running multiple instances locally?
Have you shutdown elasticsearch properly so that no other instance is
lingering around (you can use jps or or to check)


--Alex


On Mon, Jun 2, 2014 at 1:16 PM, Fatih Karatana  wrote:

> I try to create an index in a couple of seconds and i got this:
> [2014-06-02 14:10:14,414][WARN ][index.engine.internal] [shardicaprio]
> [myindex][0] Could not lock IndexWriter isLocked [false]
>
>
> And here is full stack trace:
>
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out:
> NativeFSLock@/var/lib/elasticsearch/data/shardicaprio/nodes/0/indices/
> myindex/0/index/write.lock
> at org.apache.lucene.store.Lock.obtain(Lock.java:84)
> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:702
> )
> at org.elasticsearch.index.engine.internal.InternalEngine.
> createWriter(InternalEngine.java:1388)
> at org.elasticsearch.index.engine.internal.InternalEngine.start(
> InternalEngine.java:256)
> at org.elasticsearch.index.shard.service.InternalIndexShard.
> postRecovery(InternalIndexShard.java:684)
> at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.
> recover(LocalIndexShardGateway.java:158)
> at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(
> IndexShardGatewayService.java:189)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [2014-06-02 14:10:14,533][WARN ][indices.cluster  ] [shardicaprio]
> [myindex][0] failed to start shard
> org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [
> myindex][0] failed recovery
> at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(
> IndexShardGatewayService.java:248)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.elasticsearch.index.engine.EngineCreationFailureException:
> [myindex][0] failed to create engine
> at org.elasticsearch.index.engine.internal.InternalEngine.start(
> InternalEngine.java:258)
> at org.elasticsearch.index.shard.service.InternalIndexShard.
> postRecovery(InternalIndexShard.java:684)
> at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.
> recover(LocalIndexShardGateway.java:158)
> at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(
> IndexShardGatewayService.java:189)
> ... 3 more
> Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain
> timed out: NativeFSLock@/var/lib/elasticsearch/data/shardicaprio/nodes/0/
> indices/myindex/0/index/write.lock
> at org.apache.lucene.store.Lock.obtain(Lock.java:84)
> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:702
> )
> at org.elasticsearch.index.engine.internal.InternalEngine.
> createWriter(InternalEngine.java:1388)
> at org.elasticsearch.index.engine.internal.InternalEngine.start(
> InternalEngine.java:256)
> ... 6 more
> [2014-06-02 14:10:14,536][WARN ][cluster.action.shard ] [shardicaprio]
> [myindex][0] sending failed shard for [myindex][0], node[
> kHOedr2wQpa3DSZj81ep_A], [P], s[INITIALIZING], indexUUID [29Uf2hH4S2-
> FJf1LnNrM0A], reason [Failed to start shard, message [
> IndexShardGatewayRecoveryException[[myindex][0] failed recovery]; nested:
> EngineCreationFailureException[[myindex][0] failed to create engine];
> nested: LockObtainFailedException[Lock obtain timed out: NativeFSLock@/var
> /lib/elasticsearch/data/shardicaprio/nodes/0/indices/myindex/0/index/write
> .lock]; ]]
> [2014-06-02 14:10:14,536][WARN ][cluster.action.shard ] [shardicaprio]
> [myindex][0] received shard failed for [myindex][0], node[
> kHOedr2wQpa3DSZj81ep_A], [P], s[INITIALIZING], indexUUID [29Uf2hH4S2-
> FJf1LnNrM0A], reason [Failed to start shard, message [
> IndexShardGatewayRecoveryException[[myindex][0] failed recovery]; nested:
> EngineCreationFailureException[[myindex][0] failed to create engine];
> nested: LockObtainFailedException[Lock obtain timed out: NativeFSLock@/var
> /lib/elasticsearch/data/shardicaprio/nodes/0/indices/myindex/0/index/write
> .lock]; ]]
>
> I have no memory overloading, my heapsize is fine, but CPU. CPU gets
> overload even more than 100% of usage. I tried to recover index, delete
> index, recreate index but it tells the same thing every time. I could not
> figure it out that what causes this.
>
> Any idea?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this grou

Logash 1.4.1, Windows Server 2008, Tomcat 7.0.30, stuck with localhost_access_log.2014-06-16.txt

[Антон Мацюк]

Hi!
I have stuck with logfile initialization in Windows using Logstash. I want 
to make logstash read localhost_access_log, but seems that Logstash can't 
understand (or maybe I am somewhere wrong) my path that I try to pass in 
config-file.
input section contains this:
 file {
codec => multiline {
  pattern => "^\s"
  what => "previous"
}
discover_interval => 15 # number (optional), default: 15
path => ["c:\apache\logs\foolog.log"] # array (required)
start_position => "end" # string, one of ["beginning", "end"] 
(optional), default: "end"
stat_interval => 1 # number (optional), default: 1
tags => "foo-mon"  # string (optional)
type => "fooobar"
  }
and contents of c:\apache\logs\foolog.log logfile appear in elastic.

But with C:\Apache\logs\localhost_access_log.2014-06-16.txt there is a 
problem - Logstash doesn't want to read this file.
Configuration is like this (we also tried to add codec as above):
  file {
discover_interval => 5 # number (optional), default: 15
#exclude => ... # array (optional)
path => ["c:\apache\logs\localhost_access_log.*.txt"] # array (required)
start_position => "end" # string, one of ["beginning", "end"] 
(optional), default: "end"
stat_interval => 1 # number (optional), default: 1
tags => "foo-mon"  # string (optional)
type => "access"
}

Please, can someone help?

What we also tried:
if I rename localhost_access_log.2014-06-16.txt to localhost_access_log.txt 
(I also changed name in config), started logstash and changed contents of 
localhost_access_log.txt - it finds file and writes to elastic.
Renamed to localhost_access_log.2014.txt and used wildcard 
localhost_access_log.*.txt - not working, so this is not a minus character. 
:(

C:\logstash\lib\logstash\version.rb says
LOGSTASH_VERSION = "1.4.1"

Someone have any ideas why wildcard is not working?
Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8391119b-b30d-49f2-8133-1b46e4cc721c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Query Performance

[ravimbhatt]

Hi All, 

I am trying to improve my ES query performance. The goal is to get response 
times for 3 related queries under a second!. In my test i have seen 90th 
percentile response time (*took time*) for combined 3 queries to be ~1.8 
seconds. Here are the details: 

*Cluster*: 
- 5 Machines, 5 Shards, Currently on m3.2xlarge. (Had started with less 
powerful boxes and went up one by one, started from m3.large)
- 4 indexes. 
 - one index with *~90 million* recrods (total *19.3 GB *on all shards
*.*)
 - one with *~24 million* (total *6GB* on all shards.)
 - Other two are in 780K and 340K ( total *160MB* and *190MB*)
- All *fields* in the larger indexes are *integers*.
- Record size is small-ish.
- indexes are *compressed*. 
- I have given *15 GB to ES* instances. 
- Indexes are stored on *EBS* volumes. Each instance has *250GB* volume 
with it. (Keeping SSDs as last resort) 

The indexes are not changing (for now, in future they would change once a 
day). So no indexing is taking place while we query. *Therefore*, I have 
tried things like *reducing number of segments* in the two larger indexes. 
That helped to a point. 

*Querying Technique*:

- use python ES client. 
- *3 small instance* forking *10 threads* at the same time. 
- Each thread would fire *3 queries* before reporting a time. 
- At time there would be *~100 concurren*t queries on the machines. settles 
around ~50-60. 
- I take *'took'* time from ES response to measure times. 
- I *discard 100 records* before measuring times. 
- A total of *5000 unique users* are used for which 3 ES queries would be 
fired. A total of *4900 users' times* are measured.  

*Observations*:

- RAM is never under stress. Well below 15 GB allotted. 
- CPU comes under strain, goes upto 85-95 region on all instances during 
the tests. 

*Queries*: 

*1. On an index with ~24 Million records*: 

res = es.search( index="index1", 
body={"query":{"bool":{"must":[{"term":{"cid":value}}]}}}, sort=[ 
"source:desc", "cdate:desc" ], size=100, fields=["wiid"], _source="true")

i parse results of these queries to get certain fields out and pass on to 
the 2nd query. Lets call those fields as: *q1.field1* and *q2.field2*

*2. On an index with ~90 million records:*

res1 = es.search(index="index2", 
body={"query":{"filtered":{"filter":{"bool":{"must":{"terms":{"*col_a*":
*q1.field1*}},"must_not":{"terms":{"*col_b*":*q1.field1*
}},"aggs":{"i2B":{"terms":{"field":"*col_b*", "size": 1000 
,"shard_size":1, "order" : { "mss.sum":"desc"} 
},"aggs":{"mss":{"stats":{"script":"ca = _source.*col_a*; 
index=wiids.indexOf(ca); sval=0; if(index!=-1) sval=svalues.get(index); 
else sval=-1; return _source.*col_x**sval; ","params":{"wiids":*q1.field1*
,"svalues":*q1.field2*}}},"simSum":{"stats":{"script":"return _source.
*col_x* "}}, size=1)

- it uses *filtered query*.
- uses *2 aggregations*
- uses *script in aggregation*.  
- use *shard_size* 

Again, i parse results and get a filed out. Lets call that field as: 
*q2.field1*

3. *On an index with ~340K records:*

 res2 = es.search(index="index3", body= { "query" :  { "filtered" : { 
"query":{ "terms":{ "wiid":*q2.field1*  }  }, "filter" : { "bool" : { 
"must" : [ {  "range" : {"isInRange": { "gte" : 10  } } } , { "term" : { 
"isCondA" : "false" } } , { "term" : { "isCondB" : "false"} }, { "term" : { 
"isCondC" : "false" }  }  ]  }  } } } }   ,  size=1000)

Please let me know if any other information would help you help me. 

Query 2 above is doing aggregations and using a custom script. This is 
where times reach few seconds, like 2-3 seconds or even 4+ seconds at 
times. 

I can move to a high end CPU machine and may be the performance would 
improve. Wanted to check if there is anything else that i am missing. 

Thanks!
Ravi

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9148ddfb-1a72-49db-b716-f2f9405392e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Help needed: CentOS ES-Logstash-Kibana running, but 0 docs, 0 indices

['Robert AMS' via elasticsearch]

Tnx.
Maybe there is an connection to my other problem: No index created: 
IndexCreationException: failed to create index 

(For that post I might better go to the Logstash mailinglist too, but I 
think it might be an ES config problem.)

Op maandag 16 juni 2014 12:27:08 UTC+2 schreef Mark Walkom:
>
> You might want to take this over to the Logstash mailing list instead as 
> that appears to be the problem.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>  
>
> On 16 June 2014 17:39, 'Robert AMS' via elasticsearch <
> elasti...@googlegroups.com > wrote:
>
>>
>> 
>> This did not help. Ik paramedic it looks as if everything is running, 
>> except there are no docs (and indices) coming in ES. Seen screenshot.
>>
>> Op zondag 15 juni 2014 02:11:09 UTC+2 schreef Mark Walkom:
>>>
>>> Try swapping the host in your LS output to the actual IP of VM.
>>>
>>> Regards,
>>> Mark Walkom
>>>
>>> Infrastructure Engineer
>>> Campaign Monitor
>>> email: ma...@campaignmonitor.com
>>> web: www.campaignmonitor.com
>>>  
>>>
>>> On 14 June 2014 18:34, 'Robert AMS' via elasticsearch <
>>> elasti...@googlegroups.com> wrote:
>>>
 Als POC I've setup a single machine with Logstash, Elasticsearch and 
 Kibana, But I am stuck: It is running, but no docs are getting in ES and 
 Kibana stays only a dark page. Can someone help me how to get the logs 
 into 
 ES en searchable in Kibana?

 This is my machine. Just install a CentOS 6.5 (virtual) server, su to 
 root and copy-paste this to the command line and all will be installed:

 Pastebin: Install_ES_Logstash_Kibana_for_CentOS6.5 
  ( http://pastebin.com/fpz5AQzd )

 Wat is missing?


  -- 
 You received this message because you are subscribed to the Google 
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/ee7d3476-5551-40d2-aef5-28276e7783bd%
 40googlegroups.com 
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/46611243-97d3-4601-9045-1e27af9804be%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e39be043-7d23-48d2-8edb-873352422ff1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

No index created: IndexCreationException: failed to create index

['Robert AMS' via elasticsearch]

Logstash: 1.4
Elasticsearch: 1.2


When trying the examples of 'Logstash 10-minute Tutorial 
' step 
3 goes wrong. First, the problem is 'embedded' is not accepted, resulting 
in Java-errors. So I changed the output a little bit:
output {
  stdout {
codec => rubydebug
  }
  elasticsearch_http {
host => "localhost"
  }
}

But now, the errors and warnings keep flushing over my screen, which can 
only be stopped by another shell. What caused the problem, I have no idea.

If I didn't misread, the problem is it failed to create an index.

# bin/logstash agent -f hello-search.conf
Using milestone 2 output plugin 'elasticsearch_http'. This plugin should be 
stable, but if you see strange behavior, please let us know! For more 
information on plugin milestones, see 
http://logstash.net/docs/1.4.1/plugin-milestones {:level=>:warn}
testing
{
   "message" => "testing",
  "@version" => "1",
"@timestamp" => "2014-06-16T11:03:11.013Z",
  "type" => "human",
  "host" => "localhost.localdomain"
}
Error writing (bulk) to elasticsearch {:response=>#"application/json; 
charset=UTF-8", "content-length"=>"213"}>, @body=, @status=500, @reason="", 
@logger=#, 
@data={}, @metrics=#, @metrics={}, 
@metrics_lock=#>, @subscribers={}, @level=:info>, 
@version=1.1>, 
:response_body=>"{\"error\":\"IndexCreationException[[logstash-2014.06.16] 
failed to create index]; nested: NoClassDefFoundError[Could not initialize 
class org.elasticsearch.index.codec.postingsformat.PostingFormats]; 
\",\"status\":500}", :request_body=>"", :level=>:error}
Failed to flush outgoing items {:outgoing_count=>1, :exception=>RuntimeError
, :backtrace=>["/opt/logstash/lib/logstash/outputs/elasticsearch_http.rb:240:in 
`post'", "/opt/logstash/lib/logstash/outputs/elasticsearch_http.rb:213:in 
`flush'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:219:in
 
`buffer_flush'", "org/jruby/RubyHash.java:1339:in `each'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:216:in
 
`buffer_flush'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:159:in
 
`buffer_receive'", 
"/opt/logstash/lib/logstash/outputs/elasticsearch_http.rb:191:in 
`receive'", "/opt/logstash/lib/logstash/outputs/base.rb:86:in `handle'", 
"(eval):21:in 
`initialize'", "org/jruby/RubyProc.java:271:in `call'", 
"/opt/logstash/lib/logstash/pipeline.rb:266:in 
`output'", "/opt/logstash/lib/logstash/pipeline.rb:225:in `outputworker'", 
"/opt/logstash/lib/logstash/pipeline.rb:152:in 
`start_outputs'"], :level=>:warn}
... (etcetera)

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/21e4eecd-cb31-4683-ad7e-67f6e46b88dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Creating a browse interface from ES

[joergpra...@gmail.com]

What about this:

- build author name index

- page size is static (e.g. 20)

- absolute position: you must index each author name with absolute position
info (sort author names before indexing, use a counter and increment it
while indexing)

- sort asc/desc works on author's name keyword analyzed field

- jump function: execute constant_score query, with an optional filterered
query of prefix query on author name keyword analyzed field (search for 'A'
jumps to author names with 'A', search for 'B' jumps to 'B' etc.)

- search function is trivial

- relative move: paging back and forth through the result is done by using
the absolute position info from the hits and the 'from' / 'size' ES
parameters, ignoring the filtered query (since this is used just for
jumping)

This is how I implement "register search"

Jörg



On Mon, Jun 16, 2014 at 5:01 AM, Robin Sheat  wrote:

>
> That's close, but a bit different from what I want. If I have an 'author'
> index, and I search for things starting with 'Smith', sorting A->Z, I want
> to be able to page back, and get the results that are closer to the start
> of the alphabet. That is to say, it should tell me the "Smith" is the 524th
> (e.g.) author entry across the whole index when sorted, then I can set up
> my results page so the user can page backwards. Or, if I could do a
> startswith search, and have a negative "from" so it looks backwards in the
> results...
>
> But that still won't work, as startswith doesn't give me a place in the
> index, it gives me a subset of the index restricted by the query.
>
> Essentially, I think that this would give me an authority searcher, when I
> want an authority browser. I _could_ browse through it starting at 'A', but
> I really need to be able to jump to a point in the middle and go
> backwards/forwards from there.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHrTGTXh5ADseC0CAXehsOTxrfOutGOZsgi01zo7pMu7w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch inserting issue

[Ciprian]

Hi,

  My application is reading data from a sqs and writtes the data in 
elasticsearch(alias ES) and mysql. We are writting in both for back-up(for 
the moment). This is done with 2 consumers with 5 workers. One worker is 
writing in mysql and ES. The app is written in java and supports inserting, 
deleting and updating documents(one by one and not bulk). The ES insertings 
and updatings are done with upsert(@update : a document may be updated if 
exists or inserted if it doesn't; @insert : insert the document or if the 
update is first done, we shouldn't do anything). 
  The ES is installed on a different server(ubuntu; 8gb Ram; 4 cpu) than 
mysql. 
   ES configuration :
/etc/elasticsearch/elasticsearch.yml
  bootstrap.mlockall: true
  ES_MIN_MEM: 4g
  ES_MAX_MEM: 4g
  http.max_initial_line_length : 48k
/etc/default/elasticsearch
  ES_HEAP_SIZE=4g
  MAX_LOCKED_MEMORY=unlimited
When I run in console 
   
   - ulimit  the answer is unlimited
   - ulimit -n the answer is 1024

   The code for upsert : 

client.prepareUpdate(indexListener, tableListener, suId)
.setScript("ctx._source." + columnName + "=\"" + 
columnValue + "\"")

.setConsistencyLevel(WriteConsistencyLevel.DEFAULT).setRetryOnConflict(10)
.setUpsert(map)
.execute().actionGet();

   Node configuration :

  *localhost:9200/_nodes?pretty*

{
  "cluster_name" : "test",
  "nodes" : {
"Aa-bbb" : {
  "settings" : {
"node" : {
  "name" : "testing"
},
"bootstrap" : {
  "mlockall" : "true"
},
"http" : {
  "max_initial_line_length" : "48k"
},
"ES_MIN_MEM" : "4g",
"name" : "testing",
"pidfile" : "/var/run/elasticsearch.pid",
"path" : {
  "data" : "/var/lib/elasticsearch",
  "work" : "/tmp/elasticsearch",
  "home" : "/usr/share/elasticsearch",
  "conf" : "/etc/elasticsearch",
  "logs" : "/var/log/elasticsearch"
},
"cluster" : {
  "name" : "test"
},
"ES_MAX_MEM" : "4g",
"config" : "/etc/elasticsearch/elasticsearch.yml"
  },
  "os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
  "vendor" : "Intel",
  "model" : "Xeon",
  "mhz" : 2800,
  "total_cores" : 4,
  "total_sockets" : 4,
  "cores_per_socket" : 32,
  "cache_size_in_bytes" : 25600
},
"mem" : {
  "total_in_bytes" : 7812386816
},
"swap" : {
  "total_in_bytes" : 0
}
  },
  "process" : {
"refresh_interval" : 1000,
"id" : ,
"max_file_descriptors" : 65535,
"mlockall" : true
  },
  "jvm" : {
"pid" : ,
"version" : "1.7.0_55",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "vers",
"vm_vendor" : "Oracle Corporation",
"start_time" : 1402687303832,
"mem" : {
  "heap_init_in_bytes" : 4294967296,
  "heap_max_in_bytes" : 4260102144,
  "non_heap_init_in_bytes" : 24313856,
  "non_heap_max_in_bytes" : 224395264,
  "direct_max_in_bytes" : 4260102144
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor 
Space", "CMS Old Gen", "CMS Perm Gen" ]
  },
  "thread_pool" : {
"generic" : {
  "type" : "cached",
  "keep_alive" : "30s"
},
"index" : {
  "type" : "fixed",
  "min" : 4,
  "max" : 4,
  "queue_size" : "200"
},
"get" : {
  "type" : "fixed",
  "min" : 4,
  "max" : 4,
  "queue_size" : "1k"
},
"snapshot" : {
  "type" : "scaling",
  "min" : 1,
  "max" : 2,
  "keep_alive" : "5m"
},
"merge" : {
  "type" : "scaling",
  "min" : 1,
  "max" : 2,
  "keep_alive" : "5m"
},
"suggest" : {
  "type" : "fixed",
  "min" : 4,
  "max" : 4,
  "queue_size" : "1k"
},
"bulk" : {
  "type" : "fixed",
  "min" : 4,
  "max" : 4,
  "queue_size" : "50"
},
"optimize" : {
  "type" : "fixed",
  "min" : 1,
  "max" : 1
},
"warmer" : {
  "type" : "scaling",
  "min" : 1,
  "max" : 2,
  "keep_alive" : "5m"
},
"flush" : {
  "type" : "scaling",
  "min" : 1,
  "max" : 2,
  "keep_alive" : "5m"
},
"search" : {
  "type" : "fixed",
  "min" : 12,
  "max" : 12,
  "queue_size" : "1k"
},
"percolate" : {
  "type" : "fixed",
  "min" : 4,
   

Re: Help needed: CentOS ES-Logstash-Kibana running, but 0 docs, 0 indices

[Mark Walkom]

You might want to take this over to the Logstash mailing list instead as
that appears to be the problem.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 16 June 2014 17:39, 'Robert AMS' via elasticsearch <
elasticsearch@googlegroups.com> wrote:

>
> 
> This did not help. Ik paramedic it looks as if everything is running,
> except there are no docs (and indices) coming in ES. Seen screenshot.
>
> Op zondag 15 juni 2014 02:11:09 UTC+2 schreef Mark Walkom:
>>
>> Try swapping the host in your LS output to the actual IP of VM.
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 14 June 2014 18:34, 'Robert AMS' via elasticsearch <
>> elasti...@googlegroups.com> wrote:
>>
>>> Als POC I've setup a single machine with Logstash, Elasticsearch and
>>> Kibana, But I am stuck: It is running, but no docs are getting in ES and
>>> Kibana stays only a dark page. Can someone help me how to get the logs into
>>> ES en searchable in Kibana?
>>>
>>> This is my machine. Just install a CentOS 6.5 (virtual) server, su to
>>> root and copy-paste this to the command line and all will be installed:
>>>
>>> Pastebin: Install_ES_Logstash_Kibana_for_CentOS6.5
>>>  ( http://pastebin.com/fpz5AQzd )
>>>
>>> Wat is missing?
>>>
>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/ee7d3476-5551-40d2-aef5-28276e7783bd%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/46611243-97d3-4601-9045-1e27af9804be%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624bo%3D1CK2nsaidLS0mxEi4Pcmh5R3Q8jFbhaDaUzsw9B-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch not starting after switching network.

[Mark Walkom]

It could be a change of network interface which has caused problems with
the firewall?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 16 June 2014 19:06, Ashok Chidige  wrote:

> yes, but I've not changed anything in the firewall. not sure if the my
> company pushed anything silently. i'll check. Regardless I cannot turn off
> the firewall. What firewall rules do I've to enable to get it working?
>
> On Sunday, June 15, 2014 6:59:14 PM UTC-4, Mark Walkom wrote:
>>
>> Are you on Windows, is your firewall enabled?
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 16 June 2014 01:39, Ashok Chidige  wrote:
>>
>>> Hi,
>>>
>>> I've been using ES from my office and home. I really liked it but
>>> suddenly today it stopped working when I tried to start at my home. See
>>> below the log file details. Restarting my machine/ES many times but did not
>>> help. AFAIK nothing has changed from network and my machine. Please help! I
>>> won't be going to ofice for a few days and getting this working on my local
>>> with out re-installing ES is very imp for me.
>>>
>>>
>>> [2014-06-15 08:12:18,530][WARN ][bootstrap] jvm uses the
>>> client vm, make sure to run `java` with the server vm for best performance
>>> by adding `-server` to the command line
>>> [2014-06-15 08:12:18,747][INFO ][node ] [Kymaera]
>>> version[1.1.1], pid[7608], build[f1585f0/2014-04-16T14:27:12Z]
>>> [2014-06-15 08:12:18,747][INFO ][node ] [Kymaera]
>>> initializing ...
>>> [2014-06-15 08:12:19,012][INFO ][plugins  ] [Kymaera]
>>> loaded [marvel, support-1.1.0.7-1720683, jdbc-1.1.0.2-b016a06], sites
>>> [marvel]
>>> [2014-06-15 08:12:22,868][INFO ][node ] [Kymaera]
>>> initialized
>>> [2014-06-15 08:12:22,869][INFO ][node ] [Kymaera]
>>> starting ...
>>> [2014-06-15 08:12:23,159][INFO ][transport] [Kymaera]
>>> bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
>>> 192.168.1.73:9300]}
>>> [2014-06-15 08:12:26,344][INFO ][cluster.service  ] [Kymaera]
>>> new_master [Kymaera][eC55kfbfSpqPjRCxhMcThA][SVCCN-ACHIDIGE2][inet[/
>>> 192.168.1.73:9300]], reason: zen-disco-join (elected_as_master)
>>> [2014-06-15 08:12:26,479][WARN ][transport.netty  ] [Kymaera]
>>> exception caught on transport layer [[id: 0x4b95b541]], closing connection
>>> java.net.SocketException: Permission denied: no further information
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>> at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
>>>  at org.elasticsearch.common.netty.channel.socket.nio.
>>> NioClientBoss.connect(NioClientBoss.java:150)
>>> at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.
>>> processSelectedKeys(NioClientBoss.java:105)
>>>  at org.elasticsearch.common.netty.channel.socket.nio.
>>> NioClientBoss.process(NioClientBoss.java:79)
>>> at org.elasticsearch.common.netty.channel.socket.nio.
>>> AbstractNioSelector.run(AbstractNioSelector.java:318)
>>>  at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(
>>> NioClientBoss.java:42)
>>> at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
>>> ThreadRenamingRunnable.java:108)
>>>  at org.elasticsearch.common.netty.util.internal.
>>> DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
>>> Source)
>>>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>> at java.lang.Thread.run(Unknown Source)
>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/b8fbfe85-58a9-429a-95bf-3fc37cc8fd00%
>>> 40googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/de4e1721-a9c3-4cec-83ac-08bafcda2814%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsea

Re: How to get rid of org.elasticsearch.plugins information logging

[David Pilato]

You can probably put org.elasticsearch.plugins module to ERROR level instead of 
INFO here.

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 16 juin 2014 à 10:57:17, Georgi Ivanov (georgi.r.iva...@gmail.com) a écrit:

Hello,
How can i get rid of 

Jun 16, 2014 10:38:13 AM org.elasticsearch.plugins
Information: [Thinker] loaded [], sites []


every time my client connects to ES ?

It is not a big problem, but this output is messing up with my shell scripts.

I am using transport client if this matters.
Is this some log4j configuration ? I am not using log4j atm.

Regards,
Georgi
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/caea4a0b-bff7-4dfe-af92-654ab1a802ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.539eb563.3352255a.f9a2%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch not starting after switching network.

[Ashok Chidige]

yes, but I've not changed anything in the firewall. not sure if the my 
company pushed anything silently. i'll check. Regardless I cannot turn off 
the firewall. What firewall rules do I've to enable to get it working?

On Sunday, June 15, 2014 6:59:14 PM UTC-4, Mark Walkom wrote:
>
> Are you on Windows, is your firewall enabled?
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
>
> On 16 June 2014 01:39, Ashok Chidige > 
> wrote:
>
>> Hi, 
>>
>> I've been using ES from my office and home. I really liked it but 
>> suddenly today it stopped working when I tried to start at my home. See 
>> below the log file details. Restarting my machine/ES many times but did not 
>> help. AFAIK nothing has changed from network and my machine. Please help! I 
>> won't be going to ofice for a few days and getting this working on my local 
>> with out re-installing ES is very imp for me.
>>
>>
>> [2014-06-15 08:12:18,530][WARN ][bootstrap] jvm uses the 
>> client vm, make sure to run `java` with the server vm for best performance 
>> by adding `-server` to the command line
>> [2014-06-15 08:12:18,747][INFO ][node ] [Kymaera] 
>> version[1.1.1], pid[7608], build[f1585f0/2014-04-16T14:27:12Z]
>> [2014-06-15 08:12:18,747][INFO ][node ] [Kymaera] 
>> initializing ...
>> [2014-06-15 08:12:19,012][INFO ][plugins  ] [Kymaera] 
>> loaded [marvel, support-1.1.0.7-1720683, jdbc-1.1.0.2-b016a06], sites 
>> [marvel]
>> [2014-06-15 08:12:22,868][INFO ][node ] [Kymaera] 
>> initialized
>> [2014-06-15 08:12:22,869][INFO ][node ] [Kymaera] 
>> starting ...
>> [2014-06-15 08:12:23,159][INFO ][transport] [Kymaera] 
>> bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
>> 192.168.1.73:9300]}
>> [2014-06-15 08:12:26,344][INFO ][cluster.service  ] [Kymaera] 
>> new_master 
>> [Kymaera][eC55kfbfSpqPjRCxhMcThA][SVCCN-ACHIDIGE2][inet[/192.168.1.73:9300]],
>>  
>> reason: zen-disco-join (elected_as_master)
>> [2014-06-15 08:12:26,479][WARN ][transport.netty  ] [Kymaera] 
>> exception caught on transport layer [[id: 0x4b95b541]], closing connection
>> java.net.SocketException: Permission denied: no further information
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
>>  at 
>> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
>> at 
>> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
>>  at 
>> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
>> at 
>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
>>  at 
>> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
>> at 
>> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>  at 
>> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>> at java.lang.Thread.run(Unknown Source)
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/b8fbfe85-58a9-429a-95bf-3fc37cc8fd00%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/de4e1721-a9c3-4cec-83ac-08bafcda2814%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How to get rid of org.elasticsearch.plugins information logging

[Georgi Ivanov]

Hello,
How can i get rid of 

Jun 16, 2014 10:38:13 AM org.elasticsearch.plugins
Information: [Thinker] loaded [], sites []


every time my client connects to ES ?

It is not a big problem, but this output is messing up with my shell 
scripts.

I am using transport client if this matters.
Is this some log4j configuration ? I am not using log4j atm.

Regards,
Georgi

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/caea4a0b-bff7-4dfe-af92-654ab1a802ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: java client, do I put the elasticsearch.yml in my /resources?

[David Pilato]

You should define more than one node in the transport list but you can also set 
client.transport.sniff to true to sniff other nodes.

If you define in your cluster a load balancing only node (no data), you 
probably should define it (or them) in the transport list of nodes.

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 16 juin 2014 à 10:48:46, Ciprian (turcancipria...@gmail.com) a écrit:


For "2/" : let's suppose I have a cluster with next arhitecture : non-data 
node, primary node and data nodes and I want to insert documents with java api. 
Which is the node which should be set for transportClient in 
addTransportAddress? The primary, the data or the balancing(non-data) node? 
What's happen if the set node is failing? Shouln't we add another one for 
managing the failling?


luni, 16 iunie 2014, 08:08:11 UTC+3, David Pilato a scris:
1/ yes. If you have an elasticsearch.yml file in your classpath under /, it 
will be used.
Otherwise, defaults are used.

2/ A NodeClient knows where all shards live. It means that it will route 
directly your requests to the right nodes. 


HTH

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 16 juin 2014 à 04:29, gitted  a écrit :

1. Newbie question, but when I use an elastic search client (java client), do I 
put the elasticsearch.yml file in my /resources i.e classpath?

I ran my java app w/o the .yml file and it worked fine when I indexed to ES, I 
guess by default it connects to the default ES port?


2.  What I don't understand, when creating a client, why is there even an 
option to have this client be part of the cluster?  I mean you are creating a 
client using the nodeBuilder, what is the reason for this?  My guess is the 
code is re-used so you don't have to repeat things internally?

Thanks.
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearc...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c87213d6-02c5-4c76-b115-14a2ebeceeaf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4b7e8af8-565c-4c8f-8509-9edd5d39db86%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.539eb0cb.1f16e9e8.f9a2%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: java client, do I put the elasticsearch.yml in my /resources?

[Ciprian]

For "2/" : let's suppose I have a cluster with next arhitecture : non-data 
node, primary node and data nodes and I want to insert documents with java 
api. Which is the node which should be set for transportClient in 
addTransportAddress? The primary, the data or the balancing(non-data) node? 
What's happen if the set node is failing? Shouln't we add another one for 
managing the failling?


luni, 16 iunie 2014, 08:08:11 UTC+3, David Pilato a scris:
>
> 1/ yes. If you have an elasticsearch.yml file in your classpath under /, 
> it will be used.
> Otherwise, defaults are used.
>
> 2/ A NodeClient knows where all shards live. It means that it will route 
> directly your requests to the right nodes. 
>
>
> HTH
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
>
> Le 16 juin 2014 à 04:29, gitted > a 
> écrit :
>
> 1. Newbie question, but when I use an elastic search client (java client), 
> do I put the elasticsearch.yml file in my /resources i.e classpath?
>
> I ran my java app w/o the .yml file and it worked fine when I indexed to 
> ES, I guess by default it connects to the default ES port?
>
>
> 2.  What I don't understand, when creating a client, why is there even an 
> option to have this client be part of the cluster?  I mean you are creating 
> a client using the nodeBuilder, what is the reason for this?  My guess is 
> the code is re-used so you don't have to repeat things internally?
>
> Thanks.
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/c87213d6-02c5-4c76-b115-14a2ebeceeaf%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4b7e8af8-565c-4c8f-8509-9edd5d39db86%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Help needed: CentOS ES-Logstash-Kibana running, but 0 docs, 0 indices

['Robert AMS' via elasticsearch]

This did not help. Ik paramedic it looks as if everything is running, 
except there are no docs (and indices) coming in ES. Seen screenshot.

Op zondag 15 juni 2014 02:11:09 UTC+2 schreef Mark Walkom:
>
> Try swapping the host in your LS output to the actual IP of VM.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>  
>
> On 14 June 2014 18:34, 'Robert AMS' via elasticsearch <
> elasti...@googlegroups.com > wrote:
>
>> Als POC I've setup a single machine with Logstash, Elasticsearch and 
>> Kibana, But I am stuck: It is running, but no docs are getting in ES and 
>> Kibana stays only a dark page. Can someone help me how to get the logs into 
>> ES en searchable in Kibana?
>>
>> This is my machine. Just install a CentOS 6.5 (virtual) server, su to 
>> root and copy-paste this to the command line and all will be installed:
>>
>> Pastebin: Install_ES_Logstash_Kibana_for_CentOS6.5 
>>  ( http://pastebin.com/fpz5AQzd )
>>
>> Wat is missing?
>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/ee7d3476-5551-40d2-aef5-28276e7783bd%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/46611243-97d3-4601-9045-1e27af9804be%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.