Re: elasticsearch client is giving OutOfMemoryError once connection is lost to elaticsearch server

[Ed Kim]

Other members can correct me if I'm wrong, but I notice that when you lose 
connection with the server, the transportclient queues retries of whatever 
operations you try to execute, and it starts to queue listeners into a 
'generic' threadpool (which I read somewhere that it was unbounded). We've 
seen this problem when we thrash ES until it eventually stops responding, 
and our bulk requests start to back up and eventually cause the application 
to halt due to OOM.

I don't know exactly what your application is doing when your ES node(s) go 
down, but perhaps you can proactively stop requests to ES servers once your 
application sees the no node exception error (which you should get when ES 
goes down). You could also close the transportclient and shutdown its 
threadpool and reconnect/instantiate after a timed delay to clean up 
whatever is floating around in the transportclient. We have been able to 
solve most of our native thread issues by protecting our use of 
transportclient and doing a soft restart of this client. 


On Saturday, January 10, 2015 at 9:29:56 AM UTC-8, Subhadip Bagui wrote:
>
> Hi,
>
> I'm using elasticsearch using TransportClient for multiple operation. The 
> issue I'm facing now is if my es server goes down my client side app 
> getting OutOfMemoryError.  Getting the below exception. I had to restart my 
> tomcat every time after this to make my application up. Can some one please 
> suggest how to prevent this. 
>
>
> Jan 9, 2015 5:38:44 PM org.apache.catalina.core.StandardWrapperValve invoke
> SEVERE: Servlet.service() for servlet [spring] in context with path 
> [/aricloud] threw exception [Handler processing failed; nested exception is 
> java.lang.OutOfMemoryError: unable to create new native thread] with root 
> cause
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:640)
> at 
> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
> at 
> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
> at 
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
> at 
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
> at 
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:349)
> at 
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:100)
> at 
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:52)
> at 
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.(NioWorker.java:45)
> at 
> org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
> at 
> org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
>
>
> Thanks,
> Subhadip
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0ab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

real time match analysis

[Ed Kim]

Hello all, I was wondering if anyone could offer some feedback on whether 
there is a way to determine how a document matched in real time. I 
currently use custom analyzers at index time to allow a broad array of 
matches for a given text field. I try to match based on phrases, synonyms, 
substrings, stemming, etc of a given phrase, and I would like to be able to 
figure out at search time, which analyzer was attributed to causing the 
match. 

Currently, I've gotten around this by creating child documents where the 
fields are fanned out to their respective analyzer types. So I have a child 
document where the field only applies stemming, another that uses only 
synonyms, etc. However, due to the growing number of fields that require 
analysis and the growth of my data set, I'd much prefer if I had less 
documents (and less complex too). I was hoping there would be a way to tag 
tokens at the analysis phase that could be used at the search phase to 
quickly determine my match level, but I was not able to find anything like 
this.

Having said that, has anyone else ever tried to figure this out, or have an 
thoughts on how to leverage ES at a lower level to determine match? 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4222f994-d448-4b61-a71e-3dca03a5a0fc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Concurrency problem when automatically creating an index

[joergpra...@gmail.com]

I think you can safely ignore "failed to process cluster event (acquire
index lock) within 1s" in that case. These messages come from index
creation requests that are submitted concurrently - only one request can
succeed, the other will get stuck.

Jörg

On Fri, Jan 9, 2015 at 8:16 PM, Tom  wrote:

> Sorry, didn't mean to say "the same indexing operation" but multiple
> indexing operations (distinct data) on the same non-existent index.
>
> El viernes, 9 de enero de 2015, 16:13:52 (UTC-3), Tom escribió:
>
>> Well yes. We also have a cluster for the app where each node talks to the
>> elastic cluster independently.
>>
>> Remember that we are not creating the index manually. Each app node
>> issues an index operation on an index that may yet not exist and we expect
>> ES to take care of the index creation on demand. Many processes may issue
>> the same indexing operation on the ES cluster "simultaneously" and only one
>> of them must succeed in triggering the index creation.
>>
>> Tom;
>>
>> El viernes, 9 de enero de 2015, 15:53:01 (UTC-3), Jörg Prante escribió:
>>>
>>> It seems there are more than one process trying to create the index, it
>>> that possible?
>>>
>>> Jörg
>>>
>>> On Fri, Jan 9, 2015 at 7:16 PM, Tomas Andres Rossi 
>>> wrote:
>>>
 We enlarged our cluster to 5 nodes and now the QUORUM error message
 seems to have disappeared.
 "failed to process cluster event (acquire index lock) within 1s" kind
 of messages are still happening though.

 :(

 Tom;

 On Fri, Jan 9, 2015 at 3:11 PM, Tomas Andres Rossi >>> > wrote:

> We enlarged our cluster to 5 nodes and now the QUORUM error message
> seems to have disappeared.
> "failed to process cluster event (acquire index lock) within 1s" kind
> of messages are still happening though.
>
> :(
>
> Tom;
>
> On Fri, Jan 9, 2015 at 1:25 PM, joerg...@gmail.com  > wrote:
>
>> Exactly, with 3 nodes, the error will be gone.
>>
>> Please, always use an odd number of data nodes, in particular with
>> replica > 0, in order not to confuse ES quorum formula, and also to avoid
>> split brains with minimun_master_nodes
>>
>> Jörg
>>
>> On Fri, Jan 9, 2015 at 3:17 PM, Tom  wrote:
>>
>>> Also, we have another cluster (for different purposes) that has 3
>>> nodes but we didn't experience such errors with it (for this ES we 
>>> create
>>> indices on a daily basis).
>>>
>>> El jueves, 8 de enero de 2015, 16:23:12 (UTC-3), Tom escribió:
>>>
 4

 El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió:
>
> How many nodes do you have in the cluster?
>
> Jörg
>
> On Thu, Jan 8, 2015 at 6:57 PM, Tom  wrote:
>
>> Hi, we'd been using ES for a while now. Specifically version
>> 0.90.3. A couple of months ago we decided to migrate to the latest 
>> version
>> which was finally frozen to be 1.4.1. No data migration was necessary
>> because we have a redundant MongoDB, but yesterday we enabled data 
>> writing
>> to the new ES cluster. All was running smoothly when we noticed that 
>> at
>> o'clock times there were bursts of four or five log messages of the
>> following kinds:
>>
>> Error indexing None into index ind-analytics-2015.01.08. Total
>> elapsed time: 1065 ms. org.elasticsearch.cluster.metadata.
>> ProcessClusterEventTimeoutException: failed to process cluster
>> event (acquire index lock) within 1s
>> at org.elasticsearch.cluster.metadata.MetaDataCreateIndexServic
>> e$1.run(MetaDataCreateIndexService.java:148) ~[org.elasticsearch.
>> elasticsearch-1.4.1.jar:na]
>> at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> ~[na:1.7.0_17]
>> at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> ~[na:1.7.0_17]
>> at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0_17]
>>
>> [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy -
>> Scattering to failed in 1043ms 
>> org.elasticsearch.action.UnavailableShardsException:
>> [ind-2015.01.08.00][0] Not enough active copies to meet write 
>> consistency
>> of [QUORUM] (have 1, needed 2). Timeout: [1s], request: index
>> {[ind-2015.01.08.00][search][...]}
>> at org.elasticsearch.action.support.replication.TransportShardR
>> eplicationOperationAction$AsyncShardOperationAction.retryBec
>> auseUnavailable(TransportShardReplicationOperationAction.java:784)
>> ~[org.elasticsearch.elasticsearch-1.4.1.jar:na]
>> at org.elasticsearch.action.support.replication.TransportShardR
>> eplicationOperationAction$AsyncShardOp

Re: Writing custom scripts for indexing data in Elasticsearch

[joergpra...@gmail.com]

If you can set up shell scripting, it should be viable to define a curl
command in a script that copies the table and columns into an SQL statement
and performs something similar to step 7 in
https://github.com/jprante/elasticsearch-river-jdbc/wiki/Quickstart

Jörg

On Fri, Jan 9, 2015 at 2:45 PM, Amtul Nazneen 
wrote:

> Hi,
>
> I'm a newbie to Elasticsearch, and I would like to know about writing
> custom scripts in Elasticsearch. Basically, I want to have a script which
> takes a table name and the columns in it as parameters, and would then
> start a JDBC river plugin and index the corresponding data into my
> Elasticsearch. Basically, I'm kind of looking for a mechanism through which
> I can automatically index data into my Elasticsearch on just specifying my
> required table and columns. And I would very much like to know if this
> viable and also any other ideas through which I could implement this?
>
> Thank you.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/f2c89e51-f690-433e-b320-f18659958eee%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEr02XoB%3Dh1bE7qko7HSavJKZm7Zp5EYNg_aifwQMi%2B4g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

elasticsearch client is giving OutOfMemoryError once connection is lost to elaticsearch server

[Subhadip Bagui]

Hi,

I'm using elasticsearch using TransportClient for multiple operation. The 
issue I'm facing now is if my es server goes down my client side app 
getting OutOfMemoryError.  Getting the below exception. I had to restart my 
tomcat every time after this to make my application up. Can some one please 
suggest how to prevent this. 


Jan 9, 2015 5:38:44 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [spring] in context with path 
[/aricloud] threw exception [Handler processing failed; nested exception is 
java.lang.OutOfMemoryError: unable to create new native thread] with root 
cause
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at 
java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
at 
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
at 
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
at 
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:349)
at 
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:100)
at 
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:52)
at 
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.(NioWorker.java:45)
at 
org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
at 
org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)


Thanks,
Subhadip

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6930af3b-13ad-476c-82cc-7c8d792dea67%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: To Raid or not to Raid

[Bhaskar Karambelkar]

Hi Jörg,
I'm the author of those slides, and that statement, even when taken out of 
context starts with  "Prefer", 
I don't think I need to explain what prefer means, but just in case ...
Using JBOD will be your safest bet as opposed to using something like RAID 
/ SAN/ NAS unless you really know what you're doing.
I never said DON'T EVER use RAID or even SAN|NAS, just "prefer" JBOD.

I do agree with your assessment of RAID 0 below, but do remember that, that 
one statement was taken out of context, for full context I suggest you go 
through the whole slide deck and better yet the whole talk whose video was 
posted on elasticsearch site. I even made a point about some of my 
recommendations not being applicable to cloud deployments etc.

As to your point about simplification of NAS|SAN, that's the whole point of 
presenting to a wide audience, one simplifies things such that they can be 
applied to majority of the cases, and not concentrate on esoteric 
deployments :). As to local gateway, that's the only one ES recommends now, 
the  shared FS, HDFS, S3 gateways were long deprecated.

FWIW I fully agree with your statement on taking control over complete 
hardware setup, heck there's a full slide in there dedicated to this point, 
titled 'Know your platform'.

At the end of the day, there's no single silver bullet, everyone will have 
to evaluate what works best for their situation, what worked for us may not 
work well for others. It would be indeed very naive to take my slides as 
laws, they are more or less pointers worth exploring. Some may work for you 
some won't. They worked fairly well for us.

I might sound a bit defensive here, but hey we did build that cluster and 
we're nearing a Trillion documents in it, so I guess we must be doing 
something right :). 


Bhaskar


On Saturday, December 13, 2014 at 10:48:55 AM UTC-5, Jörg Prante wrote:
>
> The statement is related to performance and I can't agree with it. You can 
> easily build a RAID 0 system which has massive I/O throughput performance 
> and is superior to JBOD, because RAID striping does not slow things down, 
> it is as always as much as fast than a single drive and in most RAID levels 
> it is much faster. 
>
> In the past, RAID was invented for mirroring cheap and error-prone spindle 
> disk arrays, while mirrors increase costs but decrease fault probability.
>
> With Elasticsearch, the decision is if you still want to handle disk 
> faults by drive redundancy (RAID) and all other hardware faults like power 
> outages by server downtime. This is just a matter of organization and of 
> cost. I would suggest from my experience: take control over your complete 
> hardware setup, equip your systems with expensive SAS2 (or even better) 
> controllers with RAID 0 to reduce cost and maximize performance, and handle 
> all kind of hardware faults by server downtime, because ES replica level > 
> 0 allows that.
>
> There is also a simplification of SAN/NAS in the statement but that is a 
> different discussion. Never use SAN/NAS for ES local gateway.
>
> Jörg
>
> On Fri, Dec 12, 2014 at 7:28 PM, Elvar Böðvarsson  > wrote:
>>
>>
>> Second, "Prefer JBODs for data disks over RAID, SAN/NAS", would be ok, 
>> maybe then to be safe go with 2x replicas, goes well with having 3x nodes
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d0fa5e9c-2658-4fef-a9ad-ea83873a8f28%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES 1.4.2 random node disconnect

[Revan007]

The thing is I don't think is the monitor plugin.  When this happens, my 
node gets disconnected and the cluster goes into yellow state till it 
recovers .   I am using curator optimize , it is set to 2 segments for 
indices older than 2 days .

On Saturday, January 10, 2015 at 2:56:52 PM UTC+2, Revan007 wrote:
>
> Hey, thank you for answering, I am using Marvel latest version.
>
> Here is more info about the problem :
>
> https://github.com/elasticsearch/elasticsearch/issues/9212#issuecomment-69292232
>
> On Saturday, January 10, 2015 at 2:50:02 PM UTC+2, Jörg Prante wrote:
>>
>> If you see
>>
>> cluster:monitor/nodes/stats[n]] request_id [82300775] timed out after 
>> [15000ms]
>>
>> in the logs, you have a monitor tool running that can not complete 
>> requests because it takes longer than 15 seconds to traverse all the data 
>> folders on all the nodes.
>>
>> There are a number of methods to reduce disk traversal time in the data 
>> folders:
>>
>> - switch off monitoring (not really helpful) or reduce monitor interval 
>> (maybe helpful, maybe not)
>>
>> - increase stats request timeout (if monitor tools allow this but this 
>> does not solve the cause of the problem)
>>
>> - monitor only an index subset of your cluster (monitor tools usually do 
>> not have this option)
>>
>> - reduce number of segments per node -> either by optimizing indices or 
>> adding nodes
>>
>> - wait for a fix in a future ES release 
>>
>> Have you counted the total number of segments? If the number is high, did 
>> you run _optimize with max_num_segments on your indices to reduce the 
>> number of segments?
>>
>> Jörg
>>
>> On Fri, Jan 9, 2015 at 6:55 AM, Revan007  wrote:
>>
>>> Hey,
>>>
>>> I am having trouble for some while. I am getting random node disconnects 
>>> and I cannot explain why.
>>> There is no increase in traffic ( search or index ) when this is 
>>> happening , it feels so random to me .
>>> I first thought it could be the aws cloud plugin so I removed it and 
>>> used unicast and pointed directly to my nodes IPs but that didn't seem to 
>>> be the problem .
>>> I changed the type of instances, now m3.2xlarge, added more instances, 
>>> made so much modifications in ES yml config and still nothing .
>>> Changed java oracle from 1.7 to 1.8 , changed CMS collector to G1GC and 
>>> still nothing .
>>>
>>> I am out of ideas ... how can I get more info on what is going on ?
>>>
>>> Here are the logs I can see from master node and the data node 
>>> http://pastebin.com/GhKfRkaa
>>>
>>>
>>> Current config:
>>>
>>>
>>> 6 m3.x2large, 1 master, 5 data nodes.
>>> 414 indices, index/day
>>> 7372 shards. 9 shards, 1 replica per index
>>> 208 million documents, 430 GB
>>> 15 gb heap size allocated per node
>>> ES 1.4.2
>>>
>>> Current yml config here : 
>>> http://pastebin.com/Nmdr7F6J
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4cc37e2e-4bbc-483d-bbbe-6cd0138d6689%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES 1.4.2 random node disconnect

[Revan007]

Hey, thank you for answering, I am using Marvel latest version.

Here is more info about the problem :
https://github.com/elasticsearch/elasticsearch/issues/9212#issuecomment-69292232

On Saturday, January 10, 2015 at 2:50:02 PM UTC+2, Jörg Prante wrote:
>
> If you see
>
> cluster:monitor/nodes/stats[n]] request_id [82300775] timed out after 
> [15000ms]
>
> in the logs, you have a monitor tool running that can not complete 
> requests because it takes longer than 15 seconds to traverse all the data 
> folders on all the nodes.
>
> There are a number of methods to reduce disk traversal time in the data 
> folders:
>
> - switch off monitoring (not really helpful) or reduce monitor interval 
> (maybe helpful, maybe not)
>
> - increase stats request timeout (if monitor tools allow this but this 
> does not solve the cause of the problem)
>
> - monitor only an index subset of your cluster (monitor tools usually do 
> not have this option)
>
> - reduce number of segments per node -> either by optimizing indices or 
> adding nodes
>
> - wait for a fix in a future ES release 
>
> Have you counted the total number of segments? If the number is high, did 
> you run _optimize with max_num_segments on your indices to reduce the 
> number of segments?
>
> Jörg
>
> On Fri, Jan 9, 2015 at 6:55 AM, Revan007 > 
> wrote:
>
>> Hey,
>>
>> I am having trouble for some while. I am getting random node disconnects 
>> and I cannot explain why.
>> There is no increase in traffic ( search or index ) when this is 
>> happening , it feels so random to me .
>> I first thought it could be the aws cloud plugin so I removed it and used 
>> unicast and pointed directly to my nodes IPs but that didn't seem to be the 
>> problem .
>> I changed the type of instances, now m3.2xlarge, added more instances, 
>> made so much modifications in ES yml config and still nothing .
>> Changed java oracle from 1.7 to 1.8 , changed CMS collector to G1GC and 
>> still nothing .
>>
>> I am out of ideas ... how can I get more info on what is going on ?
>>
>> Here are the logs I can see from master node and the data node 
>> http://pastebin.com/GhKfRkaa
>>
>>
>> Current config:
>>
>>
>> 6 m3.x2large, 1 master, 5 data nodes.
>> 414 indices, index/day
>> 7372 shards. 9 shards, 1 replica per index
>> 208 million documents, 430 GB
>> 15 gb heap size allocated per node
>> ES 1.4.2
>>
>> Current yml config here : 
>> http://pastebin.com/Nmdr7F6J
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/fca12a91-07f6-4152-a4e4-97098e68fd0e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES 1.4.2 random node disconnect

[joergpra...@gmail.com]

If you see

cluster:monitor/nodes/stats[n]] request_id [82300775] timed out after
[15000ms]

in the logs, you have a monitor tool running that can not complete requests
because it takes longer than 15 seconds to traverse all the data folders on
all the nodes.

There are a number of methods to reduce disk traversal time in the data
folders:

- switch off monitoring (not really helpful) or reduce monitor interval
(maybe helpful, maybe not)

- increase stats request timeout (if monitor tools allow this but this does
not solve the cause of the problem)

- monitor only an index subset of your cluster (monitor tools usually do
not have this option)

- reduce number of segments per node -> either by optimizing indices or
adding nodes

- wait for a fix in a future ES release

Have you counted the total number of segments? If the number is high, did
you run _optimize with max_num_segments on your indices to reduce the
number of segments?

Jörg

On Fri, Jan 9, 2015 at 6:55 AM, Revan007  wrote:

> Hey,
>
> I am having trouble for some while. I am getting random node disconnects
> and I cannot explain why.
> There is no increase in traffic ( search or index ) when this is happening
> , it feels so random to me .
> I first thought it could be the aws cloud plugin so I removed it and used
> unicast and pointed directly to my nodes IPs but that didn't seem to be the
> problem .
> I changed the type of instances, now m3.2xlarge, added more instances,
> made so much modifications in ES yml config and still nothing .
> Changed java oracle from 1.7 to 1.8 , changed CMS collector to G1GC and
> still nothing .
>
> I am out of ideas ... how can I get more info on what is going on ?
>
> Here are the logs I can see from master node and the data node
> http://pastebin.com/GhKfRkaa
>
>
> Current config:
>
>
> 6 m3.x2large, 1 master, 5 data nodes.
> 414 indices, index/day
> 7372 shards. 9 shards, 1 replica per index
> 208 million documents, 430 GB
> 15 gb heap size allocated per node
> ES 1.4.2
>
> Current yml config here :
> http://pastebin.com/Nmdr7F6J
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHv0wxNXq_nJrj5ByxrpZmwbdiKmMUbu4YYfjuGM5XkAA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Kibana dashboards not saving for me in ElasticSearch

[William Tarrant]

Ok, so after a couple of more tries I finally got it to work.  It seems I 
had changed the cluster name to a custom in the elasticsearch.yml config 
i.e "d-elasticsearch" but when I remove that and left it use the standard 
it worked fine.

On Friday, January 9, 2015 at 4:03:46 PM UTC, William Tarrant wrote:
>
> Hey, after a long day with no success I finally conceded that I need to 
> ask for help.  I am running ES 1.4 with kibana 3.1.2 hosted on ngnix both 
> on the same machine (I have done the cors work around).  Whenever I try to 
> save a new dashboard or retrieve it, it results in a blank page, it appears 
> in ES as an entry, there is an index for kibana-int but the _source is 
> blank for the dashboard.  Kibana is connecting to ES and I can run queries 
> through the default dashboard so I know at least it is connecting.
>
> the following is from elastic-head
> {
> "_index": "kibana-int",
> "_type": "dashboard",
> "_id": "Demo1",
> "_version": 1,
> "_score": 1,
> "_source": { }
> }
>
> The only exception I see in the javascript console is:
>
> TypeError: Cannot read property 'interval' of undefined
> at dash_load (http:///app/app.js:21:25945)
> at f (http:///app/app.js:21:28755)
> at http:///app/app.js:14:4667
> at i (http://app/app.js:9:458)
> at http:///app/app.js:9:1014
> at Object.f.$eval (http:///app/app.js:9:6963)
> at Object.f.$digest (http:///app/app.js:9:5755)
> at Object.f.$apply (http:///app/app.js:9:7111)
> at f (http:///app/app.js:9:11507)
> at r (http:///app/app.js:9:13216)
>
> I would appreciate any help as this is a real blocker for me at the moment.
>
> Thanks
>
> Will
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ef1c2c70-c16f-474c-8104-89a217d13d6b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Searching with Elasticsearch.Net

[Steve Flook]

I think you are just getting unlucky.  You are asking for the top 10 
results, starting *after *result number 10.  Only problem is the search 
response is saying there is only 10 results total, so there are no hits to 
return.

Try changing your from to 0, instead of 1 * 10.  Or you can just omit that 
all together, as ES will default to 0.

If you are coming from the world of sql, one difference in simple queries 
is ES gives back the total number of results with that total property you 
see in the response's hits section.  So even if you ask for only 10 
documents, total will give you the full count no matter what.

Steve

On Friday, January 9, 2015 at 12:43:36 PM UTC-5, Garrett Johnson wrote:
>
> Hi All,
>
> I would like to use Elasticsearch.Net (NEST requires types and I do not 
> want strong types) to do a simple _all term search.  I can do this using 
> the plugin elasticsearch head and I retrieve the appropriate documents.  
> Here is some simple code I wrote just to say hey give me all that match.
>
> var node = new Uri(http://myhost:9200);
>
> var config = new ConnectionConfiguration(node); 
>
> var exposed = config.ExposeRawResponse(true);
>
> var client = new ElasticsearchClient(config);
>
> var search = new
>
> {
>
> size = 10,
>
> from = 1 * 10,
>
> query = new { query_string = new { query = this.textBox1.Text } },
>
> };
>
> var searchResponse = client.Search("jdbc",search);
>
> This returns these results:
>
> {StatusCode: 200, 
>  Method: POST, 
>  Url: http://u4vmeqlditapp01:9200/jdbc/_search, 
>  Request: 
> {"size":10,"from":10,"query":{"query_string":{"query":"Garrett"}}}, 
>  Response: 
> {"took":5,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":10,"max_score":1.1672286,"hits":[]}}}
>
> But no documents.
>
> Here is the JSON I'm trying to replicate:
>
>
>
>- "query": {
>   - "bool": {
>  - "must": [
> - {
>- "query_string": {
>   - "default_field": "_all",
>   - "query": "Garrett"
>}
> }
>  ],
>  - "must_not": [ ],
>  - "should": [ ]
>   }
>},
>- "from": 0,
>- "size": 25000,
>- "sort": [ ],
>- "facets": { }
>
>
> I'm pretty sure it is because the query doesn't have the default_field set 
> to _all... But I don't know how to set that.  I've tried several string 
> concatenations to no avail it just searched for them.
>
> Any one with any ideas.  I want to simply search all types for a single 
> string.
>
> Garrett
>
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b4c0a2ac-acdb-43fb-a238-88520a4c1be5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Search not working unless type specified

[David Pilato]

That’s a very old version.
I just tested this and was able to get results. Tested on elasticsearch 1.4.2

DELETE twitter
PUT twitter/tweet/1
{
  "type": "tweet",
  "tweet_id": 128
}

GET /twitter/tweet/_search
{
  "query" : {"match_all": {}},
  "filter": {
"terms": {
  "tweet_id":[
128
  ]
}
  },
  "sort" : {"type" : "desc"}
}
# gives results

GET /twitter/_search
{
  "query" : {"match_all": {}},
  "filter": {
"terms": {
  "tweet_id":[
128
  ]
}
  },
  "sort" : {"type" : "desc"}
}
# gives results


-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet  | @elasticsearchfr 
 | @scrutmydocs 




> Le 9 janv. 2015 à 23:48, Stefanie  a écrit :
> 
> I am having an issue with searching results if the type is not specified.
> 
> The following search request works correctly:
> 
> curl -XGET 'http://localhost:9200/twitter/tweet/_search' -d '{
>   "query" : {"match_all": {}},
>   "filter": {
> "terms": {
>   "tweet_id":[
> 128
>   ]
> }
>   },
>   "sort" : {"type" : "desc"}
> }
> '
> 
> But this search request does not return any results:
> 
> curl -XGET 'http://localhost:9200/twitter/_search' -d '{
>   "query" : {"match_all": {}},
>   "filter": {
> "terms": {
>   "tweet_id":[
> 128
>   ]
> }
>   },
>   "sort" : {"type" : "desc"}
> }
> '
> Does anyone know why this is?
> 
> I am using elasticsearch version 0.90.13.
> 
> Thank you for all the help.
> Stefanie
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/31092564-4f29-4e45-a9e3-a8118838c5b5%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6B050A32-21EE-4FF8-A299-A0BFE36A870B%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.