Filtered dsl return different results for num of "query"

[Hajime]

Can somebody explain that why this three filtered dsl

queries
could potentially bring different result set?

1: one query: filtered -> query
{
   "query":{
  "filtered":{
 "query":{
"bool":{
   "must":[
  {
 "match":{
"body":"hello"
 }
  },
  {
 "match":{
"body":"world"
 }
  },
  {
 "match":{
"body":"test"
 }
  }
   ]
}
 }
  },
  "filter":{

  }
   }
}

2: two queries: filtered->query,filtered->filter->query
{
   "query":{
  "filtered":{
 "query":{
"bool":{
   "must":[
  {
 "match":{
"body":"hello"
 }
  },
  {
 "match":{
"body":"world"
 }
  }
   ]
}
 }
  },
  "filter":{
 "query":{
"match":{
   "body":"test"
}
 }
  }
   }
}

3:one query: filtered->filter->query
{
   "query":{
  "filtered":{
  "filter":{
 "query":{
"bool":{
   "must":[
  {
 "match":{
"body":"hello"
 }
  },
  {
 "match":{
"body":"world"
 }
  },
  {
 "match":{
"body":"test"
 }
  },
   ]
}
 }
  },
  }
   }
}

thanks,

Hajime

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zsozec30sOVHDy944acCXxZLJvkbeCyXiReYbXxFR5izHA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
//Use one query: filtered query
{  
   "query":{  
  "filtered":{  
 "query":{  
"bool":{  
   "must":[  
  {  
 "match":{  
"body":"hello"
 }
  },
  {  
 "match":{  
"body":"world"
 }
  },
  {  
 "match":{  
"body":"test"
 }
  }
   ]
}
 }
  },
  "filter":{  

  }
   }
}

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zsozec30sOVHDy944acCXxZLJvkbeCyXiReYbXxFR5izHA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
//Use two query: filtered query and filtered filter query

{  
   "query":{  
  "filtered":{  
 "query":{  
"bool":{  
   "must":[  
  {  
 "match":{  
"body":"hello"
 }
  },
  {  
 "match":{  
"body":"world"
 }
  }
   ]
}
 }
  },
  "filter":{  
 "query":{  
"match":{  
   "body":"test"
}
 }
  }
   }
}

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHm3Zsozec30sOVHDy944acCXxZLJvkbeCyXiReYbXxFR5izHA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
{  
   "query":{  
  "filtered":{  
  "filter":{  
 "query":{  
"bool":{  
   "must":[  
  {  
 "match":{  
"body":"hello"
 }
  },
  {  
 "match

Re: how to create multiple cluster in elasticsearch?

[pradeep raju]

Tnks a lot *Jürgen Wagner.*


On Wednesday, 24 December 2014 11:47:12 UTC+5:30, pradeep raju wrote:
>
> how to create multiple cluster in elasticsearch?
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a88daef8-32e1-42a8-b964-849665b29fc8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: how to create multiple cluster in elasticsearch?

[Jürgen Wagner (DVT)]

Simply have clusters with different cluster names.
Soon enough?

On 24.12.2014 07:17, pradeep raju wrote:
> how to create multiple cluster in elasticsearch?
> -- 
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to elasticsearch+unsubscr...@googlegroups.com
> .
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/5d0b3797-d538-4a13-8443-950a5b551b2e%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.


-- 

Mit freundlichen Grüßen/Kind regards/Cordialement vôtre/Atentamente/С
уважением
*i.A. Jürgen Wagner*
Head of Competence Center "Intelligence"
& Senior Cloud Consultant

Devoteam GmbH, Industriestr. 3, 70565 Stuttgart, Germany
Phone: +49 6151 868-8725, Fax: +49 711 13353-53, Mobile: +49 171 864 1543
E-Mail: juergen.wag...@devoteam.com
, URL: www.devoteam.de



Managing Board: Jürgen Hatzipantelis (CEO)
Address of Record: 64331 Weiterstadt, Germany; Commercial Register:
Amtsgericht Darmstadt HRB 6450; Tax Number: DE 172 993 071



-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/549A5E25.1070603%40devoteam.com.
For more options, visit https://groups.google.com/d/optout.
<>

Re: how to create multiple cluster in elasticsearch?

[pradeep raju]

Please help me as soon as possible..??


On Wednesday, 24 December 2014 11:47:12 UTC+5:30, pradeep raju wrote:
>
> how to create multiple cluster in elasticsearch?
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b786a160-b8e1-445e-8f63-40adbfda6d20%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

how to create multiple cluster in elasticsearch?

[pradeep raju]

how to create multiple cluster in elasticsearch?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5d0b3797-d538-4a13-8443-950a5b551b2e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch 1.3.1 not_analysed problem

[Kalyan Srinivas]

Thanks David,i solved this problem by adding 
curl -XPUT localhost:9200/my_index -d '@mapping.json'
replace POST with PUT and added single-quote on the file name and 
everything is working as expected,its bizarre but i guess i wasnt doing 
this right before

Kalyan


On Tuesday, December 23, 2014 7:34:07 PM UTC-8, David Pilato wrote:
>
> You should check that your mapping has been applied.
>
> GET my_index/store/_mapping
>
>
> David
>
> Le 24 déc. 2014 à 00:35, Kalyan Srinivas  > a écrit :
>
> HI All
> I am having a very weird problem with fields that are configured as 
> not_analysed for some reason the value is being tokenised even when i have 
> the field configured as not_analyzed i am sure i am missing something very 
> simple i just cant seem to figure out what,this is how my mapping looks like
>  "settings": {
> "number_of_shards": 5,
> "number_of_replicas": 2
>   },
>   "mappings": {
> "story": {
>   "properties": {
> "section": {
>   "type": "string",
>   "index": "not_analyzed"
> }
>   }
> }
>   }
> curl -XPOST localhost:9200/my_index -d @mapping.json
>
> when i index data
>
> curl -XPOST localhost:9200/my_index/story -d '{
>
>   "section": "fast-forward"
>
> }'
> and i look at 
> http://localhost:9200/my_index/_termlist
> i see it as individual terms.i know i am missing something simple cant 
> figure out what it is,somebody please help
>
> Kalyan
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/5cf375a8-b63e-4aa9-8f62-0cb6e46ad890%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8c0f52cc-11aa-46a0-97a3-807b968ea789%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

JDBC River plugin Error

[Ashutosh Parab]

I am trying to use JDBC River plugin to connect my DB To Elastic Search. I 
was able to put the river config using curl but I am getting the following 
error on startup :-

[2014-12-24 09:52:45,170][WARN ][river] [LDN] failed to 
create river [jdbc][mybooks]
org.elasticsearch.common.settings.NoClassSettingsException: Failed to load 
class
 with value [jdbc]
at 
org.elasticsearch.river.RiverModule.loadTypeModule(RiverModule.java:8
7)
at 
org.elasticsearch.river.RiverModule.spawnModules(RiverModule.java:58)

at 
org.elasticsearch.common.inject.ModulesBuilder.add(ModulesBuilder.jav
a:44)
at 
org.elasticsearch.river.RiversService.createRiver(RiversService.java:
137)
at 
org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(Rivers
Service.java:275)
at 
org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(Rivers
Service.java:269)
at 
org.elasticsearch.action.support.TransportAction$ThreadedActionListen
er$1.run(TransportAction.java:93)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: jdbc
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at 
org.elasticsearch.river.RiverModule.loadTypeModule(RiverModule.java:7
3)
  

My config file is as follows:-
{
"type": "jdbc",
"jdbc": {
"driver": "com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url": 

"jdbc:sqlserver://LOCALHOST\\SQLEXPRESS;databaseName=Report",
"user": "user",
"password": "password",
"sql": "select * from abc"
}
}

Please advice. My Elastic Search version is 1.0.1 and JDBC plugin version 
is 1.4.0.6.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9e9a1906-3f9c-4da3-9037-c84a599444f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch 1.3.1 not_analysed problem

[David Pilato]

You should check that your mapping has been applied.

GET my_index/store/_mapping


David

> Le 24 déc. 2014 à 00:35, Kalyan Srinivas  a écrit :
> 
> HI All
> I am having a very weird problem with fields that are configured as 
> not_analysed for some reason the value is being tokenised even when i have 
> the field configured as not_analyzed i am sure i am missing something very 
> simple i just cant seem to figure out what,this is how my mapping looks like
>  "settings": {
> "number_of_shards": 5,
> "number_of_replicas": 2
>   },
>   "mappings": {
> "story": {
>   "properties": {
> "section": {
>   "type": "string",
>   "index": "not_analyzed"
> }
>   }
> }
>   }
> curl -XPOST localhost:9200/my_index -d @mapping.json
> 
> when i index data
> curl -XPOST localhost:9200/my_index/story -d '{
> 
>   "section": "fast-forward"
> 
> }'
> 
> and i look at 
> http://localhost:9200/my_index/_termlist
> i see it as individual terms.i know i am missing something simple cant figure 
> out what it is,somebody please help
> 
> Kalyan
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/5cf375a8-b63e-4aa9-8f62-0cb6e46ad890%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/560605D1-68D0-4925-8465-0B9324DE783A%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch 1.3.1 not_analysed problem

[Kalyan Srinivas]

HI All
I am having a very weird problem with fields that are configured as 
not_analysed for some reason the value is being tokenised even when i have 
the field configured as not_analyzed i am sure i am missing something very 
simple i just cant seem to figure out what,this is how my mapping looks like
 "settings": {
"number_of_shards": 5,
"number_of_replicas": 2
  },
  "mappings": {
"story": {
  "properties": {
"section": {
  "type": "string",
  "index": "not_analyzed"
}
  }
}
  }
curl -XPOST localhost:9200/my_index -d @mapping.json

when i index data

curl -XPOST localhost:9200/my_index/story -d '{

  "section": "fast-forward"

}'
and i look at 
http://localhost:9200/my_index/_termlist
i see it as individual terms.i know i am missing something simple cant 
figure out what it is,somebody please help

Kalyan

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5cf375a8-b63e-4aa9-8f62-0cb6e46ad890%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Running elasticsearch 1.4.2 and kibana 4 as service

[joergpra...@gmail.com]

The question was about having a service under RHEL.

The Tanuki service wrapper has the advantage of guarding the JVM and
restarts it if it exits with failure exit code, similar to Solaris SMF.

The RPM packaging in the ES repository has some peculiarities which I find
inadequate (mixture of SUSE Linux/RHEL scripts, no RPM spec, no build from
source, strange file locations like binaries in /usr/share, surprise sysctl
/ ulimit calls hidden in scripts and vm.max.map.count setting instead of
letting the superuser set persistent settings in /etc/sysctl.conf) I hope
the Fedora team gets soon ready with the official Red Hat Linux packaging.
Just my 2¢.

Jörg



On Tue, Dec 23, 2014 at 10:53 PM, Itamar Syn-Hershko 
wrote:

> I'd actually prefer to install from repositories as they take care of
> placing things in the right place and create a user to run ES under
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko 
> Freelance Developer & Consultant
> Author of RavenDB in Action 
>
> On Tue, Dec 23, 2014 at 11:45 PM, joergpra...@gmail.com <
> joergpra...@gmail.com> wrote:
>
>> Use https://github.com/elasticsearch/elasticsearch-servicewrapper to run
>> ES as a service under RHEL 6.
>>
>> Jörg
>>
>> On Tue, Dec 23, 2014 at 10:02 PM, Ram Maram 
>> wrote:
>>
>>> Hi,
>>>
>>> Right now I am running kibana 3 and elasticsearch 1.3.2 for our ELK
>>> stack, I would like to use kibana 4 and elasticsearch 1.4.2.
>>>
>>> Can someone please let me know how to install kibana 4 and elasticsearch
>>> 1.4.2 as a service on linux?
>>>
>>> I was able to run them manually but I couldn't figure how to run them as
>>> a service.
>>>
>>> Thanks,
>>>
>>> Ram
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/13d0fe92-bb67-4552-b8da-f482a4291dd1%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoERpmJxSASTyghjpVA7omKqx3N1Y7CdMX_GRpfJh5J6Hg%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zs6xznALSGzjQGz2OGsU%3D3xh88Ab5HOZw9bLVn%3Dcjc3YQ%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGAh1eYnn8uqEpVwxyvAbBg4fM9rh7MRkg7zqnv5SWQnQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: If I had user has number of nested obj then what is our server architecture

[Mark Walkom]

Your email is a little unclear.
What exactly is the problem?

On 23 December 2014 at 16:47, sandeep kaushal verma <
sandeepkaushalve...@gmail.com> wrote:

> Current server architecture/configuration
>
> 1 Master Node (m3.large)
> 1 Search Node (m3.large)
> 3 Data Node  (c3.xlarge)
>
> 20 shard
>
>
> user has 6 nested obj
> {
> "user" : {
>   "dynamic_templates" : [
>{
>"string_template" : {
>   "match" : "*",
> "match_mapping_type" : "string",
> "mapping" : {
> "type" : "string",
> "index" : "analyzed",
> "analyzer":"string_lowercase"
> }
>}
>
>  }
>],
>"_routing": {"required": true, "path": "userId"},
>"properties":{
>"userId":{"type":"long"},
>"prop": {"type": "nested"}
>,"campaign": {"type": "nested"},
>"category": {"type": "nested"},
>"product": {"type": "nested"},
>"brand": {"type": "nested"},
>"event": {"type": "nested"}
>
>}
>}
>
> }
> Also set heap Size configuration and  other ulimit
> What i m doing wrong That I m getting let response from response
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/a644d931-3a98-4cd1-95c1-d88d755c6dfc%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-Gx%2B0LPdsh0nE1f5spZNBjF38YVy75%3DwQyCvi-cLUAMg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Dedicated master node and client node

[Mark Walkom]

   1. That is ok, but just make sure you size the heap to account for large
   queries (ie aggs) or your master could still OOM (which is bad). You may
   find as your cluster grows it'll make sense to split the masters and
   clients.
   2. Should be ok, the master doesn't need much heap. But you don't want
   to go smaller than that.
   3. Depends. Unless those nodes are massive and you are doing a *lot* of
   heavy queries then I (personally) wouldn't bother with masters or clients,
   and just round robin with your existing nodes.

If you are combining master+data+client nodes, then make sure you monitor
them for heap use :)

On 24 December 2014 at 02:28, Han JU  wrote:

> Hi,
>
> Currently in our ElasticSearch cluster we have one dedicated master node
> (master: true, data: false) and it's also the nodes that receives all
> requests. This is bad and we want to change it.
> Here's some questions:
>
>   1. Is it a good idea to merge dedicated master nodes with client nodes?
> Say in the cluster, we have 3 machines at dedicated master nodes and we do
> some kind of round robin DNS on these 3 for incoming requests.
>   2. If we opt for separated, dedicated master, could we use smaller
> instances? Say 3 machines of m3.medium.
>   3. In general, what's the general suggestion for a cluster setup? We are
> currently talking about: 5 data nodes, 3 dedicated master node and 2 client
> nodes.
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/4a6ad3d7-b8dd-4e7d-8395-a583b3b70df4%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_8%3DX79-NeqDfoOc-OH5_8YG6eK_6O-XvZT2QbLdak1uA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Startup issues with ES 1.3.5

[Mark Walkom]

Can you elaborate on your dataset and structure; how many indexes, how many
shards, how big they are etc.

On 24 December 2014 at 07:36, Chris Moore  wrote:

> Updating again:
>
> If we reduce the number of shards per node to below ~350, the system
> operates fine. Once we go above that (number_of_indices *
> number_of_shards_per_index * number_of_replicas / number_of_nodes), we
> start running into the described issues.
>
> On Friday, December 12, 2014 2:11:08 PM UTC-5, Chris Moore wrote:
>>
>> Just a quick update, we duplicated our test environment to see if this
>> issue was fixed by upgrading to 1.4.1 instead. We received the same errors
>> under 1.4.1.
>>
>> On Friday, December 5, 2014 4:52:05 PM UTC-5, Chris Moore wrote:
>>>
>>> As a followup, I closed all the indices on the cluster. I would then
>>> open 1 index and optimize it down to 1 segment. I made it through ~60% of
>>> the indices (and probably ~45% of the data) before the same errors showed
>>> up in the master log and the same behavior resumed.
>>>
>>> On Friday, December 5, 2014 3:57:12 PM UTC-5, Chris Moore wrote:

 I replied once, but it seems to have disappeared, so if this gets
 double posted, I'm sorry.

 We disabled all monitoring when we started looking into the issues to
 ensure there was no external load on ES. Everything we are currently seeing
 is just whatever activity ES generates internally.

 My understanding regarding optimizing indices is that you shouldn't
 call it explicitly on indices that are regularly updating, rather you
 should let the background merge process handle things. As the majority of
 our indices regularly update, we don't explicitly call optimize on them. I
 can try to call it on them all and see if it helps.

 As for disk speed, we are currently running ES on SSDs. We have it in
 our roadmap to change that to RAIDed SSDs, but it hasn't been a priority as
 we have been getting acceptable performance thus far.

 On Friday, December 5, 2014 2:59:11 PM UTC-5, Jörg Prante wrote:
>
> Do you have a monitor tool running?
>
> I recommend to switch it off, and optimize your indices, and then
> update your monitoring tools.
>
> Seems you have many segments/slow disk to get them reported in 15s.
>
> Jörg
> Am 05.12.2014 16:10 schrieb "Chris Moore" :
>
>> This is running on Amazon EC2 in a VPC on dedicated instances.
>> Physical network infrastructure is likely fine. Are there specific 
>> network
>> issues you think we should look into?
>>
>> When we are in a problem state, we can communicate between the nodes
>> just fine. I can run curl requests to ES (health checks, etc) from the
>> master node to the data nodes directly and they return as expected. So,
>> there doesn't seem to be a socket exhaustion issue (additionally there 
>> are
>> no kernel errors being reported).
>>
>> It feels like there is a queue/buffer filling up somewhere that once
>> it has availability again, things start working. But, /_cat/thread_pool?v
>> doesn't show anything above 0 (although, when we are in the problem 
>> state,
>> it doesn't return a response if run on master), nodes/hot_threads doesn't
>> show anything going on, etc.
>>
>> On Thursday, December 4, 2014 4:10:37 PM UTC-5, Support Monkey wrote:
>>>
>>> I would think the network is a prime suspect then, as there is no
>>> significant difference between 1.2.x and 1.3.x in relation to memory 
>>> usage.
>>> And you'd certainly see OOMs in node logs if it was a memory issue.
>>>
>>> On Thursday, December 4, 2014 12:45:58 PM UTC-8, Chris Moore wrote:

 There is nothing (literally) in the log of either data node after
 the node joined events and nothing in the master log between index 
 recovery
 and the first error message.

 There are 0 queries run before the errors start occurring (access
 to the nodes is blocked via a firewall, so the only communications are
 between the nodes). We have 50% of the RAM allocated to the heap on 
 each
 node (4GB each).

 This cluster operated without issue under 1.1.2. Did something
 change between 1.1.2 and 1.3.5 that drastically increased idle heap
 requirements?


 On Thursday, December 4, 2014 3:29:23 PM UTC-5, Support Monkey
 wrote:
>
> Generally ReceiveTimeoutTransportException is due to network
> disconnects or a node failing to respond due to heavy load. What does 
> the
> log of pYi3z5PgRh6msJX_armz_A show you? Perhaps it has too little heap
> allocated. Rule of thumb is 1/2 available memory but <= 31GB
>
> On Wednesday, December 3, 2014 12:52:58 PM UTC-8, Jeff Keller
> wrote:
>>
>>>

Re: Running elasticsearch 1.4.2 and kibana 4 as service

[Itamar Syn-Hershko]

I'd actually prefer to install from repositories as they take care of
placing things in the right place and create a user to run ES under

--

Itamar Syn-Hershko
http://code972.com | @synhershko 
Freelance Developer & Consultant
Author of RavenDB in Action 

On Tue, Dec 23, 2014 at 11:45 PM, joergpra...@gmail.com <
joergpra...@gmail.com> wrote:

> Use https://github.com/elasticsearch/elasticsearch-servicewrapper to run
> ES as a service under RHEL 6.
>
> Jörg
>
> On Tue, Dec 23, 2014 at 10:02 PM, Ram Maram  wrote:
>
>> Hi,
>>
>> Right now I am running kibana 3 and elasticsearch 1.3.2 for our ELK
>> stack, I would like to use kibana 4 and elasticsearch 1.4.2.
>>
>> Can someone please let me know how to install kibana 4 and elasticsearch
>> 1.4.2 as a service on linux?
>>
>> I was able to run them manually but I couldn't figure how to run them as
>> a service.
>>
>> Thanks,
>>
>> Ram
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/13d0fe92-bb67-4552-b8da-f482a4291dd1%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoERpmJxSASTyghjpVA7omKqx3N1Y7CdMX_GRpfJh5J6Hg%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zs6xznALSGzjQGz2OGsU%3D3xh88Ab5HOZw9bLVn%3Dcjc3YQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Running elasticsearch 1.4.2 and kibana 4 as service

[joergpra...@gmail.com]

Use https://github.com/elasticsearch/elasticsearch-servicewrapper to run ES
as a service under RHEL 6.

Jörg

On Tue, Dec 23, 2014 at 10:02 PM, Ram Maram  wrote:

> Hi,
>
> Right now I am running kibana 3 and elasticsearch 1.3.2 for our ELK stack,
> I would like to use kibana 4 and elasticsearch 1.4.2.
>
> Can someone please let me know how to install kibana 4 and elasticsearch
> 1.4.2 as a service on linux?
>
> I was able to run them manually but I couldn't figure how to run them as a
> service.
>
> Thanks,
>
> Ram
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/13d0fe92-bb67-4552-b8da-f482a4291dd1%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoERpmJxSASTyghjpVA7omKqx3N1Y7CdMX_GRpfJh5J6Hg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Running elasticsearch 1.4.2 and kibana 4 as service

[Itamar Syn-Hershko]

It's basic Linux administration stuff, see
http://arstechnica.com/civis/viewtopic.php?p=2147913&sid=16c526bdb60201e802cf7f6b8bc598e2#p2147913
for example (and the rest of the instructions on chkconfig). Just update
the script to point at your Kibana files.

--

Itamar Syn-Hershko
http://code972.com | @synhershko 
Freelance Developer & Consultant
Author of RavenDB in Action 

On Tue, Dec 23, 2014 at 11:28 PM, Ram Maram  wrote:

> Thank you Itamar for your quick respone, my distribution is redhat linux
> 6.x and the tools that have installed are logstash, java, elasticsearch.
>
> Can you guide me on how to create the init file for kibana 4 or can I host
> it on apache ?
>
> Thanks,
>
> Ram
>
> On Tuesday, December 23, 2014 4:06:25 PM UTC-5, Itamar Syn-Hershko wrote:
>>
>> Elasticsearch has packages which will do this for you on every Linux
>> distribution: http://www.elasticsearch.org/guide/en/
>> elasticsearch/reference/current/setup-repositories.html
>>
>> For Kibana 4 you'll need to use init.d and /sbin/service , the specifics
>> are going to depend on the distribution and the tools you have installed
>>
>> --
>>
>> Itamar Syn-Hershko
>> http://code972.com | @synhershko 
>> Freelance Developer & Consultant
>> Author of RavenDB in Action 
>>
>> On Tue, Dec 23, 2014 at 11:02 PM, Ram Maram  wrote:
>>
>>> Hi,
>>>
>>> Right now I am running kibana 3 and elasticsearch 1.3.2 for our ELK
>>> stack, I would like to use kibana 4 and elasticsearch 1.4.2.
>>>
>>> Can someone please let me know how to install kibana 4 and elasticsearch
>>> 1.4.2 as a service on linux?
>>>
>>> I was able to run them manually but I couldn't figure how to run them as
>>> a service.
>>>
>>> Thanks,
>>>
>>> Ram
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/13d0fe92-bb67-4552-b8da-f482a4291dd1%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/a2229186-903a-46c7-b132-b0cae3737236%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZvJsvbhB9Qy5yesH004YK-3RWVXc1fvjRz3RBuooK94-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Running elasticsearch 1.4.2 and kibana 4 as service

[Ram Maram]

Thank you Itamar for your quick respone, my distribution is redhat linux 
6.x and the tools that have installed are logstash, java, elasticsearch.

Can you guide me on how to create the init file for kibana 4 or can I host 
it on apache ?

Thanks,

Ram

On Tuesday, December 23, 2014 4:06:25 PM UTC-5, Itamar Syn-Hershko wrote:
>
> Elasticsearch has packages which will do this for you on every Linux 
> distribution: 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-repositories.html
>
> For Kibana 4 you'll need to use init.d and /sbin/service , the specifics 
> are going to depend on the distribution and the tools you have installed 
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko 
> Freelance Developer & Consultant
> Author of RavenDB in Action 
>
> On Tue, Dec 23, 2014 at 11:02 PM, Ram Maram  > wrote:
>
>> Hi,
>>
>> Right now I am running kibana 3 and elasticsearch 1.3.2 for our ELK 
>> stack, I would like to use kibana 4 and elasticsearch 1.4.2.
>>
>> Can someone please let me know how to install kibana 4 and elasticsearch 
>> 1.4.2 as a service on linux?
>>
>> I was able to run them manually but I couldn't figure how to run them as 
>> a service.
>>
>> Thanks,
>>
>> Ram
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/13d0fe92-bb67-4552-b8da-f482a4291dd1%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a2229186-903a-46c7-b132-b0cae3737236%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Kibana histogram display using timestamp with nanosecond

[Bob M]

HI There,

I have two issues and wondering if anyone can help me.

I am using Elasticsearch as my prime database and wondering if it is 
possible to have timestamp up to the nanoseconds and have it successfully 
displayed in Kibana.

Right now the timestamp is define by me and when i tried to see the 
histogram, i only have a big bar instead of individuals timestamp what am i 
doing wrong?

Change the filter setting of kibana to pick up i have added to the system 
but it doesn't seem to work.

Will be expecting a message

Thanks in advance

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f7eb5c55-ad35-4c74-9272-99b907d3717a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Running elasticsearch 1.4.2 and kibana 4 as service

[Itamar Syn-Hershko]

Elasticsearch has packages which will do this for you on every Linux
distribution:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-repositories.html

For Kibana 4 you'll need to use init.d and /sbin/service , the specifics
are going to depend on the distribution and the tools you have installed

--

Itamar Syn-Hershko
http://code972.com | @synhershko 
Freelance Developer & Consultant
Author of RavenDB in Action 

On Tue, Dec 23, 2014 at 11:02 PM, Ram Maram  wrote:

> Hi,
>
> Right now I am running kibana 3 and elasticsearch 1.3.2 for our ELK stack,
> I would like to use kibana 4 and elasticsearch 1.4.2.
>
> Can someone please let me know how to install kibana 4 and elasticsearch
> 1.4.2 as a service on linux?
>
> I was able to run them manually but I couldn't figure how to run them as a
> service.
>
> Thanks,
>
> Ram
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/13d0fe92-bb67-4552-b8da-f482a4291dd1%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtVSuCKY8sBoPhw5yzbdMaHiXB2XsHieBvtvRNfJGL5hg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Running elasticsearch 1.4.2 and kibana 4 as service

[Ram Maram]

Hi,

Right now I am running kibana 3 and elasticsearch 1.3.2 for our ELK stack, 
I would like to use kibana 4 and elasticsearch 1.4.2.

Can someone please let me know how to install kibana 4 and elasticsearch 
1.4.2 as a service on linux?

I was able to run them manually but I couldn't figure how to run them as a 
service.

Thanks,

Ram

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/13d0fe92-bb67-4552-b8da-f482a4291dd1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Marvel question: An alternative for having to enable HTTP for data nodes?

[Nick Canzoneri]

We're testing out marvel and noticed that it causes failures unless HTTP is
enabled for the node. This isn't ideal for data nodes that we've disabled
HTTP on.

Is this just "the way things work" or is there an alternative I'm not aware
of?

It's not a big deal, but does means we can't use the Sniffing Connection
Pool that some clients support because it round robins across all nodes
capable of HTTP traffic.

Thanks,

-- 
Nick Canzoneri
Developer, Wildbit 
Beanstalk , Postmark ,
dploy.io

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKWm5yMtDHLFsOXPagxH4J0fsxFXpOYOhV7XkqSsChwVHnaydg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Startup issues with ES 1.3.5

[Chris Moore]

Updating again:

If we reduce the number of shards per node to below ~350, the system 
operates fine. Once we go above that (number_of_indices * 
number_of_shards_per_index * number_of_replicas / number_of_nodes), we 
start running into the described issues.

On Friday, December 12, 2014 2:11:08 PM UTC-5, Chris Moore wrote:
>
> Just a quick update, we duplicated our test environment to see if this 
> issue was fixed by upgrading to 1.4.1 instead. We received the same errors 
> under 1.4.1.
>
> On Friday, December 5, 2014 4:52:05 PM UTC-5, Chris Moore wrote:
>>
>> As a followup, I closed all the indices on the cluster. I would then open 
>> 1 index and optimize it down to 1 segment. I made it through ~60% of the 
>> indices (and probably ~45% of the data) before the same errors showed up in 
>> the master log and the same behavior resumed.
>>
>> On Friday, December 5, 2014 3:57:12 PM UTC-5, Chris Moore wrote:
>>>
>>> I replied once, but it seems to have disappeared, so if this gets double 
>>> posted, I'm sorry.
>>>
>>> We disabled all monitoring when we started looking into the issues to 
>>> ensure there was no external load on ES. Everything we are currently seeing 
>>> is just whatever activity ES generates internally.
>>>
>>> My understanding regarding optimizing indices is that you shouldn't call 
>>> it explicitly on indices that are regularly updating, rather you should let 
>>> the background merge process handle things. As the majority of our indices 
>>> regularly update, we don't explicitly call optimize on them. I can try to 
>>> call it on them all and see if it helps.
>>>
>>> As for disk speed, we are currently running ES on SSDs. We have it in 
>>> our roadmap to change that to RAIDed SSDs, but it hasn't been a priority as 
>>> we have been getting acceptable performance thus far.
>>>
>>> On Friday, December 5, 2014 2:59:11 PM UTC-5, Jörg Prante wrote:

 Do you have a monitor tool running?

 I recommend to switch it off, and optimize your indices, and then 
 update your monitoring tools.

 Seems you have many segments/slow disk to get them reported in 15s.

 Jörg
 Am 05.12.2014 16:10 schrieb "Chris Moore" :

> This is running on Amazon EC2 in a VPC on dedicated instances. 
> Physical network infrastructure is likely fine. Are there specific 
> network 
> issues you think we should look into?
>
> When we are in a problem state, we can communicate between the nodes 
> just fine. I can run curl requests to ES (health checks, etc) from the 
> master node to the data nodes directly and they return as expected. So, 
> there doesn't seem to be a socket exhaustion issue (additionally there 
> are 
> no kernel errors being reported).
>
> It feels like there is a queue/buffer filling up somewhere that once 
> it has availability again, things start working. But, /_cat/thread_pool?v 
> doesn't show anything above 0 (although, when we are in the problem 
> state, 
> it doesn't return a response if run on master), nodes/hot_threads doesn't 
> show anything going on, etc.
>
> On Thursday, December 4, 2014 4:10:37 PM UTC-5, Support Monkey wrote:
>>
>> I would think the network is a prime suspect then, as there is no 
>> significant difference between 1.2.x and 1.3.x in relation to memory 
>> usage. 
>> And you'd certainly see OOMs in node logs if it was a memory issue.
>>
>> On Thursday, December 4, 2014 12:45:58 PM UTC-8, Chris Moore wrote:
>>>
>>> There is nothing (literally) in the log of either data node after 
>>> the node joined events and nothing in the master log between index 
>>> recovery 
>>> and the first error message.
>>>
>>> There are 0 queries run before the errors start occurring (access to 
>>> the nodes is blocked via a firewall, so the only communications are 
>>> between 
>>> the nodes). We have 50% of the RAM allocated to the heap on each node 
>>> (4GB 
>>> each).
>>>
>>> This cluster operated without issue under 1.1.2. Did something 
>>> change between 1.1.2 and 1.3.5 that drastically increased idle heap 
>>> requirements?
>>>
>>>
>>> On Thursday, December 4, 2014 3:29:23 PM UTC-5, Support Monkey wrote:

 Generally ReceiveTimeoutTransportException is due to network 
 disconnects or a node failing to respond due to heavy load. What does 
 the 
 log of pYi3z5PgRh6msJX_armz_A show you? Perhaps it has too little heap 
 allocated. Rule of thumb is 1/2 available memory but <= 31GB

 On Wednesday, December 3, 2014 12:52:58 PM UTC-8, Jeff Keller wrote:
>
>
> ES Version: 1.3.5
>
> OS: Ubuntu 14.04.1 LTS
>
> Machine: 2 Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz, 8 GB RAM at 
> AWS
>
> master (ip-10-0-1-18), 2 data nodes (ip-10-0

Re: [Kibana] group by request?

[sonnend]

I'm not sure whether I understand your issue in full depth but you can use 
nested aggregations to have hierarchical grouping in Kibana 4. Maybe this 
solves your issue?

Am Montag, 22. Dezember 2014 09:58:57 UTC+1 schrieb stephanos:
>
> Thanks for the answer!
> I think wasn't clear enough: all our log messages already have a 
> requestID. So if there *was* a grouping feature we'd apply it to that 
> field.
>
> I'm just wondering, how do you troubleshoot a issue of a user? When we see 
> a problem we look at all requests of that user in the GAE log viewer. Then 
> you quickly see requests that have non-200 status codes. Then we drill into 
> a request and see all logs of *that* request chronologically. While in 
> Kibana I can also look at all logs from a user ordered by time, but it's 
> not always completely clear which request log messages belong to. It's more 
> like one big stream.
>
> My point is, you should really try out the Google App Engine log viewer - 
> then you would know what you are missing! :)
>
> Stephan
>
>
> On Monday, December 22, 2014 7:38:26 AM UTC+1, Magnus Bäck wrote:
>>
>> On Tuesday, December 16, 2014 at 10:03 CET, 
>>  stephanos  wrote: 
>>
>> > we are using Google App Engine to host our SaaS app. Google offers a 
>> > nice log browser but it is way too slw. So one of my colleagues 
>> > suggested we pipe our logs to logstash and make them accessible via 
>> > Kibana. So far so good, we managed to set everything up. 
>> > But when Kibana was shown to the other team members they weren't 
>> > really excited. It was much faster, yes. It allowed to make better 
>> > queries, yes. BUT it broke the pattern they knew from the Google App 
>> > Engine log browser: 
>> > /some-request 
>> > log message 1 
>> > log message 2 
>> > /another-request 
>> > log message 3 
>> > /yet-another-request 
>> > log message 4 
>> > While Kibana works like this: 
>> > log message 1/some-request 
>> > log message 2/some-request 
>> > log message 3/another-request 
>> > log message 4/yet-another-request 
>> > So basically App Engine groups log messages by request. To get my 
>> > team on board, can we make Kibana do the same? 
>>
>> Not out of the box, no. Kibana doesn't have any such contextual 
>> understanding of messages and currently can't be configured as 
>> such either. 
>>
>> -- 
>> Magnus Bäck| Software Engineer, Development Tools 
>> magnu...@sonymobile.com | Sony Mobile Communications 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0513eb37-5742-46c8-b7c6-fd56f609d0e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: this query brought down my cluster :{"query": {"function_score": {"query": {"match_all":{}}, "random_score": {"seed": 123456}}}}

[Jinyuan Zhou]

OK, I think I miss read old generation gc issue. the line I quoted shows
the old generation region mem size after a young gc. it should be expected
after a young generation gc because some surviving objects are promoted
from young to survival and from survivor to old.

Jinyuan (Jack) Zhou

On Mon, Dec 22, 2014 at 7:28 PM, Jinyuan Zhou 
wrote:

> Hi,
> I like to share my experience and in the same time hope I can get some
> tips.
>
> The query was run against  an index with about 700 million  documents.
> Two things happens,
> 1. The node run  this query crashed. It is the node configured not to
> proccess data.
>
> 2. The data nodes start crazy on GC. eventually old generation gc cannot
> reduce the heep usage and the nodes becomes unresponsive.  in some cases.
> OLD generation gc even increased size of the heap:
>
>
> *2014-12-20 07:21:03,370][WARN ][monitor.jvm  ] [**]
> [gc][young][2796041][224976] duration [1.1s], collections [1]/[1.3s], total
> [1.1s]/[3.4h], memory [21.5gb]->[21.2gb]/[29.8gb], all_pools {[young]
> [1.4gb]->[3.4mb]/[1.4gb]}{[survivor]
> [191.3mb]->[191.3mb]/[191.3mb]}{[old] [19.9gb]->[21gb]/[28.1gb]}*
>
>
> It is a bad query by itself. But I expected ES cluster handles it
> gracefully. It does throw this exception:
>
> * Caused by: org.elasticsearch.common.breaker.CircuitBreakingException:
> [FIELDDATA] Data too large, data for [_uid] would be larger than limit of
> [19206989414 <%5B19206989414>/17.8gb]*
> I guess ES stopped at some point because  field data exceeds the default
> limit. But it is too late to stop the query that caused heap memory issue.
> I am wondering if there is any obvious wrong with my ES cluster
> configuration.
> I have 5 box eah with 125 ram and 32 cores. I deploy two data nodes on
> each of them the heap fixed at 31G  and configuration is favor bulk
> ingesting. I actually saw above 60+K document ingesting through put per
> second.  It was working fine until that query comes.
>
> Thanks,
>
> Jack
>
>
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/k2RkmjuO5OI/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/ae1b7ea6-d801-4d67-b047-69ab54f1f38b%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CANBTPCHAn78f%2BEbZ9R_6sf5jMKYBL%3DDAZeAL2Lg_JZV1L3peWw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

How to emit console.log output when using javascript-lang for custom scripts?

[Tim Heckel]

Hi all -- I'd like to emit to stdout information within my custom .js 
script (located in /elasticsearch/1.4.2/config/scripts/test.js).

I've tried both console.log and print (just a guess), but neither work:

"error": "ElasticsearchIllegalArgumentException[failed to execute script]; 
nested: EcmaError[ReferenceError: \"console\" is not defined. 
(Script2.js#19)]; "
"error": "ElasticsearchIllegalArgumentException[failed to execute script]; 
nested: EcmaError[ReferenceError: \"print\" is not defined. 
(Script1.js#19)]; "

Any way for me to hook into stdout to emit logging information this way? 
Thanks very much.

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ea610adb-986c-4de2-a429-6fb03b45a640%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ignore a field in the scoring

[Roger de Cordova Farias]

Hello

Our documents have metadata indexed with them, but we don't want the
metadata to interfere in the scoring

After a user searches for documents, they can bookmark them (what means we
add more metadata to the document), then in the next search with the same
query the bookmarked document  appears in a lower (worse) position

Is there a way to completely ignore one or more specific fields in the
scoring of every query? as in indexing time or something?

Note that we are not using the metadata field in the query, but yet it
lowers the score of every query

We cannot set the "index" attribute of this field to "no" because we are
gonna use it in other queries

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Best practice for java transport client

[joergpra...@gmail.com]

As said, this is not chatter, it is definitely a feature, for fault
tolerance.

What do you mean by "unlogged communication"? ES does not log node
communication. For this, you will need tools like query profiling
https://github.com/elasticsearch/elasticsearch/pull/6699 or the upcoming
Shield product which offers audit trail capability.

Maybe you have noticed that all nodes (except TransportClients) are always
receiving updates to the cluster state from the master node, otherwise,
they won't be able to continue their work.

Jörg

On Tue, Dec 23, 2014 at 6:33 PM, Elaine Cario  wrote:

> I answered a couple of questions  myself after reproducing it on a local
> system:  modify log4j config for com.elasticsearch to ERROR turns the
> messages off, and indeed setting client.transport.nodes_sample_interval to
> something larger than 5s reduced the chatter.  I'd still like to understand
> though what is happening in the background (i.e. is there also unlogged
> communication going during idle moments?)
>
>
>
> On Tuesday, December 23, 2014 11:34:47 AM UTC-5, Elaine Cario wrote:
>>
>> I'm rather new to Elasticsearch, and have just begun intergrating it with
>> an existing application.
>>
>> The existing process runs in the background waiting for messages from a
>> message queue.  In some cases, those messages are processed as either
>> indexing or percolation requests into ElasticSearch.  So far, everything is
>> working fine in terms of the functionality.  We are using the java
>> TransportClient to connect to ES.  The first time we get a message to go to
>> ElasticSearch, we create that Client, and then re-use that same Client for
>> all other subsequent messages.  The background process essentially runs
>> forever (although it does get periodically restarted for various reasons),
>> so the Client sticks around for as long as the process.
>>
>> A couple of questions:
>>
>> We noticed that once we initialize the client, we get a HUGE number of
>> TRACE messages in the application's log (these 2 messages repeat every 5
>> secs):
>>
>> 2014-12-22 18:01:18 TRACE ChildMemoryCircuitBreaker.addWithoutBreaking()
>> - [Dr. Lemuel Dorcas] [REQUEST] Adjusted breaker by [16440] bytes, now
>> [16440]
>> 2014-12-22 18:01:18 TRACE ChildMemoryCircuitBreaker.addWithoutBreaking()
>> - [Dr. Lemuel Dorcas] [REQUEST] Adjusted breaker by [-16440] bytes, now [0]
>>
>> They just repeat forever.  If I restart the entire process, they go away,
>> until a message triggers the creation of a new Client.  I'm presuming that
>> 1. there is some way that I can turn off these messages through log4j and
>> 2. I can reduce the interval of these message by setting
>> client.transport.nodes_sampler_interval.  Can anyone confirm that this
>> will indeed work? For the logging part, any hints on whether I need to do
>> this in log4j or elsewhere would be appreciated, as it's painful setting up
>> the environment for each experiment (and I spend way too much time hunting
>> and pecking in log4j configs trying to turn off verbose logging of
>> 3rd-party apps, just google apache httpclient logging issues to see what I
>> mean!).
>>
>> But that also leads me to ask the next question, as I have some concerns
>> that the once created, the Client isn't quietly sitting in the background
>> waiting to be called upon by the main process, but is either sending or
>> receiving ping messages (to itself? to the remote ES cluster?).
>>
>> So now I wonder if keeping the Client alive over a long period of time
>> the best practice, since it seems to be generating quite a bit of extra
>> traffic or using CPU cycles for not much benefit?  Or should I just close
>> the client when we're done processing a message and re-open a new Client
>> when needed for a subsequent message (not all messages will need ES)?
>>
>> Thanks!
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/2a450277-fbca-4e40-a354-39cda58f4987%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEKuJXCATKZKDrXvuFxw%2BRNtH6gSszGYt_sUqG%3D8AZwDw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Question about highlight query.

[Yang Liu]

No one knows anything about this? I really appreciate anything you offered.

On Monday, December 22, 2014 5:27:57 PM UTC-5, Yang Liu wrote:
>
> Hi, guys,
> I have a question about highlight query in ES.
> *Below is my query,*
> {
>   "_source": [
>  
>  .
>   ],
>   "highlight": {
> "fields": {
>   "FDS_ATTACHMENTS": {
> "type": "plain"
>   },
>   "FDS_ATTACHMENTS.no_stem": {
> "type": "plain"
>   },
>   "FDS_ATTACHMENTS.with_case": {
> "type": "plain"
>   },
>   "headline": {
> "type": "plain"
>   },
>   "headline.no_stem": {
> "type": "plain"
>   },
>   "headline.with_case": {
> "type": "plain"
>   }
> },
> "fragment_size": 500,
> "highlight_query": {
>   "bool": {
> "must": [
>   {
> "bool": {
>   "minimum_should_match": 1,
>   "should": [
> {
>   "span_near": {
> "clauses": [
>   {
> "span_term": {
>   "FDS_ATTACHMENTS.no_stem": "rights"
> }
>   },
>   {
> "span_term": {
>   "FDS_ATTACHMENTS.no_stem": "agreement"
> }
>   }
> ],
> "in_order": true,
> "slop": 0
>   }
> }
>   ]
> }
>   },
>   {
> "bool": {
>   "minimum_should_match": 1,
>   "should": [
> {
>   "span_near": {
> "clauses": [
>   {
> "span_term": {
>   "FDS_ATTACHMENTS.no_stem": "rights"
> }
>   },
>   {
> "span_term": {
>   "FDS_ATTACHMENTS.no_stem": "agreement"
> }
>   },
>   {
> "span_term": {
>   "FDS_ATTACHMENTS.no_stem": "merger"
> }
>   }
> ],
> "in_order": false,
> "slop": 5
>   }
> }
>   ]
> }
>   }
> ]
>   }
> },
> "number_of_fragments": 50,
> "post_tags": [
>   ""
> ],
> "pre_tags": [
>   ""
> ],
> "require_field_match": true
>   },
>   "query": {
> "filtered": {
>   "filter": {
> "range": {
>   "story_datetime": {
> "gte": "20141221t00",
> "lte": "20141222t235959"
>   }
> }
>   },
>   "query": {
> "bool": {
>   "must": [
> {
>   "bool": {
> "minimum_should_match": 1,
> "should": [
>   {
> "span_near": {
>   "clauses": [
> {
>   "span_term": {
> "FDS_ATTACHMENTS.no_stem": "rights"
>   }
> },
> {
>   "span_term": {
> "FDS_ATTACHMENTS.no_stem": "agreement"
>   }
> }
>   ],
>   "in_order": true,
>   "slop": 0
> }
>   },
>   {
> "span_near": {
>   "clauses": [
> {
>   "span_term": {
> "headline.no_stem": "rights"
>   }
> },
> {
>   "span_term": {
> "headline.no_stem": "agreement"
>   }
> }
>   ],
>   "in_order": true,
>   "slop": 0
> }
>   },
>   {
> "span_near": {
>   "clauses": [
> {
>   "span_term": {
> "headline2.no_stem": "rights"
>   }
> },
> {
>   "span_term": {
> "headline2.no_stem": "agreement"
>   }
> }
>   ],
>   "in_order": true,
>   "slop": 0
> }
>   

Re: Best practice for java transport client

[Elaine Cario]

I answered a couple of questions  myself after reproducing it on a local 
system:  modify log4j config for com.elasticsearch to ERROR turns the 
messages off, and indeed setting client.transport.nodes_sample_interval to 
something larger than 5s reduced the chatter.  I'd still like to understand 
though what is happening in the background (i.e. is there also unlogged 
communication going during idle moments?)


On Tuesday, December 23, 2014 11:34:47 AM UTC-5, Elaine Cario wrote:
>
> I'm rather new to Elasticsearch, and have just begun intergrating it with 
> an existing application.  
>
> The existing process runs in the background waiting for messages from a 
> message queue.  In some cases, those messages are processed as either 
> indexing or percolation requests into ElasticSearch.  So far, everything is 
> working fine in terms of the functionality.  We are using the java 
> TransportClient to connect to ES.  The first time we get a message to go to 
> ElasticSearch, we create that Client, and then re-use that same Client for 
> all other subsequent messages.  The background process essentially runs 
> forever (although it does get periodically restarted for various reasons), 
> so the Client sticks around for as long as the process.
>
> A couple of questions:
>
> We noticed that once we initialize the client, we get a HUGE number of 
> TRACE messages in the application's log (these 2 messages repeat every 5 
> secs):
>
> 2014-12-22 18:01:18 TRACE ChildMemoryCircuitBreaker.addWithoutBreaking() - 
> [Dr. Lemuel Dorcas] [REQUEST] Adjusted breaker by [16440] bytes, now [16440]
> 2014-12-22 18:01:18 TRACE ChildMemoryCircuitBreaker.addWithoutBreaking() - 
> [Dr. Lemuel Dorcas] [REQUEST] Adjusted breaker by [-16440] bytes, now [0]
>
> They just repeat forever.  If I restart the entire process, they go away, 
> until a message triggers the creation of a new Client.  I'm presuming that 
> 1. there is some way that I can turn off these messages through log4j and 
> 2. I can reduce the interval of these message by setting 
> client.transport.nodes_sampler_interval.  Can anyone confirm that this will 
> indeed work? For the logging part, any hints on whether I need to do this 
> in log4j or elsewhere would be appreciated, as it's painful setting up the 
> environment for each experiment (and I spend way too much time hunting and 
> pecking in log4j configs trying to turn off verbose logging of 3rd-party 
> apps, just google apache httpclient logging issues to see what I mean!).
>
> But that also leads me to ask the next question, as I have some concerns 
> that the once created, the Client isn't quietly sitting in the background 
> waiting to be called upon by the main process, but is either sending or 
> receiving ping messages (to itself? to the remote ES cluster?).  
>
> So now I wonder if keeping the Client alive over a long period of time the 
> best practice, since it seems to be generating quite a bit of extra traffic 
> or using CPU cycles for not much benefit?  Or should I just close the 
> client when we're done processing a message and re-open a new Client when 
> needed for a subsequent message (not all messages will need ES)?
>
> Thanks!
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2a450277-fbca-4e40-a354-39cda58f4987%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Combing relevancy and trending scores

[hespoddi]

We originally thought rescoring would work as well! I actually implemented 
it, but low relevancy documents continued to show up at the top of our 
results. I didn't understand this at first, but on re-reading the 
documentation I saw what the problem was: the rescore is executed on the 
shard *before* the results are returned to the node handling the overall 
request. So: if an individual shard only had low relevancy, high trending 
documents for a query, then we'd run into the exact same problem as before. 
And, in fact, that does seem to happen with enough frequency to be an issue 
for us. 

At the moment, the only way I see to solve the problem is to do a 
post-processing step on the returned documents. 

On Monday, December 22, 2014 11:50:24 PM UTC-5, vineeth mohan wrote:
>
> Hi , 
>
> Maybe you are looking for this - 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-rescore.html
>
> Thanks
> Vineeth
>
> On Tue, Dec 23, 2014 at 2:46 AM, hespoddi  > wrote:
>
>> Hi all,
>>
>> The problem isn't really the query. The problem is we'd like the limit 
>> the results of the query to just "high" scores before we apply the 
>> function_score. There is a min_score parameter we could use:
>>
>>
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-min-score.html
>>
>> But what the min_score should be will, obviously, vary significantly 
>> depending on the query. Ideally, we'd set the min_score to some percentile 
>> of the max score for query, but I don't think that's possible:
>>
>> https://github.com/elasticsearch/elasticsearch/issues/719
>>
>> I was curious if anyone had any other ideas about how to do this (or 
>> something close)?
>>
>> -Chris
>>
>> On Monday, December 22, 2014 1:34:08 PM UTC-5, vineeth mohan wrote:
>>>
>>> Hi ,
>>>
>>> I dont see why it should run into performance issue.
>>> In anyway you do it , the _score and score due to  a field have to be 
>>> computed/loaded.
>>> If you precompile your script by placing it in config directory , that 
>>> should be good enough.
>>> Also feel free to write the same in Java code and attach that as script.
>>>
>>> Thanks
>>> Vineeth
>>>
>>> On Mon, Dec 22, 2014 at 10:30 PM, Scott Decker  
>>> wrote:
>>>
 If you did the script score path, don't you run into performance 
 issues?  I would think, running say thousands of queries like that would 
 probably not be performant.
 unless I am missing something.

 On Sunday, December 21, 2014 8:01:06 PM UTC-8, vineeth mohan wrote:
>
> Hello , 
>
> My advice would be to use the script function type. Inside it you can 
> access _score which is the score given by the query and the value of the 
> field. Mix them together in whatever logic you want. 
>
> Thanks
> Vineeth
>
> On Thu, Dec 11, 2014 at 7:28 PM, hespoddi  
> wrote:
>
>> Hi all,
>>
>> We'd like to combine the query score with our own custom trending 
>> score for a given document. Currently, our query looks like:
>>
>> {
>> "query": {
>> "filtered": {
>> "filter": {
>> "and": [
>> {
>> "range": {
>> "trendingScore": {
>> "gt": 0.0
>> }
>> }
>> }
>> ]
>> }, 
>> "query": {
>> "function_score": {
>> "functions": [
>> {
>> "field_value_factor": {
>> "field": "trendingScore"
>> }
>> }
>> ], 
>> "query": {
>> "bool": {
>> "minimum_should_match": 1, 
>> "should": [
>> {
>> "match_phrase": {
>> "title": {
>> "boost": 1.5, 
>> "query": ""
>> }
>> }
>> }, 
>> {
>> "match_phrase": {
>> "content": {
>> "boost": 1.0, 
>> "query": ""
>> }
>> }
>> }
>>  

Re: Getting error while starting server

[joergpra...@gmail.com]

JBoss performed a double close in concurrency to ES which actually was
still active in closing down itself. Check if you are using a
ServletContextListener for the client instance. There are
 contextInitialized()/contextDestroyed() methods which are called by JBoss
exactly once during the life cycle of a web application.

Jörg

On Tue, Dec 23, 2014 at 3:09 PM, vineeth mohan 
wrote:

> Hi,
>
> Hope you are using maven or else there are hell lot of dependent jars you
> have to include yourself.
>
> Thanks
>   Vineeth
> On Dec 23, 2014 4:54 PM, "Vijayakumari B N" 
> wrote:
>
>> Hi,
>>
>> I used only one jar for elasticsearch i.e. elasticsearch-1.4.1.jar, i
>> don't think we need any other jars apart from this and my elastic server is
>> also running. Please let me know if there are any additional jars which i
>> need to add.
>>
>> Thanks,
>> Vijaya
>>
>> On Tuesday, December 23, 2014 2:38:43 PM UTC+5:30, vineeth mohan wrote:
>>>
>>> Hi ,
>>>
>>> Make sure all the dependent JAR files are present.
>>> Its missing some of these.
>>>
>>> Thanks
>>> Vineeth
>>>
>>> On Tue, Dec 23, 2014 at 11:15 AM, Vijayakumari B N >> > wrote:
>>>
 Hi,

 I get the below error when i tried to restart my jboss server after
 integrating with elastic server. I am using elastic search 1.4 version. Can
 some one let me know where i am doing wrong. Elastic server absolutely work
 fine apart from below error.

 public static Client getESClient() {
 if (esClient == null) {
  Client esClient = new TransportClient().addTransportAddress(new
 InetSocketTransportAddress("127.0.0.1", 9300));
 return esClient;
 }
 return esClient;
 }




 2014-12-23 10:44:46,933 INFO  
 [org.apache.catalina.loader.WebappClassLoader]
 Illegal access: this web application instance has been stopped already.
 Could not load org.elasticsearch.transport.
 SendRequestTransportException.  The eventual following stack trace is
 caused by an error thrown for debugging purposes as well as to attempt to
 terminate the thread which caused the illegal access, and has no functional
 impact.
 java.lang.IllegalStateException
 at org.apache.catalina.loader.WebappClassLoader.loadClass(
 WebappClassLoader.java:1244)
 at org.apache.catalina.loader.WebappClassLoader.loadClass(
 WebappClassLoader.java:1204)
 at org.elasticsearch.transport.TransportService.sendRequest(
 TransportService.java:213)
 at org.elasticsearch.transport.TransportService.submitRequest(
 TransportService.java:177)
 at org.elasticsearch.client.transport.TransportClientNodesService$
 SimpleNodeSampler.doSample(TransportClientNodesService.java:349)
 at org.elasticsearch.client.transport.TransportClientNodesService$
 NodeSampler.sample(TransportClientNodesService.java:288)
 at org.elasticsearch.client.transport.TransportClientNodesService$
 ScheduledNodeSampler.run(TransportClientNodesService.java:321)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(
 ThreadPoolExecutor.java:1145)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(
 ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)
 2014-12-23 10:44:46,935 INFO  [org.elasticsearch.client.transport]
 [Hammerhead] failed to get node info for [#transport#-1][BLRD17730][
 inet[/127.0.0.1:9300]], disconnecting...
 java.lang.NoClassDefFoundError: org/elasticsearch/transport/
 SendRequestTransportException
 at org.elasticsearch.transport.TransportService.sendRequest(
 TransportService.java:213)
 at org.elasticsearch.transport.TransportService.submitRequest(
 TransportService.java:177)
 at org.elasticsearch.client.transport.TransportClientNodesService$
 SimpleNodeSampler.doSample(TransportClientNodesService.java:349)
 at org.elasticsearch.client.transport.TransportClientNodesService$
 NodeSampler.sample(TransportClientNodesService.java:288)
 at org.elasticsearch.client.transport.TransportClientNodesService$
 ScheduledNodeSampler.run(TransportClientNodesService.java:321)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(
 ThreadPoolExecutor.java:1145)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(
 ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)
 Caused by: java.lang.ClassNotFoundException:
 org.elasticsearch.transport.SendRequestTransportException
 at org.apache.catalina.loader.WebappClassLoader.loadClass(
 WebappClassLoader.java:1358)
 at org.apache.catalina.loader.WebappClassLoader.loadClass(
 WebappClassLoader.java:1204)
 ... 8 more
 2014-12-23 10:44:46,943 INFO  
 [org.apache.catalina.loader.WebappClassLoader]
 Illegal access: this web application instance has been stopped already.
 Could not load org.elasticsearch.common.netty.channel.Channels$4.  The
 eventual fo

Re: Best practice for java transport client

[joergpra...@gmail.com]

1. The messages are harmless. You should not set log level to TRACE. Check
config/logging.yml, there you can set the global log level to INFO.

2. client.transport.nodes_sampler_interval is a setting for the interval
between pings where the client checks for nodes being still alive. It has
nothing to do with the circuit breaker which generates diagnostic messages.

Once up, you should let the client instance run being active all the time.
The client automatically pings connected node each 5 seconds to get the
newest information about the cluster, including nodes that went down/up.
This is very beneficial in case of node failures or cluster maintenance
because the client can drop broken connections and switch over to new nodes
automatically.

Jörg


On Tue, Dec 23, 2014 at 5:34 PM, Elaine Cario  wrote:

> I'm rather new to Elasticsearch, and have just begun intergrating it with
> an existing application.
>
> The existing process runs in the background waiting for messages from a
> message queue.  In some cases, those messages are processed as either
> indexing or percolation requests into ElasticSearch.  So far, everything is
> working fine in terms of the functionality.  We are using the java
> TransportClient to connect to ES.  The first time we get a message to go to
> ElasticSearch, we create that Client, and then re-use that same Client for
> all other subsequent messages.  The background process essentially runs
> forever (although it does get periodically restarted for various reasons),
> so the Client sticks around for as long as the process.
>
> A couple of questions:
>
> We noticed that once we initialize the client, we get a HUGE number of
> TRACE messages in the application's log (these 2 messages repeat every 5
> secs):
>
> 2014-12-22 18:01:18 TRACE ChildMemoryCircuitBreaker.addWithoutBreaking() -
> [Dr. Lemuel Dorcas] [REQUEST] Adjusted breaker by [16440] bytes, now [16440]
> 2014-12-22 18:01:18 TRACE ChildMemoryCircuitBreaker.addWithoutBreaking() -
> [Dr. Lemuel Dorcas] [REQUEST] Adjusted breaker by [-16440] bytes, now [0]
>
> They just repeat forever.  If I restart the entire process, they go away,
> until a message triggers the creation of a new Client.  I'm presuming that
> 1. there is some way that I can turn off these messages through log4j and
> 2. I can reduce the interval of these message by setting
> client.transport.nodes_sampler_interval.  Can anyone confirm that this will
> indeed work? For the logging part, any hints on whether I need to do this
> in log4j or elsewhere would be appreciated, as it's painful setting up the
> environment for each experiment (and I spend way too much time hunting and
> pecking in log4j configs trying to turn off verbose logging of 3rd-party
> apps, just google apache httpclient logging issues to see what I mean!).
>
> But that also leads me to ask the next question, as I have some concerns
> that the once created, the Client isn't quietly sitting in the background
> waiting to be called upon by the main process, but is either sending or
> receiving ping messages (to itself? to the remote ES cluster?).
>
> So now I wonder if keeping the Client alive over a long period of time the
> best practice, since it seems to be generating quite a bit of extra traffic
> or using CPU cycles for not much benefit?  Or should I just close the
> client when we're done processing a message and re-open a new Client when
> needed for a subsequent message (not all messages will need ES)?
>
> Thanks!
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/598245d1-47e9-4ea6-98ab-0c2790e4c4f1%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoESo66TUoaoGYr-_aoLg0CUN6VbJY8re%2BqA4sAyyPu%2BpQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Best practice for java transport client

[Elaine Cario]

I'm rather new to Elasticsearch, and have just begun intergrating it with 
an existing application.  

The existing process runs in the background waiting for messages from a 
message queue.  In some cases, those messages are processed as either 
indexing or percolation requests into ElasticSearch.  So far, everything is 
working fine in terms of the functionality.  We are using the java 
TransportClient to connect to ES.  The first time we get a message to go to 
ElasticSearch, we create that Client, and then re-use that same Client for 
all other subsequent messages.  The background process essentially runs 
forever (although it does get periodically restarted for various reasons), 
so the Client sticks around for as long as the process.

A couple of questions:

We noticed that once we initialize the client, we get a HUGE number of 
TRACE messages in the application's log (these 2 messages repeat every 5 
secs):

2014-12-22 18:01:18 TRACE ChildMemoryCircuitBreaker.addWithoutBreaking() - 
[Dr. Lemuel Dorcas] [REQUEST] Adjusted breaker by [16440] bytes, now [16440]
2014-12-22 18:01:18 TRACE ChildMemoryCircuitBreaker.addWithoutBreaking() - 
[Dr. Lemuel Dorcas] [REQUEST] Adjusted breaker by [-16440] bytes, now [0]

They just repeat forever.  If I restart the entire process, they go away, 
until a message triggers the creation of a new Client.  I'm presuming that 
1. there is some way that I can turn off these messages through log4j and 
2. I can reduce the interval of these message by setting 
client.transport.nodes_sampler_interval.  Can anyone confirm that this will 
indeed work? For the logging part, any hints on whether I need to do this 
in log4j or elsewhere would be appreciated, as it's painful setting up the 
environment for each experiment (and I spend way too much time hunting and 
pecking in log4j configs trying to turn off verbose logging of 3rd-party 
apps, just google apache httpclient logging issues to see what I mean!).

But that also leads me to ask the next question, as I have some concerns 
that the once created, the Client isn't quietly sitting in the background 
waiting to be called upon by the main process, but is either sending or 
receiving ping messages (to itself? to the remote ES cluster?).  

So now I wonder if keeping the Client alive over a long period of time the 
best practice, since it seems to be generating quite a bit of extra traffic 
or using CPU cycles for not much benefit?  Or should I just close the 
client when we're done processing a message and re-open a new Client when 
needed for a subsequent message (not all messages will need ES)?

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/598245d1-47e9-4ea6-98ab-0c2790e4c4f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fielddata cache and doc values

[vineeth mohan]

Hi  ,

Yes , definitely doc values would be a better idea.
As it is not 100% memory resident , it will give a lot better stability and
memory optimization to the system.
On the flip side, performance might go down to , say 10 to 15%.

Thanks
 Vineeth

On Tue, Dec 23, 2014 at 8:47 PM, Han JU  wrote:

> Hi,
>
> We are reviewing our ElasticSearch setup & settings, here's a question on
> fielddata cache and usage of doc values.
>
> Currently we set fielddata cache to be 30% of the heap size and we've
> enabled doc_values for all fields that we want to sort or aggregate (except
> 2 boolean fields).
> As I understand, doc values are a form of on disk cache of field data. So
> my questions is, by using doc values, can we reduce the memory fielddata
> cache size? Say to 10%, is it a good idea?
>
> Thanks!
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/914962d7-66ca-467e-bd35-fe520317dda9%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DXOZHcYj%2BZTGTuF%3DCmuDcvawObAxskS2N4CMBJ53RCwQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Dedicated master node and client node

[Han JU]

Hi,

Currently in our ElasticSearch cluster we have one dedicated master node 
(master: true, data: false) and it's also the nodes that receives all 
requests. This is bad and we want to change it.
Here's some questions:

  1. Is it a good idea to merge dedicated master nodes with client nodes? 
Say in the cluster, we have 3 machines at dedicated master nodes and we do 
some kind of round robin DNS on these 3 for incoming requests.
  2. If we opt for separated, dedicated master, could we use smaller 
instances? Say 3 machines of m3.medium.
  3. In general, what's the general suggestion for a cluster setup? We are 
currently talking about: 5 data nodes, 3 dedicated master node and 2 client 
nodes.

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4a6ad3d7-b8dd-4e7d-8395-a583b3b70df4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Fielddata cache and doc values

[Han JU]

Hi,

We are reviewing our ElasticSearch setup & settings, here's a question on 
fielddata cache and usage of doc values.

Currently we set fielddata cache to be 30% of the heap size and we've 
enabled doc_values for all fields that we want to sort or aggregate (except 
2 boolean fields).
As I understand, doc values are a form of on disk cache of field data. So 
my questions is, by using doc values, can we reduce the memory fielddata 
cache size? Say to 10%, is it a good idea?

Thanks!


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/914962d7-66ca-467e-bd35-fe520317dda9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Plugin broadcasting/scalability

[AndrewK]

Thank you for the information - that has clarified things for me!
Andrew

Am Dienstag, 23. Dezember 2014 14:38:34 UTC+1 schrieb Jörg Prante:
>
> 1. Plugins only run on the node they are installed on. For this, a plugin 
> should be installed on all nodes, no matter if it is used on all nodes or 
> not. Plugins can have actions implemented that can be addressed by other 
> nodes (broadcast operations).
>
> 2. The NodeClient you get is sitting on the local node but can potentially 
> address all nodes. Bulk imports automatically use nodes in parallel, i.e. 
> they forward actions to the nodes where the shards of the addressed index 
> of an operation live.
>
> The JDBC feeder mode works off the cluster and uses a TransportClient. The 
> reason for the feeder mode is that it works without being dependent on an 
> ES node life cycle - rivers are flaky when they are forced to hop to 
> another node and are restarted when the river nodes goes down. 
>
> Rivers use NodeClient instances and the bulk indexing scales well. The 
> part that does not scale in rivers is the river management by ES and the 
> "pull" style, the non-existent distributed architecture of the fetch phase. 
> So, a river instance soon becomes a bottleneck or a single point of 
> failure, a design which does not fit to the well-designed architecture of 
> the rest of the ES system.
>
> Jörg
>
>
> On Tue, Dec 23, 2014 at 10:23 AM, AndrewK  > wrote:
>
>> A while ago I used a plugin on a project to control the migration to, and 
>> the maintenance of, a secondary data source/index in ElasticSearch. It 
>> worked really well and avoided lots of messy dependency management in the 
>> host application, as all the logic was "hidden" behind an ES rest end-point.
>>
>> But I was never really sure how scalable this was. I have two questions:
>>
>> 1.) when I register a plugin, is that available for all nodes? (I'm 
>> assuming "yes", which would mean controlling parallel/overlapping call outs 
>> is important)
>>
>> 2.) when i call my end-point, I have a Client Object passed in the 
>> constructor: when I debug this, it is an instance of NodeClient which 
>> presumably means I am working on a single node. Is it possible to construct 
>> a TransportClient from this, so that I can address more than node and take 
>> advantage of e.g. bulk imports in parallel?
>>
>> Regarding 2.), I've had a look at the JDBC River Code and the Feeder mode 
>> (addressing the cluster from a component running in a separate JVM) seems 
>> to be there precisely because of this drawback. The River mode seems to 
>> work off one node like my plugin did/does.
>>
>> Is my understanding correct?
>>
>> Andrew
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/16664d7b-3934-4a90-8b1f-209ede3ff08c%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/aee5743a-f706-4034-b139-4a0582d6223a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Getting error while starting server

[vineeth mohan]

Hi,

Hope you are using maven or else there are hell lot of dependent jars you
have to include yourself.

Thanks
  Vineeth
On Dec 23, 2014 4:54 PM, "Vijayakumari B N" 
wrote:

> Hi,
>
> I used only one jar for elasticsearch i.e. elasticsearch-1.4.1.jar, i
> don't think we need any other jars apart from this and my elastic server is
> also running. Please let me know if there are any additional jars which i
> need to add.
>
> Thanks,
> Vijaya
>
> On Tuesday, December 23, 2014 2:38:43 PM UTC+5:30, vineeth mohan wrote:
>>
>> Hi ,
>>
>> Make sure all the dependent JAR files are present.
>> Its missing some of these.
>>
>> Thanks
>> Vineeth
>>
>> On Tue, Dec 23, 2014 at 11:15 AM, Vijayakumari B N 
>> wrote:
>>
>>> Hi,
>>>
>>> I get the below error when i tried to restart my jboss server after
>>> integrating with elastic server. I am using elastic search 1.4 version. Can
>>> some one let me know where i am doing wrong. Elastic server absolutely work
>>> fine apart from below error.
>>>
>>> public static Client getESClient() {
>>> if (esClient == null) {
>>>  Client esClient = new TransportClient().addTransportAddress(new
>>> InetSocketTransportAddress("127.0.0.1", 9300));
>>> return esClient;
>>> }
>>> return esClient;
>>> }
>>>
>>>
>>>
>>>
>>> 2014-12-23 10:44:46,933 INFO  [org.apache.catalina.loader.WebappClassLoader]
>>> Illegal access: this web application instance has been stopped already.
>>> Could not load org.elasticsearch.transport.
>>> SendRequestTransportException.  The eventual following stack trace is
>>> caused by an error thrown for debugging purposes as well as to attempt to
>>> terminate the thread which caused the illegal access, and has no functional
>>> impact.
>>> java.lang.IllegalStateException
>>> at org.apache.catalina.loader.WebappClassLoader.loadClass(
>>> WebappClassLoader.java:1244)
>>> at org.apache.catalina.loader.WebappClassLoader.loadClass(
>>> WebappClassLoader.java:1204)
>>> at org.elasticsearch.transport.TransportService.sendRequest(
>>> TransportService.java:213)
>>> at org.elasticsearch.transport.TransportService.submitRequest(
>>> TransportService.java:177)
>>> at org.elasticsearch.client.transport.TransportClientNodesService$
>>> SimpleNodeSampler.doSample(TransportClientNodesService.java:349)
>>> at org.elasticsearch.client.transport.TransportClientNodesService$
>>> NodeSampler.sample(TransportClientNodesService.java:288)
>>> at org.elasticsearch.client.transport.TransportClientNodesService$
>>> ScheduledNodeSampler.run(TransportClientNodesService.java:321)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> ThreadPoolExecutor.java:1145)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>> 2014-12-23 10:44:46,935 INFO  [org.elasticsearch.client.transport]
>>> [Hammerhead] failed to get node info for [#transport#-1][BLRD17730][
>>> inet[/127.0.0.1:9300]], disconnecting...
>>> java.lang.NoClassDefFoundError: org/elasticsearch/transport/
>>> SendRequestTransportException
>>> at org.elasticsearch.transport.TransportService.sendRequest(
>>> TransportService.java:213)
>>> at org.elasticsearch.transport.TransportService.submitRequest(
>>> TransportService.java:177)
>>> at org.elasticsearch.client.transport.TransportClientNodesService$
>>> SimpleNodeSampler.doSample(TransportClientNodesService.java:349)
>>> at org.elasticsearch.client.transport.TransportClientNodesService$
>>> NodeSampler.sample(TransportClientNodesService.java:288)
>>> at org.elasticsearch.client.transport.TransportClientNodesService$
>>> ScheduledNodeSampler.run(TransportClientNodesService.java:321)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> ThreadPoolExecutor.java:1145)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.elasticsearch.transport.SendRequestTransportException
>>> at org.apache.catalina.loader.WebappClassLoader.loadClass(
>>> WebappClassLoader.java:1358)
>>> at org.apache.catalina.loader.WebappClassLoader.loadClass(
>>> WebappClassLoader.java:1204)
>>> ... 8 more
>>> 2014-12-23 10:44:46,943 INFO  [org.apache.catalina.loader.WebappClassLoader]
>>> Illegal access: this web application instance has been stopped already.
>>> Could not load org.elasticsearch.common.netty.channel.Channels$4.  The
>>> eventual following stack trace is caused by an error thrown for debugging
>>> purposes as well as to attempt to terminate the thread which caused the
>>> illegal access, and has no functional impact.
>>> java.lang.IllegalStateException
>>> at org.apache.catalina.loader.WebappClassLoader.loadClass(
>>> WebappClassLoader.java:1244)
>>> at org.apache.catalina.loader.WebappClassLoader.loadClass(
>>> WebappClassLoader.java:1204)
>>> at org.elasticsearch.common.netty.channel.Channels.
>>> fireChannelDisconnectedLater(Channels.

Re: Plugin broadcasting/scalability

[joergpra...@gmail.com]

1. Plugins only run on the node they are installed on. For this, a plugin
should be installed on all nodes, no matter if it is used on all nodes or
not. Plugins can have actions implemented that can be addressed by other
nodes (broadcast operations).

2. The NodeClient you get is sitting on the local node but can potentially
address all nodes. Bulk imports automatically use nodes in parallel, i.e.
they forward actions to the nodes where the shards of the addressed index
of an operation live.

The JDBC feeder mode works off the cluster and uses a TransportClient. The
reason for the feeder mode is that it works without being dependent on an
ES node life cycle - rivers are flaky when they are forced to hop to
another node and are restarted when the river nodes goes down.

Rivers use NodeClient instances and the bulk indexing scales well. The part
that does not scale in rivers is the river management by ES and the "pull"
style, the non-existent distributed architecture of the fetch phase. So, a
river instance soon becomes a bottleneck or a single point of failure, a
design which does not fit to the well-designed architecture of the rest of
the ES system.

Jörg


On Tue, Dec 23, 2014 at 10:23 AM, AndrewK  wrote:

> A while ago I used a plugin on a project to control the migration to, and
> the maintenance of, a secondary data source/index in ElasticSearch. It
> worked really well and avoided lots of messy dependency management in the
> host application, as all the logic was "hidden" behind an ES rest end-point.
>
> But I was never really sure how scalable this was. I have two questions:
>
> 1.) when I register a plugin, is that available for all nodes? (I'm
> assuming "yes", which would mean controlling parallel/overlapping call outs
> is important)
>
> 2.) when i call my end-point, I have a Client Object passed in the
> constructor: when I debug this, it is an instance of NodeClient which
> presumably means I am working on a single node. Is it possible to construct
> a TransportClient from this, so that I can address more than node and take
> advantage of e.g. bulk imports in parallel?
>
> Regarding 2.), I've had a look at the JDBC River Code and the Feeder mode
> (addressing the cluster from a component running in a separate JVM) seems
> to be there precisely because of this drawback. The River mode seems to
> work off one node like my plugin did/does.
>
> Is my understanding correct?
>
> Andrew
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/16664d7b-3934-4a90-8b1f-209ede3ff08c%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHnkAd6UoFFj%2Biv%2B5hZo63X%3D9_LiwTM3sQiE-OFbZPGNw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Exception with posting jdbc river config

[joergpra...@gmail.com]

Use data binary flag

curl -XPUT "http://localhost:9200/_river/mybooks/_meta";  --data-binary
"@put.json"

Jörg

On Tue, Dec 23, 2014 at 12:25 PM, Ashutosh Parab 
wrote:

> I have created a put.json file with the following contents:-
>
> {
> "type":"jdbc",
> "jdbc":
> {"driver":"com.microsoft.sqlserver.jdbc.SQLServerDriver",
>
>
> "url":"jdbc:sqlserver://LOCALHOST\\SQLEXPRESS;databaseName=Report
>
> _ME",
>   "user":"user",
>   "password":"password","sql":"select * from
>
> [Report.[dbo].[abc]"
> }
>
> }
>
> I try to post config using the command  *curl -XPUT
> "http://localhost:9200/_river/mybooks/_meta
> " -d put.json.*
>
> Got the below error:-
> C:\>curl -XPUT "http://localhost:9200/_river/mybooks/_meta"; -d put.json
> {"error":"MapperParsingException[failed to parse]; nested:
> ElasticsearchParseExc
> eption[Failed to derive xcontent from (offset=0, length=8): [112, 117,
> 116, 46,
> 106, 115, 111, 110]]; ","status":400}
>
> Please advice
>
>
> On Tuesday, 23 December 2014 16:15:00 UTC+5:30, Jason Zhang wrote:
>>
>> 2 ways to post data using curl:
>>
>> 1. the data here need to be bracketed like `curl -XPUT '' -d '{ "foo":
>> "bar" }'`
>> 2. put the data into file, then `curl -XPUT '' -d @put.json`.
>>
>> I think the second is preferred.
>>
>> On Tue 23.23.14 02:18, Ashutosh Parab wrote:
>> > I am trying to post jdbc river config using CURL -XPUT on Windows Xp. I
>> > want to transfer data from MS SQL database into Elastic Search. However
>> I
>> > am getting some exceptions. My Elastic Search version is 1.0.1
>> >
>> > C:\>curl -XPUT "http://localhost:9200/_river/mybooks/_meta"; -d
>> > {"type":"jdbc","j
>> > dbc":{"driver":"com.microsoft.sqlserver.jdbc.
>> SQLServerDriver","url":"jdbc:sqlser
>> > ver://LOCALHOST\\SQLEXPRESS;databaseName=Report","user":
>> > "user","password":"password","sql":"select * from [Report].[dbo
>> > ].[abc]","index":"pkId","type":"bigint"}}
>> >
>> >
>> >
>> > {"error":"MapperParsingException[failed to parse]; nested:
>> > JsonParseException[Un
>> > recognized token 'jdbc': was expecting ('true', 'false' or 'null')\n at
>> > [Source:
>> >  [B@c7947e; line: 1, column: 12]]; ","status":400}
>> > C:\>
>> >
>> > Can someone please check the syntax of my curl command?
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "elasticsearch" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to elasticsearc...@googlegroups.com.
>> > To view this discussion on the web visit https://groups.google.com/d/
>> msgid/elasticsearch/e6fdea56-ca5e-4905-a68b-042472a77fc1%
>> 40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/6ce878a8-9ee6-47dd-8f6b-177248da2b31%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG2sTq8B404kHiDY%3DmH0z2K1No_3iZeAgaMKz_7suasUg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Queue Size 5000 exceeded error (Elastic Search Optimization)

[joergpra...@gmail.com]

What do you mean by "after that server stops"?

You give very few details but in the general case, you overwhelm your
hardware, and the easiest remedy is to change queries, indexing, or add
nodes.

Jörg

On Tue, Dec 23, 2014 at 12:39 PM, nilesh makwana 
wrote:

> I removed all those lines. After that I use this command ab -n 500 -c 500
> http://localhost/es_curl.php to benchmark elasticsearch. It works till
> 300 concurrent requests. After that server stops. We have 2 indexes. 1 is
> having 1.1 million documents, 2nd is having 10K documents.
>
> What could be possible issues?
>
> -Nilesh Makwana
>
> ᐧ
>
> On Tue, Dec 23, 2014 at 4:28 PM, joergpra...@gmail.com <
> joergpra...@gmail.com> wrote:
>
>> Please remove
>>
>> threadpool.search.queue_size: 5000 threadpool.search.type: fixed
>> threadpool.bulk.queue_size: 5000 threadpool.bulk.type: fixed
>> threadpool.index.queue_size: 5000 threadpool.index.type: fixed
>>
>> These settings way too large and will bog down your system.
>>
>> Jörg
>>
>>
>>
>> On Tue, Dec 23, 2014 at 10:56 AM, nilesh makwana <
>> make.nalesw...@gmail.com> wrote:
>>
>>> Here is elasticsearch.yml file for elasticsearch service. Logs are
>>> showing queue size error as I told earlier. I don't have logs at time I
>>> first time changed configurations. After that I am tweaking configurations
>>> to get optimized service.
>>> ᐧ
>>>
>>> On Mon, Dec 22, 2014 at 6:42 PM, joergpra...@gmail.com <
>>> joergpra...@gmail.com> wrote:
>>>
 Please describe your custom settings, and examine the server logs. You
 may have encountered a server error because you have changed the settings.

 Jörg

 On Mon, Dec 22, 2014 at 12:44 PM, nilesh makwana <
 make.nalesw...@gmail.com> wrote:

> Hello All,
>
> I am nilesh makwana. I  have set up elastic search for my web site
> which is world wide event discovery portal.We use elasticsearch for search
> operations.  I am experiencing issues in elastic search, It stops working
> randomly. Qbox client gives error "all shards failed". I looked up the
> issue and increased thread pull queue size to 5000. Still server stops
> randomly. I test sever performance using
> Android Benchmark. Server can not handle 150 simultaneous request. It
> stops every time I run script using ab. What should I do? I except server
> to at least handle 1000 concurrent requests.
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/93c566aa-02a1-46a5-9bcf-d61c40b2a054%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

  --
 You received this message because you are subscribed to a topic in the
 Google Groups "elasticsearch" group.
 To unsubscribe from this topic, visit
 https://groups.google.com/d/topic/elasticsearch/fERgjfFt9l4/unsubscribe
 .
 To unsubscribe from this group and all its topics, send an email to
 elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/CAKdsXoERF_6wLNEyWuQi-%2Bq7J6FiW09Qu3H6v%3DC1McCc1qFjBQ%40mail.gmail.com
 
 .

 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/CAGDNVTfgaY51j3_uWW0qW9zfARdA8-GexbBJJwtAbm2dmpNarQ%40mail.gmail.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/fERgjfFt9l4/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGWzhoHqLVg09Czzk61uZCmbjUcR7aOJqo

Re: Queue Size 5000 exceeded error (Elastic Search Optimization)

[nilesh makwana]

I removed all those lines. After that I use this command ab -n 500 -c 500
http://localhost/es_curl.php to benchmark elasticsearch. It works till 300
concurrent requests. After that server stops. We have 2 indexes. 1 is
having 1.1 million documents, 2nd is having 10K documents.

What could be possible issues?

-Nilesh Makwana

ᐧ

On Tue, Dec 23, 2014 at 4:28 PM, joergpra...@gmail.com <
joergpra...@gmail.com> wrote:

> Please remove
>
> threadpool.search.queue_size: 5000 threadpool.search.type: fixed
> threadpool.bulk.queue_size: 5000 threadpool.bulk.type: fixed
> threadpool.index.queue_size: 5000 threadpool.index.type: fixed
>
> These settings way too large and will bog down your system.
>
> Jörg
>
>
>
> On Tue, Dec 23, 2014 at 10:56 AM, nilesh makwana  > wrote:
>
>> Here is elasticsearch.yml file for elasticsearch service. Logs are
>> showing queue size error as I told earlier. I don't have logs at time I
>> first time changed configurations. After that I am tweaking configurations
>> to get optimized service.
>> ᐧ
>>
>> On Mon, Dec 22, 2014 at 6:42 PM, joergpra...@gmail.com <
>> joergpra...@gmail.com> wrote:
>>
>>> Please describe your custom settings, and examine the server logs. You
>>> may have encountered a server error because you have changed the settings.
>>>
>>> Jörg
>>>
>>> On Mon, Dec 22, 2014 at 12:44 PM, nilesh makwana <
>>> make.nalesw...@gmail.com> wrote:
>>>
 Hello All,

 I am nilesh makwana. I  have set up elastic search for my web site
 which is world wide event discovery portal.We use elasticsearch for search
 operations.  I am experiencing issues in elastic search, It stops working
 randomly. Qbox client gives error "all shards failed". I looked up the
 issue and increased thread pull queue size to 5000. Still server stops
 randomly. I test sever performance using
 Android Benchmark. Server can not handle 150 simultaneous request. It
 stops every time I run script using ab. What should I do? I except server
 to at least handle 1000 concurrent requests.

 --
 You received this message because you are subscribed to the Google
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/93c566aa-02a1-46a5-9bcf-d61c40b2a054%40googlegroups.com
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "elasticsearch" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/elasticsearch/fERgjfFt9l4/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoERF_6wLNEyWuQi-%2Bq7J6FiW09Qu3H6v%3DC1McCc1qFjBQ%40mail.gmail.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAGDNVTfgaY51j3_uWW0qW9zfARdA8-GexbBJJwtAbm2dmpNarQ%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/fERgjfFt9l4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGWzhoHqLVg09Czzk61uZCmbjUcR7aOJqoFw1mKEc2OkA%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving ema

Re: Exception with posting jdbc river config

[Ashutosh Parab]

I have created a put.json file with the following contents:-

{
"type":"jdbc",
"jdbc":
{"driver":"com.microsoft.sqlserver.jdbc.SQLServerDriver",
  

"url":"jdbc:sqlserver://LOCALHOST\\SQLEXPRESS;databaseName=Report

_ME",
  "user":"user",
  "password":"password","sql":"select * from 

[Report.[dbo].[abc]"
}

}

I try to post config using the command  *curl -XPUT 
"http://localhost:9200/_river/mybooks/_meta"; -d put.json.*

Got the below error:-
C:\>curl -XPUT "http://localhost:9200/_river/mybooks/_meta"; -d put.json
{"error":"MapperParsingException[failed to parse]; nested: 
ElasticsearchParseExc
eption[Failed to derive xcontent from (offset=0, length=8): [112, 117, 116, 
46,
106, 115, 111, 110]]; ","status":400}

Please advice


On Tuesday, 23 December 2014 16:15:00 UTC+5:30, Jason Zhang wrote:
>
> 2 ways to post data using curl: 
>
> 1. the data here need to be bracketed like `curl -XPUT '' -d '{ "foo": 
> "bar" }'` 
> 2. put the data into file, then `curl -XPUT '' -d @put.json`. 
>
> I think the second is preferred. 
>
> On Tue 23.23.14 02:18, Ashutosh Parab wrote: 
> > I am trying to post jdbc river config using CURL -XPUT on Windows Xp. I 
> > want to transfer data from MS SQL database into Elastic Search. However 
> I 
> > am getting some exceptions. My Elastic Search version is 1.0.1 
> > 
> > C:\>curl -XPUT "http://localhost:9200/_river/mybooks/_meta"; -d 
> > {"type":"jdbc","j 
> > 
> dbc":{"driver":"com.microsoft.sqlserver.jdbc.SQLServerDriver","url":"jdbc:sqlser
>  
>
> > ver://LOCALHOST\\SQLEXPRESS;databaseName=Report","user": 
> > "user","password":"password","sql":"select * from [Report].[dbo 
> > ].[abc]","index":"pkId","type":"bigint"}} 
> > 
> > 
> > 
> > {"error":"MapperParsingException[failed to parse]; nested: 
> > JsonParseException[Un 
> > recognized token 'jdbc': was expecting ('true', 'false' or 'null')\n at 
> > [Source: 
> >  [B@c7947e; line: 1, column: 12]]; ","status":400} 
> > C:\> 
> > 
> > Can someone please check the syntax of my curl command? 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "elasticsearch" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to elasticsearc...@googlegroups.com . 
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/e6fdea56-ca5e-4905-a68b-042472a77fc1%40googlegroups.com.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>
>
> -- 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6ce878a8-9ee6-47dd-8f6b-177248da2b31%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Getting error while starting server

[Vijayakumari B N]

Hi,

I used only one jar for elasticsearch i.e. elasticsearch-1.4.1.jar, i don't 
think we need any other jars apart from this and my elastic server is also 
running. Please let me know if there are any additional jars which i need 
to add.

Thanks,
Vijaya

On Tuesday, December 23, 2014 2:38:43 PM UTC+5:30, vineeth mohan wrote:
>
> Hi , 
>
> Make sure all the dependent JAR files are present. 
> Its missing some of these.
>
> Thanks
> Vineeth
>
> On Tue, Dec 23, 2014 at 11:15 AM, Vijayakumari B N  > wrote:
>
>> Hi,
>>
>> I get the below error when i tried to restart my jboss server after 
>> integrating with elastic server. I am using elastic search 1.4 version. Can 
>> some one let me know where i am doing wrong. Elastic server absolutely work 
>> fine apart from below error.
>>
>> public static Client getESClient() {
>> if (esClient == null) {
>>  Client esClient = new TransportClient().addTransportAddress(new 
>> InetSocketTransportAddress("127.0.0.1", 9300));   
>> return esClient;
>> }
>> return esClient;
>> }
>>
>>
>>
>>
>> 2014-12-23 10:44:46,933 INFO 
>>  [org.apache.catalina.loader.WebappClassLoader] Illegal access: this web 
>> application instance has been stopped already.  Could not load 
>> org.elasticsearch.transport.SendRequestTransportException.  The eventual 
>> following stack trace is caused by an error thrown for debugging purposes 
>> as well as to attempt to terminate the thread which caused the illegal 
>> access, and has no functional impact.
>> java.lang.IllegalStateException
>> at 
>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1244)
>> at 
>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204)
>> at 
>> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:213)
>> at 
>> org.elasticsearch.transport.TransportService.submitRequest(TransportService.java:177)
>> at 
>> org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:349)
>> at 
>> org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:288)
>> at 
>> org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:321)
>> at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2014-12-23 10:44:46,935 INFO  [org.elasticsearch.client.transport] 
>> [Hammerhead] failed to get node info for 
>> [#transport#-1][BLRD17730][inet[/127.0.0.1:9300]], disconnecting...
>> java.lang.NoClassDefFoundError: 
>> org/elasticsearch/transport/SendRequestTransportException
>> at 
>> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:213)
>> at 
>> org.elasticsearch.transport.TransportService.submitRequest(TransportService.java:177)
>> at 
>> org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:349)
>> at 
>> org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:288)
>> at 
>> org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:321)
>> at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.ClassNotFoundException: 
>> org.elasticsearch.transport.SendRequestTransportException
>> at 
>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1358)
>> at 
>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204)
>> ... 8 more
>> 2014-12-23 10:44:46,943 INFO 
>>  [org.apache.catalina.loader.WebappClassLoader] Illegal access: this web 
>> application instance has been stopped already.  Could not load 
>> org.elasticsearch.common.netty.channel.Channels$4.  The eventual following 
>> stack trace is caused by an error thrown for debugging purposes as well as 
>> to attempt to terminate the thread which caused the illegal access, and has 
>> no functional impact.
>> java.lang.IllegalStateException
>> at 
>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1244)
>> at 
>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204)
>> at 
>> org.elasticsearch.common.netty.channel.Channels.fireChannelDisconnectedLater(Channels.java:383)
>> at 
>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:362)
>> at 
>> org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58)
>> at 
>> org.

Index Structure for Nested Data

[Sameer Soni]

Hi Guys,
This question may have been asked before, but I couldn't find relevant 
answer so I am asking here.

I have a collection of Books, and each Book has contents composed in 
Chapters & Non-Chapters.
I want to index these Books with contents and identify the content type, ie 
is it Chapter & Non-Chapter.
Finally, for each query, I want to filter search results based on number of 
matching books, and each individual result should contain matching Chapter 
& Non-Chapter only.
For example, always 10 matching books are returned, and each result has 
collection of matched Chapter & Non-Chapters.
I identify each chapter and non-chapter by specific ids and they belong to 
a book which also has unique id.

Can somebody guide me as to how should I index my data, and construct 
query, so that most relevant results are returned.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/dd1434af-1018-43f3-8b20-a2813adae16a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ES Index Data for Expected results

[Sameer Soni]

Hi Guys,
This question may have been asked before, but I couldn't find relevant 
answer so I am asking here.

I have a collection of Books, and each Book has contents composed in 
Chapters & Non-Chapters.
I want to index these Books with contents and identify the content type, ie 
is it Chapter & Non-Chapter, and return collections of all matching 
contents for every book.
Finally, I want to filter search results based on number of matching books, 
and each individual result should contain matching Chapter & Non-Chapter 
only, Other non matching types should be ignored.

Can somebody guide me as to how should I index my data, and construct 
query, so that most relevant results are returned.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/be8a187a-00d1-4d8d-ad22-1c1722ea87c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Queue Size 5000 exceeded error (Elastic Search Optimization)

[joergpra...@gmail.com]

Please remove

threadpool.search.queue_size: 5000 threadpool.search.type: fixed
threadpool.bulk.queue_size: 5000 threadpool.bulk.type: fixed
threadpool.index.queue_size: 5000 threadpool.index.type: fixed

These settings way too large and will bog down your system.

Jörg



On Tue, Dec 23, 2014 at 10:56 AM, nilesh makwana 
wrote:

> Here is elasticsearch.yml file for elasticsearch service. Logs are showing
> queue size error as I told earlier. I don't have logs at time I first time
> changed configurations. After that I am tweaking configurations to get
> optimized service.
> ᐧ
>
> On Mon, Dec 22, 2014 at 6:42 PM, joergpra...@gmail.com <
> joergpra...@gmail.com> wrote:
>
>> Please describe your custom settings, and examine the server logs. You
>> may have encountered a server error because you have changed the settings.
>>
>> Jörg
>>
>> On Mon, Dec 22, 2014 at 12:44 PM, nilesh makwana <
>> make.nalesw...@gmail.com> wrote:
>>
>>> Hello All,
>>>
>>> I am nilesh makwana. I  have set up elastic search for my web site which
>>> is world wide event discovery portal.We use elasticsearch for search
>>> operations.  I am experiencing issues in elastic search, It stops working
>>> randomly. Qbox client gives error "all shards failed". I looked up the
>>> issue and increased thread pull queue size to 5000. Still server stops
>>> randomly. I test sever performance using
>>> Android Benchmark. Server can not handle 150 simultaneous request. It
>>> stops every time I run script using ab. What should I do? I except server
>>> to at least handle 1000 concurrent requests.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/93c566aa-02a1-46a5-9bcf-d61c40b2a054%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/fERgjfFt9l4/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoERF_6wLNEyWuQi-%2Bq7J6FiW09Qu3H6v%3DC1McCc1qFjBQ%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAGDNVTfgaY51j3_uWW0qW9zfARdA8-GexbBJJwtAbm2dmpNarQ%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGWzhoHqLVg09Czzk61uZCmbjUcR7aOJqoFw1mKEc2OkA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Exception with posting jdbc river config

[Jason Zhang]

2 ways to post data using curl:

1. the data here need to be bracketed like `curl -XPUT '' -d '{ "foo": "bar" }'`
2. put the data into file, then `curl -XPUT '' -d @put.json`.

I think the second is preferred.

On Tue 23.23.14 02:18, Ashutosh Parab wrote:
> I am trying to post jdbc river config using CURL -XPUT on Windows Xp. I 
> want to transfer data from MS SQL database into Elastic Search. However I 
> am getting some exceptions. My Elastic Search version is 1.0.1
> 
> C:\>curl -XPUT "http://localhost:9200/_river/mybooks/_meta"; -d 
> {"type":"jdbc","j
> dbc":{"driver":"com.microsoft.sqlserver.jdbc.SQLServerDriver","url":"jdbc:sqlser
> ver://LOCALHOST\\SQLEXPRESS;databaseName=Report","user":
> "user","password":"password","sql":"select * from [Report].[dbo
> ].[abc]","index":"pkId","type":"bigint"}}
> 
> 
> 
> {"error":"MapperParsingException[failed to parse]; nested: 
> JsonParseException[Un
> recognized token 'jdbc': was expecting ('true', 'false' or 'null')\n at 
> [Source:
>  [B@c7947e; line: 1, column: 12]]; ","status":400}
> C:\>
> 
> Can someone please check the syntax of my curl command?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/e6fdea56-ca5e-4905-a68b-042472a77fc1%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.


-- 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/20141223104447.GA24617%40Lain.routerbf573c.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch Spark EsHadoopNoNodesLeftException in cluster Mode

[Rahul Kumar]

Hi,

I am trying to index data trough apache spark to elasticsearch. Apache 
Spark is used for data enrichment and then i am using *saveToEs*  to index 
data.

Local mode my code is working fine, but when I run code on cluster mode 
with 1 Master and 2 slave it indexes around 1700 data and showing following 
error  

org.apache.spark.SparkException: Job aborted due to stage failure: Task 9 
> in stage 1711.0 failed 4 times, most recent failure: Lost task 9.3 in stage 
> 1711.0 (TID 27379, ip-1-169-15-116.ec2.internal): 
> org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection 
> error (check network and/or proxy settings)- all nodes failed; tried 
> [[4.212.11.16:9200]]

 

my configurations are following

sparkConf.set("es.index.auto.create", "true")
sparkConf.set("es.nodes","1.1.1.1:9200") //my elasticserver ip 
sparkConf.set("spark.eventLog.enabled","true")
sparkConf.set("es.nodes.discovery", "false") 



*finalData.foreach(row=>{*
*  sc.makeRDD(Seq(row)).saveToEs("spark71/docs")*
* println(row)*
*})*






-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3ec58ae2-8578-4f8d-8122-960812b6811a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Exception with posting jdbc river config

[Ashutosh Parab]

I am trying to post jdbc river config using CURL -XPUT on Windows Xp. I 
want to transfer data from MS SQL database into Elastic Search. However I 
am getting some exceptions. My Elastic Search version is 1.0.1

C:\>curl -XPUT "http://localhost:9200/_river/mybooks/_meta"; -d 
{"type":"jdbc","j
dbc":{"driver":"com.microsoft.sqlserver.jdbc.SQLServerDriver","url":"jdbc:sqlser
ver://LOCALHOST\\SQLEXPRESS;databaseName=Report","user":
"user","password":"password","sql":"select * from [Report].[dbo
].[abc]","index":"pkId","type":"bigint"}}



{"error":"MapperParsingException[failed to parse]; nested: 
JsonParseException[Un
recognized token 'jdbc': was expecting ('true', 'false' or 'null')\n at 
[Source:
 [B@c7947e; line: 1, column: 12]]; ","status":400}
C:\>

Can someone please check the syntax of my curl command?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e6fdea56-ca5e-4905-a68b-042472a77fc1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Queue Size 5000 exceeded error (Elastic Search Optimization)

[nilesh makwana]

Here is elasticsearch.yml file for elasticsearch service. Logs are showing
queue size error as I told earlier. I don't have logs at time I first time
changed configurations. After that I am tweaking configurations to get
optimized service.
ᐧ

On Mon, Dec 22, 2014 at 6:42 PM, joergpra...@gmail.com <
joergpra...@gmail.com> wrote:

> Please describe your custom settings, and examine the server logs. You may
> have encountered a server error because you have changed the settings.
>
> Jörg
>
> On Mon, Dec 22, 2014 at 12:44 PM, nilesh makwana  > wrote:
>
>> Hello All,
>>
>> I am nilesh makwana. I  have set up elastic search for my web site which
>> is world wide event discovery portal.We use elasticsearch for search
>> operations.  I am experiencing issues in elastic search, It stops working
>> randomly. Qbox client gives error "all shards failed". I looked up the
>> issue and increased thread pull queue size to 5000. Still server stops
>> randomly. I test sever performance using
>> Android Benchmark. Server can not handle 150 simultaneous request. It
>> stops every time I run script using ab. What should I do? I except server
>> to at least handle 1000 concurrent requests.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/93c566aa-02a1-46a5-9bcf-d61c40b2a054%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/fERgjfFt9l4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoERF_6wLNEyWuQi-%2Bq7J6FiW09Qu3H6v%3DC1McCc1qFjBQ%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGDNVTfgaY51j3_uWW0qW9zfARdA8-GexbBJJwtAbm2dmpNarQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
# Elasticsearch Configuration Example #

# This file contains an overview of various configuration settings,
# targeted at operations staff. Application developers should
# consult the guide at .
#
# The installation procedure is covered at
# 
.
#
# Elasticsearch comes with reasonable defaults for most settings,
# so you can try it out without bothering with configuration.
#
# Most of the time, these defaults are just fine for running a production
# cluster. If you're fine-tuning your cluster, or wondering about the
# effect of certain configuration option, please _do ask_ on the
# mailing list or IRC channel [http://elasticsearch.org/community].

# Any element in the configuration can be replaced with environment variables
# by placing them in ${...} notation. For example:
#
#node.rack: ${RACK_ENV_VAR}

# For information on supported formats and syntax for the config file, see
# 



### Cluster ###

# Cluster name identifies your cluster for auto-discovery. If you're running
# multiple clusters on the same network, make sure you're using unique names.
#
cluster.name: ae_es_cluster


 Node #

# Node names are generated dynamically on startup, so you're relieved
# from configuring them manually. You can tie this node to a specific name:
#
node.name: "ae_banana"

# Every node can be configured to allow or deny being eligible as the master,
# and to allow or deny to store the data.
#
# Allow this node to be eligible as a master node (enabled by default):
#
#node.master: true
#
# Allow this node to store data (enabled by default):
#
#node.data: true

# You can exploit these settings to design advanced 

Re: failed to parse in kibana panel

[Kenford Zou]

>
> Here is the error message:
>

 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f295fb6b-7203-4094-886f-fb6f511d3f80%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

failed to parse in kibana panel

[Kenford Zou]

I get failure in panel, and cannot get any clue to fix. Can anyone help me?

*This query is from one panel that works:*
curl -XGET 'http://test.com:9200/logstash-2014.12.23/_search?pretty' -d '{
  "facets": {
"0": {
  "date_histogram": {
"field": "@timestamp",
"interval": "30s"
  },
  "global": true,
  "facet_filter": {
"fquery": {
  "query": {
"filtered": {
  "query": {
"query_string": {
  "query": "@type:ihsaccess1"
}
  },
  "filter": {
"bool": {
  "must": [
{
  "range": {
"@timestamp": {
  "from": 1419321262506,
  "to": 1419324862506
}
  }
},
{
  "fquery": {
"query": {
  "query_string": {
"query": "@type:(\"ihsaccess1\")"
  }
},
"_cache": true
  }
}
  ],
  "must_not": [
{
  "fquery": {
"query": {
  "query_string": {
"query": "@fields.response:(200)"
  }
},
"_cache": true
  }
}
  ]
}
  }
}
  }
}
  }
}
  },
  "size": 0
}'

*And this one doesn't work:*
curl -XGET 'http://test.com:9200/logstash-2014.12.23/_search?pretty' -d '{
  "facets": {
"0": {
  "date_histogram": {
"key_field": "@timestamp",
"value_field": "@fields.served-time",
"interval": "30s"
  },
  "global": true,
  "facet_filter": {
"fquery": {
  "query": {
"filtered": {
  "query": {
"query_string": {
  "query": "@type:ihsaccess1"
}
  },
  "filter": {
"bool": {
  "must": [
{
  "range": {
"@timestamp": {
  "from": 1419321262510,
  "to": 1419324862510
}
  }
},
{
  "fquery": {
"query": {
  "query_string": {
"query": "@type:(\"ihsaccess1\")"
  }
},
"_cache": true
  }
}
  ],
  "must_not": [
{
  "fquery": {
"query": {
  "query_string": {
"query": "@fields.response:(200)"
  }
},
"_cache": true
  }
}
  ]
}
  }
}
  }
}
  }
}
  },
  "size": 0
}'


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e89ddd66-b23d-4ce3-a2d8-be4aec228008%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Plugin broadcasting/scalability

[AndrewK]

A while ago I used a plugin on a project to control the migration to, and 
the maintenance of, a secondary data source/index in ElasticSearch. It 
worked really well and avoided lots of messy dependency management in the 
host application, as all the logic was "hidden" behind an ES rest end-point.

But I was never really sure how scalable this was. I have two questions:

1.) when I register a plugin, is that available for all nodes? (I'm 
assuming "yes", which would mean controlling parallel/overlapping call outs 
is important)

2.) when i call my end-point, I have a Client Object passed in the 
constructor: when I debug this, it is an instance of NodeClient which 
presumably means I am working on a single node. Is it possible to construct 
a TransportClient from this, so that I can address more than node and take 
advantage of e.g. bulk imports in parallel?

Regarding 2.), I've had a look at the JDBC River Code and the Feeder mode 
(addressing the cluster from a component running in a separate JVM) seems 
to be there precisely because of this drawback. The River mode seems to 
work off one node like my plugin did/does.

Is my understanding correct?

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/16664d7b-3934-4a90-8b1f-209ede3ff08c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Getting error while starting server

[vineeth mohan]

Hi ,

Make sure all the dependent JAR files are present.
Its missing some of these.

Thanks
Vineeth

On Tue, Dec 23, 2014 at 11:15 AM, Vijayakumari B N <
vijayakumari...@gmail.com> wrote:

> Hi,
>
> I get the below error when i tried to restart my jboss server after
> integrating with elastic server. I am using elastic search 1.4 version. Can
> some one let me know where i am doing wrong. Elastic server absolutely work
> fine apart from below error.
>
> public static Client getESClient() {
> if (esClient == null) {
>  Client esClient = new TransportClient().addTransportAddress(new
> InetSocketTransportAddress("127.0.0.1", 9300));
> return esClient;
> }
> return esClient;
> }
>
>
>
>
> 2014-12-23 10:44:46,933 INFO
>  [org.apache.catalina.loader.WebappClassLoader] Illegal access: this web
> application instance has been stopped already.  Could not load
> org.elasticsearch.transport.SendRequestTransportException.  The eventual
> following stack trace is caused by an error thrown for debugging purposes
> as well as to attempt to terminate the thread which caused the illegal
> access, and has no functional impact.
> java.lang.IllegalStateException
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1244)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204)
> at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:213)
> at
> org.elasticsearch.transport.TransportService.submitRequest(TransportService.java:177)
> at
> org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:349)
> at
> org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:288)
> at
> org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:321)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2014-12-23 10:44:46,935 INFO  [org.elasticsearch.client.transport]
> [Hammerhead] failed to get node info for
> [#transport#-1][BLRD17730][inet[/127.0.0.1:9300]], disconnecting...
> java.lang.NoClassDefFoundError:
> org/elasticsearch/transport/SendRequestTransportException
> at
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:213)
> at
> org.elasticsearch.transport.TransportService.submitRequest(TransportService.java:177)
> at
> org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:349)
> at
> org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:288)
> at
> org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:321)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException:
> org.elasticsearch.transport.SendRequestTransportException
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1358)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204)
> ... 8 more
> 2014-12-23 10:44:46,943 INFO
>  [org.apache.catalina.loader.WebappClassLoader] Illegal access: this web
> application instance has been stopped already.  Could not load
> org.elasticsearch.common.netty.channel.Channels$4.  The eventual following
> stack trace is caused by an error thrown for debugging purposes as well as
> to attempt to terminate the thread which caused the illegal access, and has
> no functional impact.
> java.lang.IllegalStateException
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1244)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204)
> at
> org.elasticsearch.common.netty.channel.Channels.fireChannelDisconnectedLater(Channels.java:383)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:362)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58)
> at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
> at org.elasticsearch.common.netty.channel.Channels.close(Channels.java:812)
> at
> org.elasticsearch.common.netty.channel.AbstractChannel.close(AbstractChannel.java:197)
> at
> org.elasticsearch.transport.netty.NettyTransport$NodeChannels.closeChannelsAndWait(NettyTransport.java:1107)
> at
> org.elasticsearch.transport.netty.NettyTransport$NodeChannels.close