Re: Sort Order when relevance is equal

[Clinton Gormley]

Hi Erich

On 14 May 2014 02:49, Erich Lin  wrote:

> 1) will they always be in the same order if we set the preference
> parameter to an arbitrary string like the user’s session ID.
>

They will be, until a merge happens (eg from indexing, updating, deleting,
or just because...)


> 2) If so, is there a way to predict this deterministic order? Is it done
> by ID of the documents as a tiebreaker etc?
>

It's the order of docs in the Lucene segment


> 3) If not, could we specify that or do we have to do a secondary sort on
> ID if we wanted to do that?
>

Yes, you could provide a secondary sort on _uid (not _id)

> Also, is it possible that the default order for an equal relevance score
changed between 1.0 and 1.2 ? We've noticed seemingly random ordering in
our regression when updating.

It shouldn't have - if you can reproduce this, please open an issue.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPt3XKTVcsuHG%3DPup7iPtZsFuX9Pp50SG-Ewor5Ns2WVY2PziQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sort Order when relevance is equal

[Mateusz Kaczynski]

Seconding question.

Also, is it possible that the default order for an equal relevance score 
changed between 1.0 and 1.2 ? We've noticed seemingly random ordering in 
our regression when updating.  

On Wednesday, 14 May 2014 00:49:55 UTC, Erich Lin wrote:
>
> Ignoring the bouncing results problem with multiple shards , is the order 
> of results deterministic when sorting by relevance score or any other 
> field.  
>
> What I mean by this is if two documents have the same score, 
>
> 1) will they always be in the same order if we set the preference 
> parameter to an arbitrary string like the user’s session ID. 
> 2) If so, is there a way to predict this deterministic order? Is it done 
> by ID of the documents as a tiebreaker etc? 
> 3) If not, could we specify that or do we have to do a secondary sort on 
> ID if we wanted to do that?
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/dd34df9d-7ef0-4fec-9f50-2a7e30cfae8e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to remove a cluster setting?

[David Pilato]

Try:

PUT /_cluster/settings 
{ 
   "transient": { 
  "cluster.routing.allocation.exclude._ip": "" 
   } 
} 

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 22 juil. 2014 à 02:50, Jeffrey Zhou  a écrit :

I made the following setting to my Elasticsearch cluster in order to 
decommission some old nodes in the cluster. After removed these old nodes, now 
I need to re-enable the cluster to allocate shards on those '10.0.6.*' nodes. 
Does anyone know how to remove this setting? 

PUT /_cluster/settings 
{ 
   "transient": { 
  "cluster.routing.allocation.exclude._ip": "10.0.6.*" 
   } 
} 

Thanks in advance for any help!
-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/53df40a8-a248-4373-b789-e0490e3dab8a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/054923B7-1941-4FA0-B4B7-51A99A85F0B3%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

curl indices.memory.index_buffer_size ??

[Chris Berry]

Hello,

I am trying to set indices.memory.index_buffer_size to 30% using curl to my 
running cluster

And I am not able to make it "stick"
I am doing this: 

$  curl -XPUT http://foo:9200/_cluster/settings -d '{ "persistent" : { 
"indices.memory.index_buffer_size" : "30%" }}'
{"acknowledged":true,"persistent":{},"transient":{}

But when I check settings it is not there?

Any idea what I am doing wrong??
It is probably something obvious. But I don't see it...

Thanks,
-- Chris 

$ curl -XGET http://foo:9200/_cluster/settings?pretty=true
{
  "persistent" : {
"threadpool" : {
  "index" : {
"type" : "cached"
  }
}
  },
  "transient" : {
"cluster" : {
  "routing" : {
"allocation" : {
  "enable" : "all"
}
  }
}
  }
}


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0d6ca29a-3ecf-4050-85c8-f7672c4a964d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Handling node failure in ES cluster

[Otis Gospodnetic]

Lots of things could be the source of problems here.  Maybe you can tune 
the JVM params.  We don't know what you are using or what your GC activity 
looks like.  Can you share GC metrics graphs?  If you don't have any GC 
monitoring, you can use SPM .  Why do you have 5 
shards for all indices?  Some seem small and shouldn't need to be sharded 
so much.  Why do you have 3 replicas and not, say, just 2? (we don't know 
your query rates).

Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/


On Monday, July 21, 2014 12:38:49 PM UTC-4, kmoore.cce wrote:
>
> I have had some issues recently as I've expanded my ES cluster, where a 
> single node failure causes basically all other index/search operations to 
> timeout and fail.
>
> I am currently running elasticsearch v1.2.1 and primarily interface with 
> the indices using the elasticsearch python module.
>
> My cluster is 20 nodes, each an m1.large ec2 instance. I currently have 
> ~18 indices each with 5 shards and 3 replicas. The average size of each 
> index is ~20GB and ~10 million documents (low is ~100K documents (300mb), 
> high ~40 million (35gb)).
> I run each node with ES_MAX_SIZE=4g and ES_MIN_SIZE=512m. There are no 
> other services running on the elasticsearch nodes, except ssh. I use zen 
> unicast discovery with a set list of nodes. I have tried to enable 
> 'bootstrap.mlockall', but the ulimit settings do not seem to be working and 
> I keep getting 'Unable to lock JVM memory (ENOMEM)' errors when starting a 
> node (note: I didn't see this log message when running 0.90.7).
>
> I have a fairly constant series of new or updated documents (I don't 
> actually update, but rather reindex when a new document with the same id is 
> found) that are being ingested all the time, and a number of users who are 
> querying the data on a regular basis - most queries are set queries through 
> the python API.
>
> The issue I have now is that while data is being ingested/indexed, I will 
> hit Java heap out of memory errors. I think this is related to garbage 
> collection as that seems to be the last activity in the logs nearly 
> everytime this occurs. I have tried adjusting the heap max to 6g, and that 
> seems to help but I am not sure it solves the issue. In conjunction with 
> that, when the out of memory error occurs it seems to cause the other nodes 
> to stop working effectively, timeout errors in both indexing and searching.
>
> My question is: what is the best way to support a node failing for this 
> reason? I would obviously like to solve the underlying problem as well, but 
> I would also like to be able to support a node crashing for some reason 
> (whether it be because of me or because ec2 took it away). Shouldn't the 
> failover in replicas support the missing node? I understand the cluster 
> state would be yellow at this time, but I should be able to index and 
> search data on the remaining nodes, correct?
>
> Are there configuration changes I can make to better support the cluster 
> and identify or solve the underyling issue? 
>
> Any help is appreciated. I understand I have a lot to learn about 
> Elasticsearch, but I am hoping I can add some stability/resiliency to my 
> cluster.
>
> Thanks in advance,
> -Kevin
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9eb495e8-9ac6-4ef0-95ae-a6cc4516c67a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Common website analytics aggregation formulas

[Anki Reddy]

Unique visitors 

http://www.elasticsearch.org/blog/count-elasticsearch/ 



On Saturday, May 3, 2014 5:17:13 AM UTC+5:30, Demetrius Nunes wrote:
>
> Hi guys,
>
> At my company we're building a platform product which has a big analytics 
> component to it.
>
> I am intending to use elasticsearch to power that part of the platform.
>
> Most of the analytics examples that I see using elasticsearch aggregations 
> are around systems logs & monitoring.
>
> There quite a few metrics that I have to provide reporting that are very 
> typical of website analytics, such as time spent on site, bounce rate, 
> active users, etc.
>
> I've already implemented all the tracking code within the system and I 
> have indexes with timestamps, user-generated events such as page hits, 
> clicks, and so on.
>
> So, are there any good references, best practices, plugins or even 
> formulas on how to implement these kinds of website analytics metrics using 
> elasticsearch?
>
> Thanks a lot,
> Demetrius
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8e22b230-9929-43fe-842f-10a548cd25ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sorting a nested array

[vineeth mohan]

Hello Darin ,

This is the wrong approach to do this.
You can only sort per document and not per nested document and recieve
output as documents and not as nested documents.
So i would rather go for parent child approach.

That being said , what you asked is actually possible with little tweaking
on client side. -
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#_using_function_score


So a function score query can has a score_mode of min or max.
This means that if we give the field reward as the score value and
score_mode , it will take the highest  value per doc as the score.
With this , you can get the top 10 documents bearing top 10 rating.
Once you receive this you need to take out all ratings and do a sort again
and then find the top 10 ratings.
Possibility is that a single document might contain top 2 good reviews.

With this i believe you might need to turn include_in_parent on.

Thanks
   Vineeth


On Tue, Jul 22, 2014 at 8:15 AM, Darin Amos 
wrote:

> Hi Everyone,
>
> I am looking for a little guidance on how to setup my indexes to support
> sorting a nested array. To give a simple example, we have an index of
> products in our index, below is a dummy example document:
>
> /products/product/
>
> {
>  "name":"Toaster",
>  "desc":"For making toast",
>  "price":29.99,
>  "reviews":[
>   {
>"rating" : 5,
>"comment" : "works great!"
>   },
>   {
>"rating" : 1,
>"comment" : "Ugh!!"
>   },
>   {
>"rating" : 3,
>"comment" : "Meh"
>   }
>  ]
> }
>
>
> I would like to be able to query for product documents, but be able to
> sort the reviews array based on the rating. A customer could ask to see the
> first 10 best reviews, or the worst 5 reviews for example. Is this
> possible? Is this possible if the reviews are in a separate index and I
> execute a join query?
>
> Any information or guidance would be extremely helpful!
>
> Thanks!
>
> Darin
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/500b16ff-b00f-44a9-a238-9c7a09432bfa%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DxMYAQfaZAzx7JyuHvAuBb4%3DZ9XBPyMkp_1G9uWEOdvA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Sorting a nested array

[Darin Amos]

Hi Everyone,

I am looking for a little guidance on how to setup my indexes to support 
sorting a nested array. To give a simple example, we have an index of 
products in our index, below is a dummy example document:

/products/product/

{
 "name":"Toaster",
 "desc":"For making toast",
 "price":29.99,
 "reviews":[
  {
   "rating" : 5,
   "comment" : "works great!"
  },  
  {
   "rating" : 1,
   "comment" : "Ugh!!"
  },  
  {
   "rating" : 3,
   "comment" : "Meh"
  }
 ]
}


I would like to be able to query for product documents, but be able to sort 
the reviews array based on the rating. A customer could ask to see the 
first 10 best reviews, or the worst 5 reviews for example. Is this 
possible? Is this possible if the reviews are in a separate index and I 
execute a join query?

Any information or guidance would be extremely helpful!

Thanks!

Darin

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/500b16ff-b00f-44a9-a238-9c7a09432bfa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How to remove a cluster setting?

[Jeffrey Zhou]

I made the following setting to my Elasticsearch cluster in order to 
decommission some old nodes in the cluster. After removed these old nodes, 
now I need to re-enable the cluster to allocate shards on those '10.0.6.*' 
nodes. Does anyone know how to remove this setting? 

PUT /_cluster/settings 
{ 
   "transient": { 
  "cluster.routing.allocation.exclude._ip": "10.0.6.*" 
   } 
} 

Thanks in advance for any help! 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/53df40a8-a248-4373-b789-e0490e3dab8a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Handling node failure in ES cluster

[Mark Walkom]

Max and min memory should be the same, mlockall is probably not working due
to these being different as it can't lock a sliding window.
Try setting that and see if it helps.

Also you didn't mention your java version and release, which would be
helpful.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 22 July 2014 02:38, kmoore.cce  wrote:

> I have had some issues recently as I've expanded my ES cluster, where a
> single node failure causes basically all other index/search operations to
> timeout and fail.
>
> I am currently running elasticsearch v1.2.1 and primarily interface with
> the indices using the elasticsearch python module.
>
> My cluster is 20 nodes, each an m1.large ec2 instance. I currently have
> ~18 indices each with 5 shards and 3 replicas. The average size of each
> index is ~20GB and ~10 million documents (low is ~100K documents (300mb),
> high ~40 million (35gb)).
> I run each node with ES_MAX_SIZE=4g and ES_MIN_SIZE=512m. There are no
> other services running on the elasticsearch nodes, except ssh. I use zen
> unicast discovery with a set list of nodes. I have tried to enable
> 'bootstrap.mlockall', but the ulimit settings do not seem to be working and
> I keep getting 'Unable to lock JVM memory (ENOMEM)' errors when starting a
> node (note: I didn't see this log message when running 0.90.7).
>
> I have a fairly constant series of new or updated documents (I don't
> actually update, but rather reindex when a new document with the same id is
> found) that are being ingested all the time, and a number of users who are
> querying the data on a regular basis - most queries are set queries through
> the python API.
>
> The issue I have now is that while data is being ingested/indexed, I will
> hit Java heap out of memory errors. I think this is related to garbage
> collection as that seems to be the last activity in the logs nearly
> everytime this occurs. I have tried adjusting the heap max to 6g, and that
> seems to help but I am not sure it solves the issue. In conjunction with
> that, when the out of memory error occurs it seems to cause the other nodes
> to stop working effectively, timeout errors in both indexing and searching.
>
> My question is: what is the best way to support a node failing for this
> reason? I would obviously like to solve the underlying problem as well, but
> I would also like to be able to support a node crashing for some reason
> (whether it be because of me or because ec2 took it away). Shouldn't the
> failover in replicas support the missing node? I understand the cluster
> state would be yellow at this time, but I should be able to index and
> search data on the remaining nodes, correct?
>
> Are there configuration changes I can make to better support the cluster
> and identify or solve the underyling issue?
>
> Any help is appreciated. I understand I have a lot to learn about
> Elasticsearch, but I am hoping I can add some stability/resiliency to my
> cluster.
>
> Thanks in advance,
> -Kevin
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/74ac48ec-0c05-4683-9c78-66d8c97687fa%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624a1bjybx6b-B-7h%2BkVVy-JPEEvs0_9JaY-wbcLS5hPFhw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

explanation of each fields of _stats api data

[Jinyuan Zhou]

Hi there,
How can find documentations for explanation of json data returned by _stats 
for index. I mean meaning of each fields not just high level description. 
Specifically, For the following  data

"primaries": {"docs": {"count": 1789457,"deleted": 0},"store": {"
size_in_bytes": 2085533463,"throttle_time_in_millis": 582538},"indexing": {"
index_total": 297925,"index_time_in_millis": 113345,"index_current": 0,"
delete_total": 0,"delete_time_in_millis": 0,"delete_current": 0},

I can't find out meaning of index_total field. and index_current field. for 
this index the document count is 1789457 which is about 3 times  of the 
index_total value. 

Thanks,
Jack

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2f5ef5fa-fe47-4907-a7d0-b3f7212e2fbe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

multiple indices per document

[Prakash]

I want to use ES to index logs coming from different processes. Assume I 
have 2 sources: ProcessA and ProcessB Logs from the processes are formatted 
in json. Example log:

{"level":"DEBUG","logger":"REPOSITORY","timestamp":1405982400689,"attrs":{"profile":"ManagementServerA","organization":"FOOBAR"},"thread":"main","message":"Repository.store()
 : Stored successfully in /central/zone/cef9cccab964"}

How can I get ES to update multiple indexes when it sees a new document ? 
In this case I want indices on the profile and organization values. Do I 
have to

   1. Create indexes using the ES REST api before ES sees any logs.
   2. Supply an _index field to each json document
   3. Have multiple values in the _index field to indicate what indexes 
   must be updated ? i.e should I have: *"_index": {"ManagementServerA" , 
   "FOOBAR"}*

Please let me know if this is the correct way to do this.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c299f3e4-eebc-43a4-ab23-894605b2a752%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch recovering process took a long time.

[Ivan Brusic]

Recovery is throttled since version 0.90.1

http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/index-modules-store.html#store-throttling

Increase indices.store.throttle.max_bytes_per_sec to a level that is
suitable for your environment. Since IO should be the main bottleneck, the
setting could vary greatly depending on SSD, platter disk or shared storage.

-- 
Ivan


On Mon, Jul 21, 2014 at 1:28 PM, anivlis  wrote:

> Sorry, the setting discovery.zen.minimum_master_nodes is 8
>
>
>
>
> El lunes, 21 de julio de 2014 17:20:03 UTC-3, anivlis escribió:
>
>> Hi,
>> We are using elasticsearch 0.90.1. We had some problems with the network,
>> when the network is down (the convergence time is around 40s). Recovering
>> the elastic after this event took a long time to be available. We have 16
>> data nodes and 16 shards with 2 replica and the settings:
>>
>> discovery.zen.minimum_master_nodes: 4
>> gateway.recover_after_nodes: 4
>> gateway.expected_nodes: 6
>>
>> Any idea about how to minimize the recovery time?
>> Is it a good idea to update the version?
>> What will happen if we increase discovery.zen.fd.ping_interval and
>> ping_timeout settings? Assuming the network is completely off, is there any
>> way to wait for at least those 40 seconds to start marking servers down?
>> Does the ping_timeout ignore the failure to connect to a node?
>>
>>
>> Thanks.
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/7997de22-b421-4f7c-ae7f-e001dbfef618%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQACJs%3Dzuu4gjYTvcKpF%2BSgv_1UkYZw%3DC0G8FkY1meAg-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch recovering process took a long time.

[anivlis]

Sorry, the setting discovery.zen.minimum_master_nodes is 8




El lunes, 21 de julio de 2014 17:20:03 UTC-3, anivlis escribió:
>
> Hi,
> We are using elasticsearch 0.90.1. We had some problems with the network, 
> when the network is down (the convergence time is around 40s). Recovering 
> the elastic after this event took a long time to be available. We have 16 
> data nodes and 16 shards with 2 replica and the settings:
>
> discovery.zen.minimum_master_nodes: 4
> gateway.recover_after_nodes: 4
> gateway.expected_nodes: 6
>
> Any idea about how to minimize the recovery time?
> Is it a good idea to update the version?
> What will happen if we increase discovery.zen.fd.ping_interval and 
> ping_timeout settings? Assuming the network is completely off, is there any 
> way to wait for at least those 40 seconds to start marking servers down? 
> Does the ping_timeout ignore the failure to connect to a node?
>
>
> Thanks.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7997de22-b421-4f7c-ae7f-e001dbfef618%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch recovering process took a long time.

[anivlis]

Hi,
We are using elasticsearch 0.90.1. We had some problems with the network, 
when the network is down (the convergence time is around 40s). Recovering 
the elastic after this event took a long time to be available. We have 16 
data nodes and 16 shards with 2 replica and the settings:

discovery.zen.minimum_master_nodes: 4
gateway.recover_after_nodes: 4
gateway.expected_nodes: 6

Any idea about how to minimize the recovery time?
Is it a good idea to update the version?
What will happen if we increase discovery.zen.fd.ping_interval and 
ping_timeout settings? Assuming the network is completely off, is there any 
way to wait for at least those 40 seconds to start marking servers down? 
Does the ping_timeout ignore the failure to connect to a node?


Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a32a0c42-122a-45a3-9c56-8869b4028608%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Getting _id field in elasticsearch to map to a field in HIVE

[hiveesuser]

Hi,


I am working on a project to integrate hive and elasticsearch and for one 
of our use case we are loading data from ELASTISCSEARCH --> HIVE.


During this process I want to store the _id field which is in elasticsearch 
document in hive. I am able to get the fields which are part of _source 
like messages, @timestamp etc but I am not able to get the _id associated 
with that particular document. 


The following is the sample table I am trying to create 

create external table eshivetable (id string,eventdate timestamp, host 
string, username string, message string) STORED BY 
'org.elasticsearch.hadoop.hive.EsStorageHandler' 
TBLPROPERTIES('es.resource' = 'logstash-*/syslog', 'es.mapping.names' = 
'id:_id,eventdate:@timestamp,host:host,username:username,message:message','es.nodes'='10.10.10.50','es.port'='9200','es.query'='?q=type:syslog');
 
So when I select id it returns a null value...
 
Can some one help me with this please. 
 

 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5d2eb38d-9f0d-4329-ba2b-0d28c06f98e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ERROR][bootstrap] {1.2.2}: Initialization Failed ... - NullPointerException[null]

[Phillip Ulberg]

Turned out to be an issue with the "http-basic" auth plugin I was using.

On Monday, July 21, 2014 2:02:02 PM UTC-5, Phillip Ulberg wrote:
>
> Same issue here, I upgraded to ES 1.2.2, and jdk 1.7.0_60-b19, debug log 
> output - 
>
> [2014-07-21 18:50:05,054][INFO ][node ] [p-esmon01] 
> version[1.2.2], pid[23697], build[9902f08/2014-07-09T12:02:32Z]
> [2014-07-21 18:50:05,054][INFO ][node ] [p-esmon01] 
> initializing ...
> [2014-07-21 18:50:05,056][DEBUG][node ] [p-esmon01] 
> using home [/usr/share/elasticsearch], config [/etc/elasticsearch], data 
> [[/etc/elasticsearch/data]], logs [/var/log/elasticsearch], work 
> [/etc/elasticsearch/work], plugins [/usr/share/elasticsearch/plugins]
> [2014-07-21 18:50:05,064][DEBUG][plugins  ] [p-esmon01] 
> lucene property is not set in plugin es-plugin.properties file. Skipping 
> test.
> [2014-07-21 18:50:05,074][DEBUG][plugins  ] [p-esmon01] 
> skipping 
> [jar:file:/usr/share/elasticsearch/plugins/marvel/marvel-1.2.1.jar!/es-plugin.properties]
> [2014-07-21 18:50:05,075][DEBUG][plugins  ] [p-esmon01] 
> lucene property is not set in plugin es-plugin.properties file. Skipping 
> test.
> [2014-07-21 18:50:05,075][DEBUG][plugins  ] [p-esmon01] 
> [/usr/share/elasticsearch/plugins/http-basic-server-plugin/_site] directory 
> does not exist.
> [2014-07-21 18:50:05,077][DEBUG][plugins  ] [p-esmon01] 
> [/usr/share/elasticsearch/plugins/http-basic/_site] directory does not 
> exist.
> [2014-07-21 18:50:05,078][INFO ][plugins  ] [p-esmon01] 
> loaded [marvel, http-basic-server-plugin], sites [marvel, head]
> [2014-07-21 18:50:05,104][DEBUG][common.compress.lzf  ] using 
> [UnsafeChunkDecoder] decoder
> [2014-07-21 18:50:05,113][DEBUG][env  ] [p-esmon01] 
> using node location [[/etc/elasticsearch/data/p_es_clust_mon01/nodes/0]], 
> local_node_id [0]
> [2014-07-21 18:50:05,907][ERROR][bootstrap] {1.2.2}: 
> Initialization Failed ...
> - ExecutionError[java.lang.NoClassDefFoundError: 
> org/elasticsearch/rest/StringRestResponse]
> NoClassDefFoundError[org/elasticsearch/rest/StringRestResponse]
> 
> ClassNotFoundException[org.elasticsearch.rest.StringRestResponse]
> org.elasticsearch.common.util.concurrent.ExecutionError: 
> java.lang.NoClassDefFoundError: org/elasticsearch/rest/StringRestResponse
> at 
> org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
> at 
> org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3934)
> at 
> org.elasticsearch.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
> at 
> org.elasticsearch.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
> at 
> org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:51)
> at 
> org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:50)
> at 
> org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:50)
> at 
> org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:372)
> at 
> org.elasticsearch.common.inject.BindingProcessor$1$1.run(BindingProcessor.java:148)
> at 
> org.elasticsearch.common.inject.BindingProcessor.initializeBindings(BindingProcessor.java:204)
> at 
> org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:119)
> at 
> org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:102)
> at 
> org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
> at 
> org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
> at 
> org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
> at 
> org.elasticsearch.node.internal.InternalNode.(InternalNode.java:188)
> at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
> at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:70)
> at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:203)
> at 
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
> Caused by: java.lang.NoClassDefFoundError: 
> org/elasticsearch/rest/StringRestResponse
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2532)
> at java.lang.Class.getDeclaredConstructors(Class.java:1901)
> at 
> org.elasticsearch.common.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:177)
> at 
> org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:59)
> at 
> org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(Cons

Re: [ERROR][bootstrap] {1.2.2}: Initialization Failed ... - NullPointerException[null]

[joergpra...@gmail.com]

You have to update your plugins marvel and http-basic-server-plugin to get
them to work with ES 1.2.2

Jörg


On Mon, Jul 21, 2014 at 9:02 PM, Phillip Ulberg 
wrote:

> Same issue here, I upgraded to ES 1.2.2, and jdk 1.7.0_60-b19, debug log
> output -
>
> [2014-07-21 18:50:05,054][INFO ][node ] [p-esmon01]
> version[1.2.2], pid[23697], build[9902f08/2014-07-09T12:02:32Z]
> [2014-07-21 18:50:05,054][INFO ][node ] [p-esmon01]
> initializing ...
> [2014-07-21 18:50:05,056][DEBUG][node ] [p-esmon01]
> using home [/usr/share/elasticsearch], config [/etc/elasticsearch], data
> [[/etc/elasticsearch/data]], logs [/var/log/elasticsearch], work
> [/etc/elasticsearch/work], plugins [/usr/share/elasticsearch/plugins]
> [2014-07-21 18:50:05,064][DEBUG][plugins  ] [p-esmon01]
> lucene property is not set in plugin es-plugin.properties file. Skipping
> test.
> [2014-07-21 18:50:05,074][DEBUG][plugins  ] [p-esmon01]
> skipping
> [jar:file:/usr/share/elasticsearch/plugins/marvel/marvel-1.2.1.jar!/es-plugin.properties]
> [2014-07-21 18:50:05,075][DEBUG][plugins  ] [p-esmon01]
> lucene property is not set in plugin es-plugin.properties file. Skipping
> test.
> [2014-07-21 18:50:05,075][DEBUG][plugins  ] [p-esmon01]
> [/usr/share/elasticsearch/plugins/http-basic-server-plugin/_site] directory
> does not exist.
> [2014-07-21 18:50:05,077][DEBUG][plugins  ] [p-esmon01]
> [/usr/share/elasticsearch/plugins/http-basic/_site] directory does not
> exist.
> [2014-07-21 18:50:05,078][INFO ][plugins  ] [p-esmon01]
> loaded [marvel, http-basic-server-plugin], sites [marvel, head]
> [2014-07-21 18:50:05,104][DEBUG][common.compress.lzf  ] using
> [UnsafeChunkDecoder] decoder
> [2014-07-21 18:50:05,113][DEBUG][env  ] [p-esmon01]
> using node location [[/etc/elasticsearch/data/p_es_clust_mon01/nodes/0]],
> local_node_id [0]
> [2014-07-21 18:50:05,907][ERROR][bootstrap] {1.2.2}:
> Initialization Failed ...
> - ExecutionError[java.lang.NoClassDefFoundError:
> org/elasticsearch/rest/StringRestResponse]
> NoClassDefFoundError[org/elasticsearch/rest/StringRestResponse]
>
> ClassNotFoundException[org.elasticsearch.rest.StringRestResponse]
> org.elasticsearch.common.util.concurrent.ExecutionError:
> java.lang.NoClassDefFoundError: org/elasticsearch/rest/StringRestResponse
> at
> org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
> at
> org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3934)
> at
> org.elasticsearch.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
> at
> org.elasticsearch.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
> at
> org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:51)
> at
> org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:50)
> at
> org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:50)
> at
> org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:372)
> at
> org.elasticsearch.common.inject.BindingProcessor$1$1.run(BindingProcessor.java:148)
> at
> org.elasticsearch.common.inject.BindingProcessor.initializeBindings(BindingProcessor.java:204)
> at
> org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:119)
> at
> org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:102)
> at
> org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
> at
> org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
> at
> org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
> at
> org.elasticsearch.node.internal.InternalNode.(InternalNode.java:188)
> at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
> at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:70)
> at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:203)
> at
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
> Caused by: java.lang.NoClassDefFoundError:
> org/elasticsearch/rest/StringRestResponse
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2532)
> at java.lang.Class.getDeclaredConstructors(Class.java:1901)
> at
> org.elasticsearch.common.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:177)
> at
> org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:59)
> at
> org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:29)
> 

Re: [ANN] Log4j2 Elasticsearch appender

[joergpra...@gmail.com]

The Log4j appender is batching events because it is using bulk indexing. If
you are worried, you can set the bulk action size to an extreme high value
and increase heap to some GB only for buffering log messages, together with
a reasonable flush interval.

Jörg


On Mon, Jul 21, 2014 at 9:14 PM, Ivan Brusic  wrote:

> I was indexing events into Elasticsearch via the standard SocketAppender
> into Logstash, but I stopped doing so since the SocketAppender was not
> releasing threads. Great to see a direct approach, but I like to use
> Logstash in the middle as a buffer in order to batch events.
>
> --
> Ivan
>
>
> On Sat, Jul 19, 2014 at 8:59 AM, Alfredo Serafini 
> wrote:
>
>> I'll try it as soon as I can!
>> thanks,
>> Alfredo
>> :-)
>>
>> Il giorno venerdì 18 luglio 2014 10:08:14 UTC+2, Jörg Prante ha scritto:
>>
>>> Hi,
>>>
>>> I released a Log4j2 Elasticsearch appender
>>>
>>> https://github.com/jprante/log4j2-elasticsearch
>>>
>>> in the hope it is useful.
>>>
>>> Best,
>>>
>>> Jörg
>>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/dce481d9-ac3e-4fd0-aaba-3a4c69d07d34%40googlegroups.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCppMVB-9_0btCXtPTo67c%2BMFziYDff0qwiptj4kn-hyw%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGim0o0LVVFRoLAM2DOHgS-4wM%2BariaLrF3aBg6Vjs2Nw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Can one do a singular/plural phrase match with the Query String Query?

[David Pilato]

I think a stemmer analyzer would be fit your use case: See 
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/choosing-a-stemmer.html#choosing-a-stemmer

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 21 juillet 2014 à 20:53:09, Brian Jones (tbrianjo...@gmail.com) a écrit:

Can one perform the following query using wildcards ( instead of two distinct 
phrases ) when using a Query String Query?
"photographic film" OR "photographic films"

These do not seem to work, and return the same number of results as just 
"photographic film":
"photographic film?"
"photographic film*"

Can wildcards not be placed inside Exact Phrase queries?  Is there a way to 
mimic this?

My goal is to be able to perform queries like this:
"photo* film?"

... capturing:
"photo film"
"photo films"
"photographic films"
"photography films"
etc...
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/66cc151f-a235-40d4-a125-2236aae0f9bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.53cd6b41.19495cff.e03c%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: Synonym filter results in term facet

[vineeth mohan]

Hello Ravi ,

Your approach is wrong.
When you use synonym filter , it indexes all synonyms of that token hence
and synonym will match against that term.
So when you do a facet , you will get an aggregation of all synonyms rather
than just one.

Better approach would be to store the unique name into some other field and
take a facet of that field.

Thanks
   Vineeth


On Mon, Jul 21, 2014 at 11:21 PM,  wrote:

> Hi All,
>
> I have a requirement in which I need to find distinct company names. I was
> using "Keyword" tokenizer for that field and through term facet I was able
> to get distinct company names. However terms facet treated company names
> like "ibm suisse", "ibm corporation", "ibm" as different companies.
> Online documentation suggested me to use "Synonym filter" to solve this.
> My settings is:
>
> curl -XPUT 'http://localhost:9200/dataindex/' -d '{
>   "settings": {
> "index": {
>   "analysis": {
> "analyzer": {
>  "customAnalyzer": {
> "type": "custom",
> "tokenizer": "whitespace",
> "filter": [
>   "lowercase","synonym"
> ]
>   }
> },
> "filter": {
>   "synonym" : {
>   "type" : "synonym",
>   "tokenizer": "keyword",
>   "synonyms_path" : "analysis/synonym.txt"
>   }
> }
>   }
> }
>   }
> }'
>
> My mapping is:
>
> curl -XPUT 'http://localhost:9200/dataindex/tweet/_mapping' -d '
> {
> "tweet" : {
> "properties" : {
> "company": {
>  "type": "string",
>  "analyzer": "customAnalyzer"
> }
> }
> }
> }'
>
> In the synonym.txt file I have : ibm suisse, ibm corporation, ibm
> business, ibm => ibm corp ltd
>
> Indexed data:
> curl -XPUT 'http://localhost:9200/dataindex/tweet/1' -d '{
> "company" : "ibm"
> }'
> curl -XPUT 'http://localhost:9200/dataindex/tweet/2' -d '{
> "company" : "ibm corporation"
> }'
> curl -XPUT 'http://localhost:9200/dataindex/tweet/3' -d '{
> "company" : "ibm suisse"
> }'
> curl -XPUT 'http://localhost:9200/dataindex/tweet/4' -d '{
> "company" : "ibm business"
> }'
>
> If I run a terms facet:
> {
>   "facets": {
> "loc_facet": {
>   "terms": {
> "field": "company"
>   }
> }
>   }
> }
> I get 3 terms ie {term: ibm corp ltd, count: 3} {term: suisse, count: 1}
> {term: corporation, count: 1}
> I want the facet result to return only one term: ibm corp ltd with
> count=3. This way i will get distinct company names and also map synonym
> names into single company name.
> Please correct me if I am using wrong tokenizer or my approach is not
> correct.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/1ba32926-7015-4b8a-89ae-bf43a2561b71%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5ny%3Di76CHwpbEoY-4nGaraQfz-Tmmm5MVJbiA%2B0nrgKZQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Log4j2 Elasticsearch appender

[Ivan Brusic]

I was indexing events into Elasticsearch via the standard SocketAppender
into Logstash, but I stopped doing so since the SocketAppender was not
releasing threads. Great to see a direct approach, but I like to use
Logstash in the middle as a buffer in order to batch events.

-- 
Ivan


On Sat, Jul 19, 2014 at 8:59 AM, Alfredo Serafini  wrote:

> I'll try it as soon as I can!
> thanks,
> Alfredo
> :-)
>
> Il giorno venerdì 18 luglio 2014 10:08:14 UTC+2, Jörg Prante ha scritto:
>
>> Hi,
>>
>> I released a Log4j2 Elasticsearch appender
>>
>> https://github.com/jprante/log4j2-elasticsearch
>>
>> in the hope it is useful.
>>
>> Best,
>>
>> Jörg
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/dce481d9-ac3e-4fd0-aaba-3a4c69d07d34%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCppMVB-9_0btCXtPTo67c%2BMFziYDff0qwiptj4kn-hyw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Setting from Json File

[M_20]

Hi all,

I am trying to read the setting from Json file. Something like:

ImmutableSettings.settingsBuilder().loadFromSource( path );

But it seems "loadFromSource" doesn't work for this purpose.
Could you please point me out the method?

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/44cf6310-4e35-4c30-86ce-69a3660f1257%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ERROR][bootstrap] {1.2.2}: Initialization Failed ... - NullPointerException[null]

[Phillip Ulberg]

Same issue here, I upgraded to ES 1.2.2, and jdk 1.7.0_60-b19, debug log 
output - 

[2014-07-21 18:50:05,054][INFO ][node ] [p-esmon01] 
version[1.2.2], pid[23697], build[9902f08/2014-07-09T12:02:32Z]
[2014-07-21 18:50:05,054][INFO ][node ] [p-esmon01] 
initializing ...
[2014-07-21 18:50:05,056][DEBUG][node ] [p-esmon01] 
using home [/usr/share/elasticsearch], config [/etc/elasticsearch], data 
[[/etc/elasticsearch/data]], logs [/var/log/elasticsearch], work 
[/etc/elasticsearch/work], plugins [/usr/share/elasticsearch/plugins]
[2014-07-21 18:50:05,064][DEBUG][plugins  ] [p-esmon01] 
lucene property is not set in plugin es-plugin.properties file. Skipping 
test.
[2014-07-21 18:50:05,074][DEBUG][plugins  ] [p-esmon01] 
skipping 
[jar:file:/usr/share/elasticsearch/plugins/marvel/marvel-1.2.1.jar!/es-plugin.properties]
[2014-07-21 18:50:05,075][DEBUG][plugins  ] [p-esmon01] 
lucene property is not set in plugin es-plugin.properties file. Skipping 
test.
[2014-07-21 18:50:05,075][DEBUG][plugins  ] [p-esmon01] 
[/usr/share/elasticsearch/plugins/http-basic-server-plugin/_site] directory 
does not exist.
[2014-07-21 18:50:05,077][DEBUG][plugins  ] [p-esmon01] 
[/usr/share/elasticsearch/plugins/http-basic/_site] directory does not 
exist.
[2014-07-21 18:50:05,078][INFO ][plugins  ] [p-esmon01] 
loaded [marvel, http-basic-server-plugin], sites [marvel, head]
[2014-07-21 18:50:05,104][DEBUG][common.compress.lzf  ] using 
[UnsafeChunkDecoder] decoder
[2014-07-21 18:50:05,113][DEBUG][env  ] [p-esmon01] 
using node location [[/etc/elasticsearch/data/p_es_clust_mon01/nodes/0]], 
local_node_id [0]
[2014-07-21 18:50:05,907][ERROR][bootstrap] {1.2.2}: 
Initialization Failed ...
- ExecutionError[java.lang.NoClassDefFoundError: 
org/elasticsearch/rest/StringRestResponse]
NoClassDefFoundError[org/elasticsearch/rest/StringRestResponse]

ClassNotFoundException[org.elasticsearch.rest.StringRestResponse]
org.elasticsearch.common.util.concurrent.ExecutionError: 
java.lang.NoClassDefFoundError: org/elasticsearch/rest/StringRestResponse
at 
org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
at 
org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3934)
at 
org.elasticsearch.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
at 
org.elasticsearch.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
at 
org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:51)
at 
org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:50)
at 
org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:50)
at 
org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:372)
at 
org.elasticsearch.common.inject.BindingProcessor$1$1.run(BindingProcessor.java:148)
at 
org.elasticsearch.common.inject.BindingProcessor.initializeBindings(BindingProcessor.java:204)
at 
org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:119)
at 
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:102)
at 
org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
at 
org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
at 
org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
at 
org.elasticsearch.node.internal.InternalNode.(InternalNode.java:188)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:70)
at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:203)
at 
org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: java.lang.NoClassDefFoundError: 
org/elasticsearch/rest/StringRestResponse
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2532)
at java.lang.Class.getDeclaredConstructors(Class.java:1901)
at 
org.elasticsearch.common.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:177)
at 
org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:59)
at 
org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:29)
at 
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:37)
at 
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:33)
at 
org.elasticsearch.common.inject.internal.FailableCache

Can one do a singular/plural phrase match with the Query String Query?

[Brian Jones]

Can one perform the following query using wildcards ( instead of two 
distinct phrases ) when using a Query String Query?
"photographic film" OR "photographic films"

These do not seem to work, and return the same number of results as just 
"photographic 
film":
"photographic film?"
"photographic film*"

Can wildcards not be placed inside Exact Phrase queries?  Is there a way to 
mimic this?

My goal is to be able to perform queries like this:
"photo* film?"

... capturing:
"photo film"
"photo films"
"photographic films"
"photography films"
etc...

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/66cc151f-a235-40d4-a125-2236aae0f9bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES Failures and Recovery

[Venu Nayar]

Did you do a failure analysis and what were your findings?

Thanks,
Venu

On Tuesday, June 11, 2013 7:41:57 AM UTC-7, Anand Nalya wrote:
>
> Hi,
>
> We are using 0.90.1 version ES and are planning for high availability 
> testing. While the entire scheme to enable the cluster to be highly 
> available is clear, I wanted to get some idea about ES Service lifetime in 
> terms of Mean-Time to Failure and Time of Recovery in cases of failure. Any 
> historic evidences will also help, as it will be vital for us to calculate 
> the actual availability of the system across an year.
>
> While I understand that ES provides seamless high availability through 
> replication, but any failure, will impact the performance to some extent 
> and this calculation will help in deriving the actual number of nodes that 
> we should consider without compromising on the performance as well, while 
> the system is available.
>
> Any ideas/facts would be very helpful .
>
> Thanks,
> Anand
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5f3c823f-0b54-4213-ac45-3dfa2f0b9af3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Synonym filter results in term facet

[ravi063]

Hi All,

I have a requirement in which I need to find distinct company names. I was 
using "Keyword" tokenizer for that field and through term facet I was able 
to get distinct company names. However terms facet treated company names 
like "ibm suisse", "ibm corporation", "ibm" as different companies.
Online documentation suggested me to use "Synonym filter" to solve this. My 
settings is:

curl -XPUT 'http://localhost:9200/dataindex/' -d '{
  "settings": {
"index": {
  "analysis": {
"analyzer": {
 "customAnalyzer": {
"type": "custom",
"tokenizer": "whitespace",
"filter": [
  "lowercase","synonym"  
]
  }
},
"filter": {
  "synonym" : {
  "type" : "synonym",
  "tokenizer": "keyword",
  "synonyms_path" : "analysis/synonym.txt"
  }
}
  }
}
  }
}'

My mapping is:

curl -XPUT 'http://localhost:9200/dataindex/tweet/_mapping' -d '
{ 
"tweet" : {
"properties" : {
"company": {
 "type": "string",
 "analyzer": "customAnalyzer"
}  
}   
}   
}'

In the synonym.txt file I have : ibm suisse, ibm corporation, ibm business, 
ibm => ibm corp ltd

Indexed data:
curl -XPUT 'http://localhost:9200/dataindex/tweet/1' -d '{
"company" : "ibm"
}'
curl -XPUT 'http://localhost:9200/dataindex/tweet/2' -d '{
"company" : "ibm corporation"
}'
curl -XPUT 'http://localhost:9200/dataindex/tweet/3' -d '{
"company" : "ibm suisse"
}'
curl -XPUT 'http://localhost:9200/dataindex/tweet/4' -d '{
"company" : "ibm business"
}'

If I run a terms facet:
{
  "facets": {
"loc_facet": {
  "terms": {
"field": "company"
  }
}
  }
}
I get 3 terms ie {term: ibm corp ltd, count: 3} {term: suisse, count: 1} 
{term: corporation, count: 1}
I want the facet result to return only one term: ibm corp ltd with count=3. 
This way i will get distinct company names and also map synonym names into 
single company name.
Please correct me if I am using wrong tokenizer or my approach is not 
correct.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1ba32926-7015-4b8a-89ae-bf43a2561b71%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] Elasticsearch CouchDB River plugin 2.2.0 released

[Elasticsearch Team]

Heya,


We are pleased to announce the release of the Elasticsearch CouchDB River 
plugin, version 2.2.0.

The CouchDB River plugin allows to hook into couchdb _changes feed and 
automatically index it into elasticsearch..

https://github.com/elasticsearch/elasticsearch-river-couchdb/

Release Notes - elasticsearch-river-couchdb - Version 2.2.0


Fix:
 * [67] - Race condition: NPE when starting with no database 
(https://github.com/elasticsearch/elasticsearch-river-couchdb/issues/67)
 * [66] - Race condition: exception while closing the river 
(https://github.com/elasticsearch/elasticsearch-river-couchdb/issues/66)

Update:
 * [64] - Default script engine should be mvel 
(https://github.com/elasticsearch/elasticsearch-river-couchdb/issues/64)
 * [62] - Use `script_type` instead of `scriptType` 
(https://github.com/elasticsearch/elasticsearch-river-couchdb/issues/62)
 * [60] - Default couchdb db name should be river name 
(https://github.com/elasticsearch/elasticsearch-river-couchdb/issues/60)
 * [56] - Update to elasticsearch 1.2.0 
(https://github.com/elasticsearch/elasticsearch-river-couchdb/issues/56)

New:
 * [47] - Move tests to elasticsearch test framework 
(https://github.com/elasticsearch/elasticsearch-river-couchdb/issues/47)
 * [17] - [TEST] Check that you can create a couchdb DB after the river 
creation 
(https://github.com/elasticsearch/elasticsearch-river-couchdb/issues/17)

Doc:
 * [55] - Clarify deleting documents in multi-type example 
(https://github.com/elasticsearch/elasticsearch-river-couchdb/pull/55)
 * [51] - Add documentation and test about parent/child 
(https://github.com/elasticsearch/elasticsearch-river-couchdb/issues/51)
 * [45] - [TEST] Add test and documentation for removing fields using scripts 
(https://github.com/elasticsearch/elasticsearch-river-couchdb/issues/45)


Issues, Pull requests, Feature requests are warmly welcome on 
elasticsearch-river-couchdb project repository: 
https://github.com/elasticsearch/elasticsearch-river-couchdb/
For questions or comments around this plugin, feel free to use elasticsearch 
mailing list: https://groups.google.com/forum/#!forum/elasticsearch

Enjoy,

-The Elasticsearch team

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/53cd521a.d327b40a.404d.3899SMTPIN_ADDED_MISSING%40gmr-mx.google.com.
For more options, visit https://groups.google.com/d/optout.

Re: Possibly to use current date as synonim?

[Ivan Brusic]

It would not be possible in Elasticsearch and it would be something that
should be handled on the client. Synonyms do not work at the semantic
level. The word "latest" has no special meaning and all that synonyms do is
create additional tokens for the same position in a sentence. Your date
field is probably (should be) a separate field from your your free text
content. Synonyms work on the same field.

You would probably need to pre-scan your queries and identify words that
might influence additional filters.

Cheers,

Ivan


On Sat, Jul 19, 2014 at 8:40 AM, Bdfy  wrote:

>
>
> 
>
> Possibly to use current date as synonim ? For example for query
> "latest/breaking news" I want to get last news in search. How to do it ?
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/5da5e274-c203-4542-8b10-1a06973764bc%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAmfhZvF1_2DczNj9CXjKhr%2BZ92dM9V54pzaczOTZaccg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Need help with Java API

[joergpra...@gmail.com]

To issue 1: you create a single node cluster without index, and a client of
it.

To issue 2: you see the UnavailableShardsException caused by a timeout
while indexing to a replica shard. This means, you may have set up a single
node cluster, but with replica level 1 (default) which needs 2 nodes for
indexing. Maybe there was once another node joining the cluster and ES
wants it back abut it never came (after 60 secs). Then ES returns the
timeout error. Maybe replica level 0 helps. You should also check the
cluster health. A color of green shows everything works, yellow means there
are too few nodes to satisfy the replica condition, and read means the
cluster is not in a consistent/usable state.

To issue 3: not sure what clusterName() means. I would use settings and add
a field "cluster.name". Maybe it is equivalent. You must ensure you use the
same "cluster.name" setting throughout all nodes and clients. You also can
not reuse data from clusters that had other names (look into the "data"
folder)

To issue 4: ES takes ~5 secs for discovery, the zen modules pings and waits
for responding master nodes by default. If you just test locally on your
developer machine, you should disable zen. Most easily by disabling
networking at all, by using NodeBuilder.nodeBuilder().local(true)...

Jörg




On Mon, Jul 21, 2014 at 6:53 PM, Alain Désilets 
wrote:

> I am trying to get started with the Java API, using the excellent tutorial
> found here:
>
>http://www.slideshare.net/dadoonet/hands-on-lab-elasticsearch
>
> But I am still having a lot of trouble.
>
> Below is a sample of code that I have written:
>
> package ca.nrc.ElasticSearch;
>
> import org.codehaus.jackson.map.ObjectMapper;
> import org.elasticsearch.action.get.GetResponse;
> import org.elasticsearch.action.index.IndexResponse;
> import org.elasticsearch.client.Client;
> import org.elasticsearch.node.NodeBuilder;
>
> public class ElasticSearchRunner {
>
> static ObjectMapper mapper;
> static Client client;
> static String indexName = "meal5";
> static String typeName = "beer";
> static long startTimeMSecs;
>
> public static void main(String[] args) throws Exception {
> startTimeMSecs = System.currentTimeMillis();
> mapper = new ObjectMapper(); // create once, reuse
>  echo("Creating the ElasticSearch client...");
> client = NodeBuilder.nodeBuilder().node().client(); // Does this create a
> brand new cluster?
> // client =
> NodeBuilder.nodeBuilder().clusterName("handson").client(true).node().client();
> // Joins existing cluster called "handson"
> echo("DONE creating the ElasticSearch client... Elapsed time =
> "+elapsedSecs()+" secs.");
>  echo("Creating a beer object...");
> Beer beer = new Beer("Heineken", Colour.PALE, 0.33, 3);
> String jsonString = mapper.writeValueAsString(beer);
> echo("DONE Creating a beer object...");
>
> echo("Indexing the beer object...");
> IndexResponse ir = null;
> ir = client.prepareIndex(indexName, typeName).setSource(jsonString)
> .execute().actionGet();
> echo("DONE Indexing the beer object...");
>
> echo("Retrieving the beer object...");
> GetResponse gr = null;
> gr = client.prepareGet(indexName, typeName, ir.getId()).execute()
> .actionGet();
> echo("DONE Retrieving the beer object...");
> }
>
>  public static float elapsedSecs() {
> float elapsed = (System.currentTimeMillis() - startTimeMSecs)/1000;
> return elapsed;
> }
>  public static void echo(String mess) {
> mess = mess + " (Elapsed so far: "+elapsedSecs()+" seconds)";
> System.out.println(mess);
> }
> }
>
>
> It works, "sort of"...
>
> If I use the first method for creating the client:
>
> client = NodeBuilder.nodeBuilder().node().client();
>
> Then it works fin the first time I run it. However:
>
> *** ISSUE 1: If I try to inspect the meal index with Marvel, I don't find
> it.
>
> Also,
>
> *** ISSUE 2: If I run the application a second time, I get the following
> output:
>
> Creating the ElasticSearch client... (Elapsed so far: 0.0 seconds)
> log4j:WARN No appenders could be found for logger (org.elasticsearch.node).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
> DONE creating the ElasticSearch client... Elapsed time = 9.0 secs.
> (Elapsed so far: 9.0 seconds)
> Creating a beer object... (Elapsed so far: 9.0 seconds)
> DONE Creating a beer object... (Elapsed so far: 9.0 seconds)
> Indexing the beer object... (Elapsed so far: 9.0 seconds)
> Exception in thread "main"
> org.elasticsearch.action.UnavailableShardsException: [meal5][0] [2]
> shardIt, [0] active : Timeout waiting for [1m], request: index
> {[meal5][beer][B3F5ZEmSTruqdnlxhYviFg],
> source[{"brand":"Heineken","colour":"PALE","size":0.33,"price":3.0}]}
> at
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseTimeoutFailure(TransportShardReplicationOperationAction.java:526)
> at
> org.elasticsearch.action.support.replication.TransportS

Re: Sort Order when relevance is equal

[Mateusz Kaczynski]

Seconding question.

Also, is it possible that it changed between versions 1.0 and 1.2? We're 
trying to upgrade and noticed a seemingly random order of documents with 
equal relevance in regression testing.

On Wednesday, 14 May 2014 00:49:55 UTC, Erich Lin wrote:
>
> Ignoring the bouncing results problem with multiple shards , is the order 
> of results deterministic when sorting by relevance score or any other 
> field.  
>
> What I mean by this is if two documents have the same score, 
>
> 1) will they always be in the same order if we set the preference 
> parameter to an arbitrary string like the user’s session ID. 
> 2) If so, is there a way to predict this deterministic order? Is it done 
> by ID of the documents as a tiebreaker etc? 
> 3) If not, could we specify that or do we have to do a secondary sort on 
> ID if we wanted to do that?
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/284b6bfd-68aa-4717-ab52-73d44f7cc196%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Need help with Java API

[Alain Désilets]

I am trying to get started with the Java API, using the excellent tutorial 
found here:

   http://www.slideshare.net/dadoonet/hands-on-lab-elasticsearch

But I am still having a lot of trouble.

Below is a sample of code that I have written:

package ca.nrc.ElasticSearch;

import org.codehaus.jackson.map.ObjectMapper;
import org.elasticsearch.action.get.GetResponse;
import org.elasticsearch.action.index.IndexResponse;
import org.elasticsearch.client.Client;
import org.elasticsearch.node.NodeBuilder;

public class ElasticSearchRunner {

static ObjectMapper mapper;
static Client client;
static String indexName = "meal5";
static String typeName = "beer";
static long startTimeMSecs;

public static void main(String[] args) throws Exception {
startTimeMSecs = System.currentTimeMillis(); 
mapper = new ObjectMapper(); // create once, reuse
 echo("Creating the ElasticSearch client..."); 
client = NodeBuilder.nodeBuilder().node().client(); // Does this create a 
brand new cluster?
// client = 
NodeBuilder.nodeBuilder().clusterName("handson").client(true).node().client(); 
// Joins existing cluster called "handson"
echo("DONE creating the ElasticSearch client... Elapsed time = 
"+elapsedSecs()+" secs."); 
 echo("Creating a beer object...");
Beer beer = new Beer("Heineken", Colour.PALE, 0.33, 3);
String jsonString = mapper.writeValueAsString(beer);
echo("DONE Creating a beer object...");

echo("Indexing the beer object...");
IndexResponse ir = null;
ir = client.prepareIndex(indexName, typeName).setSource(jsonString)
.execute().actionGet();
echo("DONE Indexing the beer object...");

echo("Retrieving the beer object...");
GetResponse gr = null;
gr = client.prepareGet(indexName, typeName, ir.getId()).execute()
.actionGet();
echo("DONE Retrieving the beer object..."); 
}

 public static float elapsedSecs() {
float elapsed = (System.currentTimeMillis() - startTimeMSecs)/1000;
return elapsed;
}
 public static void echo(String mess) {
mess = mess + " (Elapsed so far: "+elapsedSecs()+" seconds)";
System.out.println(mess);
}
}


It works, "sort of"...

If I use the first method for creating the client:

client = NodeBuilder.nodeBuilder().node().client();

Then it works fin the first time I run it. However:

*** ISSUE 1: If I try to inspect the meal index with Marvel, I don't find 
it.

Also, 

*** ISSUE 2: If I run the application a second time, I get the following 
output:

Creating the ElasticSearch client... (Elapsed so far: 0.0 seconds)
log4j:WARN No appenders could be found for logger (org.elasticsearch.node).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for 
more info.
DONE creating the ElasticSearch client... Elapsed time = 9.0 secs. (Elapsed 
so far: 9.0 seconds)
Creating a beer object... (Elapsed so far: 9.0 seconds)
DONE Creating a beer object... (Elapsed so far: 9.0 seconds)
Indexing the beer object... (Elapsed so far: 9.0 seconds)
Exception in thread "main" 
org.elasticsearch.action.UnavailableShardsException: [meal5][0] [2] 
shardIt, [0] active : Timeout waiting for [1m], request: index 
{[meal5][beer][B3F5ZEmSTruqdnlxhYviFg], 
source[{"brand":"Heineken","colour":"PALE","size":0.33,"price":3.0}]}
at 
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseTimeoutFailure(TransportShardReplicationOperationAction.java:526)
at 
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:516)
at 
org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:239)
at 
org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:494)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)


For me to be able to run the application again, I have to change the 
 indexName variable to a different value (ex: "meal6").

Also: 

*** ISSUE 3: If I try using the second method for creating a Client, then 
it doesn't work. More specifically, if I use this method:

client = 
NodeBuilder.nodeBuilder().clusterName("handson").client(true).node().client(); 

This yields:

Creating the ElasticSearch client... (Elapsed so far: 0.0 seconds)
log4j:WARN No appenders could be found for logger (org.elasticsearch.node).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for 
more info.
DONE creating the ElasticSearch client... Elapsed time = 34.0 secs. 
(Elapsed so far: 34.0 seconds)
Creating a beer object... (Elapsed so far: 34.0 seconds)
DONE Creating a beer object... (Elapsed so far: 34.0 seconds)
Indexing the beer object... (Elapsed so far: 34.0 seconds)
Exception in thread "main"

Handling node failure in ES cluster

[kmoore.cce]

I have had some issues recently as I've expanded my ES cluster, where a 
single node failure causes basically all other index/search operations to 
timeout and fail.

I am currently running elasticsearch v1.2.1 and primarily interface with 
the indices using the elasticsearch python module.

My cluster is 20 nodes, each an m1.large ec2 instance. I currently have ~18 
indices each with 5 shards and 3 replicas. The average size of each index 
is ~20GB and ~10 million documents (low is ~100K documents (300mb), high 
~40 million (35gb)).
I run each node with ES_MAX_SIZE=4g and ES_MIN_SIZE=512m. There are no 
other services running on the elasticsearch nodes, except ssh. I use zen 
unicast discovery with a set list of nodes. I have tried to enable 
'bootstrap.mlockall', but the ulimit settings do not seem to be working and 
I keep getting 'Unable to lock JVM memory (ENOMEM)' errors when starting a 
node (note: I didn't see this log message when running 0.90.7).

I have a fairly constant series of new or updated documents (I don't 
actually update, but rather reindex when a new document with the same id is 
found) that are being ingested all the time, and a number of users who are 
querying the data on a regular basis - most queries are set queries through 
the python API.

The issue I have now is that while data is being ingested/indexed, I will 
hit Java heap out of memory errors. I think this is related to garbage 
collection as that seems to be the last activity in the logs nearly 
everytime this occurs. I have tried adjusting the heap max to 6g, and that 
seems to help but I am not sure it solves the issue. In conjunction with 
that, when the out of memory error occurs it seems to cause the other nodes 
to stop working effectively, timeout errors in both indexing and searching.

My question is: what is the best way to support a node failing for this 
reason? I would obviously like to solve the underlying problem as well, but 
I would also like to be able to support a node crashing for some reason 
(whether it be because of me or because ec2 took it away). Shouldn't the 
failover in replicas support the missing node? I understand the cluster 
state would be yellow at this time, but I should be able to index and 
search data on the remaining nodes, correct?

Are there configuration changes I can make to better support the cluster 
and identify or solve the underyling issue? 

Any help is appreciated. I understand I have a lot to learn about 
Elasticsearch, but I am hoping I can add some stability/resiliency to my 
cluster.

Thanks in advance,
-Kevin

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/74ac48ec-0c05-4683-9c78-66d8c97687fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Kibana settings for IPFIX/Netflow

[Dhanasekaran Anbalagan]

Hi Janets,

currently I am also trying pmacct It's processing result. I am storing data
to elasticsearch, But currently struggling with dashboard creation, can you
share your kibana dashboard file. it's very useful to me.

-Dhanasekaran.

Did I learn something today? If not, I wasted it.


On Mon, Jul 21, 2014 at 5:19 AM, Janet Sullivan  wrote:

>  I’m tired, I didn’t explain that well, we use pmacct to do 1 minute
> aggregations.
>
>
>
> *From:* elasticsearch@googlegroups.com [mailto:
> elasticsearch@googlegroups.com] *On Behalf Of *Janet Sullivan
> *Sent:* Monday, July 21, 2014 1:50 AM
> *To:* elasticsearch@googlegroups.com
> *Subject:* Kibana settings for IPFIX/Netflow
>
>
>
> Every minute, we take a 1/4096 sample of traffic using IPFIX.  I want to
> graph this data as bits/sec in a histogram.  However, my math & kibana
> skills are failing me.
>
>
>
> Here is how I think it should be set up, but it’s always too low a value
> for Gbit/s:
>
>
>
> Chart Value: total
>
> Value Field: bytes (bytes per minute field)
>
> Scale: 32768 (4096 * 8 bits in a byte)
>
> Seconds, checked
>
> Interval 1m
>
> Y Format bytes
>
>
>
> Help?  Maybe I’m missing the obvious, but its 2 a.m. and I’m mystified.
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/f60cf590cb4146698ec2e4eacc8815b8%40BY2PR07MB043.namprd07.prod.outlook.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAJzooYdPZOny6EqgCWD-QWGBXVhSbXj0HKWxc-arqAu9kbE_7A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: puppet-elasticsearch options

[Andrej Rosenheinrich]

Hi Richard,

another question: you are creating the elasticsearch user and group 
somewhere in the module (havent found exactly where yet). My problem is 
that I have to create a directory for data_dir (on a different device) that 
is needed by the class (or instance, not sure), but I need the owner and 
the group to be able to set it otherwise the service won't start. Can I set 
a requirement in my file declaration to make sure that the user and the 
group already exist? Something like

  file { "/data/elasticsearch":
ensure  => "directory",
owner   => "elasticsearch",
group   => "elasticsearch",
require => ???
  }


Once again, thanks!
Andrej

Am Dienstag, 1. Juli 2014 14:37:55 UTC+2 schrieb Richard Pijnenburg:
>
> Hi Andrej,
>
> Sorry for the late response. Didn't get an update email about it.
>
> As long as you don't setup an instance with the 'elasticsearch::instance' 
> define it will only install the package but do nothing afterwards.
> I recently fixed that the default files from the packages are being 
> removed now.
> The memory can be set via the init_defaults hash by setting the ES_HEAP 
> option.
>
> The issue with 0.90.x versions is that it automatically starts up after 
> package installation.
> Since i don't stop it, it keeps running. Its advised to run a newer 
> version of ES since 0.90.x will be EOL'd at some point.
>
>
> On Thursday, June 26, 2014 2:24:47 PM UTC+1, Andrej Rosenheinrich wrote:
>>
>> Hi Richard,
>>
>> thanks for your answer, it for sure helped! Still, I am puzzling with a 
>> few effects and questions:
>>
>> 1.) I am a bit confused by your class/instance idea. I can do something 
>> pretty simple like class { 'elasticsearch' :  version => '0.90.7' } and it 
>> will install elasticsearch in the correct version using the default 
>> settings you defined. Repeating this (I tested every step on a fresh debian 
>> instance in a VM, no different puppet installation steps in between) with a 
>> config added in class like 
>>
>> class { 'elasticsearch' :
>> version => '0.90.7',
>> config => {
>>   'cluster'=> {
>> 'name' => 'andrejtest'
>>   },
>>   'http.port' => '9210'
>> }
>> }
>>   
>> I still get elasticsearch installed, but it completely ignores everything 
>> in the config. (I should be able to curl localhost:9210, but its up and 
>> running on the old default port, using the old cluster name). You explained 
>> overwriting for instances and classes a bit, so I tried the following thing 
>> (again, blank image, no previous installation) :
>>
>>   class { 'elasticsearch' :
>> version => '0.90.7',
>> config => {
>>   'cluster'=> {
>> 'name' => 'andrejtest'
>>   },
>>   'http.port' => '9210'
>> }
>>   }
>>
>>   elasticsearch::instance { 'es-01':
>>   }
>>
>> What happened is that I have two elasticsearch instances running, one 
>> with the default value and another one (es-01) that uses the provided 
>> configuration. Even freakier, I install java7 in my script before the 
>> snippet posted , the first (default based) elasticsearch version uses the 
>> standard openjdk-6 java, the second instance (es-01) uses java7. 
>> So, where is my mistake or what am I doing wrong? What would be the way 
>> to install and start only one service using provided configuration? And 
>> does elasticsearch::instance require an instance name? I would really miss 
>> the funny comic node names ;)
>>
>> 2. As you pointed out I can define all values from elasticsearch.yml in 
>> the config hash. But what about memory settings (I usually modify the 
>> init.d script for that), can I configure Xms and Xmx settings in the puppet 
>> module somehow?
>>
>> Logging configuration would be a nice-to-have (no must-have), just in 
>> case you were wondering ;)
>>
>> I hope my questions don't sound too confusing, if you could give me a 
>> hint on what I am doing wrong I would really appreciate it.
>>
>> Thanks in advance!
>> Andrej
>>
>>
>> Am Freitag, 20. Juni 2014 09:44:49 UTC+2 schrieb Richard Pijnenburg:
>>>
>>> Hi Andrej,
>>>
>>> Thank you for using the puppet module :-)
>>>
>>> The 'port' and 'discovery minimum' settings are both configuration 
>>> settings for the elasticsearch.yml file.
>>> You can set those in the 'config' option variable, for example:
>>>
>>> elasticsearch::instance { 'instancename':
>>>   config => { 'http.port' => '9210', 
>>> 'discovery.zen.minimum_master_nodes' => 3 }
>>> }
>>>
>>>
>>> For the logging part, management of the logging.yml file is very limited 
>>> at the moment but i hope to get some feedback on extending that.
>>> The thresholds for the slowlogs can be set in the same config option 
>>> variable.
>>> See 
>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-slowlog.html#index-slow-log
>>>  
>>> for more information.
>>>
>>> If you have any further questions, let me know.
>>>
>>> Cheers
>>>
>>> On Thurs

Re: Index performace with large arrays

[Steve Mee]

>
> Thanks for the response Jörg. That tells me exactly what I need to know... 
> stay away from very large arrays here in my design :-)
>

Cheers - Steve 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f06b544f-8aec-4c44-aa38-ce53e5f0be74%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Index performace with large arrays

[joergpra...@gmail.com]

If the user can opt out, I assume you have fewer opt outs than opt ins,
then you should use opt outs for an andnot filter :)

In that case, I would create an opt out index in the form index/type/id

users/optouts/

with docs containing a quite short array of opt outs

{ "optouts: [ "id1", "id2", ... "idn" ] }

so you can get the doc, read the opt out array, and add it as an "and not"
filter to your help tip query.

You could also add this optouts array to the user index, but this depends
on your overall design. If you want to remove the opt outs, you could
simply drop the optputs mapping type.

Regarding the array length, you can add as much values as you like, ES can
handle that. If the docs get long (I mean thousands of entries), they will
take substantial time just for fetching them, so I think you should prefer
a model with data as short as possible.

Jörg








On Mon, Jul 21, 2014 at 4:58 PM, Steve Mee  wrote:

> Hi there. I'm new to ES and would appreciate some advice on design
> concepts around large arrays.
>
> I am writing a help tip feature that pops up a message each time a user
> logs in. The user can flag a checkbox if they do not want to see this
> particular tip again.
>
> After playing with ElasticSearch the solution I came up with involved
> using a HelpTip document which contains an array of UserIds (identifying
> the users who have flagged that they do not want to see this tip again).
>
> Example1:
> HelpTip
> {
> "title": "Need help getting started?",
> "text": "Watch our overview video",
> "userArray": ["id1", "id2"]
> }
>
> I know ES can cope with large arrays but I wonder if there would be
> performance issues if this array grew to 4000+ IDs. This record would be
> regularly re-indexed (each time a new user ID is added to the array). would
> there be performance issues when indexing a document containing a large
> array field?
>
> Is this a sensible approach or would I be better using a relational model
> and holding the Help Tip info and the list of users in separate documents,
> then parsing them using two separate calls from my application?
> Example 2:
> HelpTip
> {
> "title": "Need help getting started?",
> "text": "Watch our overview video"
> }
>
> HelpTipUserFlags
> {
> HelpTipId: "1",
> UserId: "ID1"
> }
>
> Hope this makes sense. Thanks in advance for any help.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/9cba2c32-6266-4b87-b708-83ee64499dbf%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFw7P8FQu9%2Bc2ajj0Vg2wNvbpz%3D%2Bo9Af8R-5p9Cj8-7FQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Clustering/Sharding impact on query performance

[Kireet Reddy]

My working assumption had been that elasticsearch executes queries across all 
shards in parallel and then merges the results. So maybe shards <= cpu cores 
would help in this case where there is only one concurrent query. But I have 
never tested this assumption, out of curiosity during the 20 shard test did you 
still only see 1 cpu being used? Did you try 2 shards and get the same results?

On Jul 20, 2014, at 1:01 AM, 'Fin Sekun' via elasticsearch 
 wrote:

> Hi Kireet, thanks for your answer and sorry for the late response. More 
> shards doesn't help. It will slow down the system because each shard takes 
> quite some overhead to maintain a Lucene index and, the smaller the shards, 
> the bigger the overhead. Having more shards enhances the indexing performance 
> and allows to distribute a big index across machines, but I don't have a 
> cluster with a lot of machines. I could observe this negative effects while 
> testing with 20 shards.
> 
> It would be very cool if somebody could answer/comment to the question 
> summarized at the end of my post. Thanks again.
> 
> 
> 
> 
> 
> On Friday, July 11, 2014 3:02:50 AM UTC+2, Kireet Reddy wrote:
> I would test using multiple primary shards on a single machine. Since your 
> dataset seems to fit into RAM, this could help for these longer latency 
> queries.
> 
> On Thursday, July 10, 2014 12:24:26 AM UTC-7, Fin Sekun wrote:
> Any hints?
> 
> 
> 
> On Monday, July 7, 2014 3:51:19 PM UTC+2, Fin Sekun wrote:
> 
> Hi,
> 
> 
> SCENARIO
> 
> Our Elasticsearch database has ~2.5 million entries. Each entry has the three 
> analyzed fields "match", "sec_match" and "thi_match" (all contains 3-20 
> words) that will be used in this query:
> https://gist.github.com/anonymous/a8d1142512e5625e4e91
> 
> 
> ES runs on two types of servers:
> (1) Real servers (system has direct access to real CPUs, no virtualization) 
> of newest generation - Very performant!
> (2) Cloud servers with virtualized CPUs - Poor CPUs, but this is generic for 
> cloud services.
> 
> See https://gist.github.com/anonymous/3098b142c2bab51feecc for (1) and (2) 
> CPU details.
> 
> 
> ES settings:
> ES version 1.2.0 (jdk1.8.0_05)
> ES_HEAP_SIZE = 512m (we also tested with 1024m with same results)
> vm.max_map_count = 262144
> ulimit -n 64000
> ulimit -l unlimited
> index.number_of_shards: 1
> index.number_of_replicas: 0
> index.store.type: mmapfs
> threadpool.search.type: fixed
> threadpool.search.size: 75
> threadpool.search.queue_size: 5000
> 
> 
> Infrastructure:
> As you can see above, we don't use the cluster feature of ES (1 shard, 0 
> replicas). The reason is that our hosting infrastructure is based on 
> different providers.
> Upside: We aren't dependent on a single hosting provider. Downside: Our 
> servers aren't in the same LAN.
> 
> This means:
> - We cannot use ES sharding, because synchronisation via WAN (internet) seems 
> not a useful solution.
> - So, every ES-server has the complete dataset and we configured only one 
> shard and no replicas for higher performance.
> - We have a distribution process that updates the ES data on every host 
> frequently. This process is fine for us, because updates aren't very often 
> and perfect just-in-time ES synchronisation isn't necessary for our business 
> case.
> - If a server goes down/crashs, the central loadbalancer removes it (the 
> resulting minimal packet lost is acceptable).
>  
> 
> 
> 
> PROBLEM
> 
> For long query terms (6 and more keywords), we have very high CPU loads, even 
> on the high performance server (1), and this leads to high response times: 
> 1-4sec on server (1), 8-20sec on server (2). The system parameters while 
> querying:
> - Very high load (usually 100%) for the thread responsible CPU (the other 
> CPUs are idle in our test scenario)
> - No I/O load (the harddisks are fine)
> - No RAM bottlenecks
> 
> So, we think the file caching is working fine, because we have no I/O 
> problems and the garbage collector seams to be happy (jstat shows very few 
> GCs). The CPU is the problem, and ES hot-threads point to the Scorer module:
> https://gist.github.com/anonymous/9cecfd512cb533114b7d 
> 
> 
> 
> 
> SUMMARY/ASSUMPTIONS
> 
> - Our database size isn't very big and the query not very complex.
> - ES is designed for huge amount of data, but the key is clustering/sharding: 
> Data distribution to many servers means smaller indices, smaller indices 
> leads to fewer CPU load and short response times.
> - So, our database isn't big, but to big for a single CPU and this means 
> especially low performance (virtual) CPUs can only be used in sharding 
> environments.
> 
> If we don't want to lost the provider independency, we have only the 
> following two options:
> 
> 1) Simpler query (I think not possible in our case)
> 2) Smaller database
> 
> 
> 
> 
> QUESTIONS
> 
> Are our assumptions correct? Especially:
> 
> - Is clustering/sharding (also small indices) the main key to performance, 
> that means the only possibility

Notifications for a query

[P lva]

Hello Everyone,

Started working with Elasticsearch recently.
Just wanted to know if there's any way of being notified when a document
matches a query. (essentially create a monitoring system)
Can I use percolator to do this ?

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAO9TxdPOJGbY8nEj4Hj31LMr%3DaVLTwc_%3DwJW%3Drc7Ly2LAYc8nw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Index performace with large arrays

[Steve Mee]

Hi there. I'm new to ES and would appreciate some advice on design concepts 
around large arrays.

I am writing a help tip feature that pops up a message each time a user 
logs in. The user can flag a checkbox if they do not want to see this 
particular tip again.

After playing with ElasticSearch the solution I came up with involved using 
a HelpTip document which contains an array of UserIds (identifying the 
users who have flagged that they do not want to see this tip again). 

Example1:
HelpTip
{
"title": "Need help getting started?",
"text": "Watch our overview video",
"userArray": ["id1", "id2"]
}

I know ES can cope with large arrays but I wonder if there would be 
performance issues if this array grew to 4000+ IDs. This record would be 
regularly re-indexed (each time a new user ID is added to the array). would 
there be performance issues when indexing a document containing a large 
array field?

Is this a sensible approach or would I be better using a relational model 
and holding the Help Tip info and the list of users in separate documents, 
then parsing them using two separate calls from my application?
Example 2:
HelpTip
{
"title": "Need help getting started?",
"text": "Watch our overview video"
}

HelpTipUserFlags
{
HelpTipId: "1",
UserId: "ID1"
}

Hope this makes sense. Thanks in advance for any help.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9cba2c32-6266-4b87-b708-83ee64499dbf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Creating index with _timestamps

[David Pilato]

Ask for it using fields: 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-fields.html#search-request-fields

Using timestamp does not modify the original source.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 21 juil. 2014 à 12:44, Surajit Roy  a écrit :

Hi All,

I am trying to create an index with a _timestamp mapping.

curl -XPOST "http://localhost:9200/test"; -d'
{ "settings" :
 { "number_of_shards" : 5,
"number_of_replicas" : 1},
"mappings" : {
"stats" : {
 "_timestamp" : {
"enabled" : true, 
"store" : true }
}}}'


When I am writing the data in the index I dont see the time stamp coming up

curl -XPOST "http://localhost:9200/test/stats/1"; -d'{"a":1}'


{"_index":"test","_type":"stats","_id":"1","_version":1,"created":true}

Why am I not getting _timestamp, can anyone help?


Thanks
Surajit
-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAAxzCObuchG-a-tBfDQa-u3_ahX7hoaPq8g3jgVZXMBiEJH4oA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/873351B2-4C21-406A-B606-D9E8EE4A410D%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Re: Solaris 10 mlockall error code

[James Pace]

Thanks for the information, I'll chase it up and let you know how I get on.

James

On Monday, 21 July 2014 12:36:09 UTC+1, Jörg Prante wrote:
>
> Error 11 is a POSIX error number (errno) and means EAGAIN, "Resource 
> temporarily unavailable", which is documented.
>
> On Solaris 10, this means, you must first allow the Elasticsearch user to 
> allocate this amount of virtual memory.
>
> Switch to Elasticsearch user and then check the following values
>
> prctl -n project.max-shm-memory $$
> prctl -n process.max-address-space $$
>
> Then the sys admin could create a project with projadd and projmod and 
> change resource limits for the Elasticsearch user.
>
> The error can also mean that Solaris has not enough memory for mlockall 
> because there is already software running using the memory, or it the free 
> memory is too fragmented.
>
> Jörg
>
>
>
> On Mon, Jul 21, 2014 at 11:09 AM, James Pace  > wrote:
>
>> We have been having issues running ES with the bootstrap.mlockall: true 
>> setting. We get the following error:
>>
>> [2014-07-21 09:56:44,436][WARN ][common.jna] Unknown mlockall error 11
>>
>> I have googled around and looked in the solaris documentation for the 
>> description of the error codes and I have been unsuccessful. The solaris 
>> docs are here 
>> 
>>  but 
>> they only list 3 error codes. Is the error code 11 generated by ES?
>>
>> Our box has a total of 64 gigs of RAM and we give 32gigs to ES. We are 
>> running it using Oracle Java 1.7.13 64bit. Any help on the matter would be 
>> greatly appreciated!
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/bc9ba19f-c798-4e7f-9977-c9707709c47e%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f503e0b0-2e6b-4b6d-9e0e-07c2858783df%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Solaris 10 mlockall error code

[joergpra...@gmail.com]

Error 11 is a POSIX error number (errno) and means EAGAIN, "Resource
temporarily unavailable", which is documented.

On Solaris 10, this means, you must first allow the Elasticsearch user to
allocate this amount of virtual memory.

Switch to Elasticsearch user and then check the following values

prctl -n project.max-shm-memory $$
prctl -n process.max-address-space $$

Then the sys admin could create a project with projadd and projmod and
change resource limits for the Elasticsearch user.

The error can also mean that Solaris has not enough memory for mlockall
because there is already software running using the memory, or it the free
memory is too fragmented.

Jörg



On Mon, Jul 21, 2014 at 11:09 AM, James Pace  wrote:

> We have been having issues running ES with the bootstrap.mlockall: true
> setting. We get the following error:
>
> [2014-07-21 09:56:44,436][WARN ][common.jna] Unknown mlockall error 11
>
> I have googled around and looked in the solaris documentation for the
> description of the error codes and I have been unsuccessful. The solaris
> docs are here
> 
>  but
> they only list 3 error codes. Is the error code 11 generated by ES?
>
> Our box has a total of 64 gigs of RAM and we give 32gigs to ES. We are
> running it using Oracle Java 1.7.13 64bit. Any help on the matter would be
> greatly appreciated!
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/bc9ba19f-c798-4e7f-9977-c9707709c47e%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH%3DJQqs6UY7e2bO7z%3DhGacdMq_AJvoZpYCFfKqCAqMWaA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

marvel dashboard

[eunever32]

Hi,

The marvel overview by default shows 20 indices (third panel)

I guess there is some way to configure this 20? To say 40?

But how to do it?

Your help appreciated.

Regards.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5ca8a37f-cce9-4d08-b2f1-240957a7f0d5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Creating index with _timestamps

[Surajit Roy]

Hi All,

I am trying to create an index with a _timestamp mapping.

curl -XPOST "http://localhost:9200/test"; -d'
{ "settings" :
 { "number_of_shards" : 5,
"number_of_replicas" : 1},
"mappings" : {
"stats" : {
 "_timestamp" : {
"enabled" : true,
"store" : true }
}}}'


When I am writing the data in the index I dont see the time stamp coming up

curl -XPOST "http://localhost:9200/test/stats/1"; -d'{"a":1}'


{"_index":"test","_type":"stats","_id":"1","_version":1,"created":true}

Why am I not getting _timestamp, can anyone help?


Thanks
Surajit

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAAxzCObuchG-a-tBfDQa-u3_ahX7hoaPq8g3jgVZXMBiEJH4oA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] ElasticUI AngularJS Directives - Easily Build an Interface on top of Elasticsearch

[Yousef El-Dardiry]

Thanks all for the enthusiastic responses! Very excited to see the first 
contributions today and many stars on github over the last couple of weeks. 
Would love to hear your feedback / use cases in case you have any already :)

Regards,

- Yousef

http://www.elasticui.com
http://www.tweetbeam.com

On Thursday, July 3, 2014 3:06:11 PM UTC+2, Petar Djekic wrote:
>
> wow, this is really cool!
>
> On Wednesday, July 2, 2014 12:56:48 PM UTC+2, Yousef El-Dardiry wrote:
>>
>> Hi all,
>>
>> I just open sourced a set of AngularJS Directives for Elasticsearch. It 
>> enables developers to rapidly build a frontend (e.g.: faceted search 
>> engine) on top of Elasticsearch.
>>
>> http://www.elasticui.com (or github 
>> )
>>
>> It makes creating an aggregation and listing the buckets as simple as:
>>
>> *> eui-aggregation="ejs.TermsAggregation('text_agg').field('text').size(10)">*
>> *{{bucket}}*
>> **
>>
>> I think this was currently missing in the ecosystem, which is why I 
>> decided to build and open source it. I'd love any kind of feedback.
>>
>> - Yousef
>>
>> *-*
>> Another example; add a checkbox facet based on a field using one of the 
>> built-in widgets 
>> :
>>
>> **
>>
>> Resulting in
>> [image: checklist screenshot]
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d2b79b01-4460-4f07-ab38-508a22f50d37%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Solaris 10 mlockall error code

[James Pace]

Ooops good point! We are running version 1.2.1

On Monday, 21 July 2014 10:18:15 UTC+1, Mark Walkom wrote:
>
> What elasticsearch version are you on?
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
>
> On 21 July 2014 19:09, James Pace > 
> wrote:
>
>> We have been having issues running ES with the bootstrap.mlockall: true 
>> setting. We get the following error:
>>
>> [2014-07-21 09:56:44,436][WARN ][common.jna] Unknown mlockall error 11
>>
>> I have googled around and looked in the solaris documentation for the 
>> description of the error codes and I have been unsuccessful. The solaris 
>> docs are here 
>> 
>>  but 
>> they only list 3 error codes. Is the error code 11 generated by ES?
>>
>> Our box has a total of 64 gigs of RAM and we give 32gigs to ES. We are 
>> running it using Oracle Java 1.7.13 64bit. Any help on the matter would be 
>> greatly appreciated!
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/bc9ba19f-c798-4e7f-9977-c9707709c47e%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9b4fa7b1-2561-4f33-8a60-969ff317206b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: Kibana settings for IPFIX/Netflow

[Janet Sullivan]

I'm tired, I didn't explain that well, we use pmacct to do 1 minute 
aggregations.

From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On 
Behalf Of Janet Sullivan
Sent: Monday, July 21, 2014 1:50 AM
To: elasticsearch@googlegroups.com
Subject: Kibana settings for IPFIX/Netflow

Every minute, we take a 1/4096 sample of traffic using IPFIX.  I want to graph 
this data as bits/sec in a histogram.  However, my math & kibana skills are 
failing me.

Here is how I think it should be set up, but it's always too low a value for 
Gbit/s:

Chart Value: total
Value Field: bytes (bytes per minute field)
Scale: 32768 (4096 * 8 bits in a byte)
Seconds, checked
Interval 1m
Y Format bytes

Help?  Maybe I'm missing the obvious, but its 2 a.m. and I'm mystified.


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f60cf590cb4146698ec2e4eacc8815b8%40BY2PR07MB043.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

Re: Solaris 10 mlockall error code

[Mark Walkom]

What elasticsearch version are you on?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 21 July 2014 19:09, James Pace  wrote:

> We have been having issues running ES with the bootstrap.mlockall: true
> setting. We get the following error:
>
> [2014-07-21 09:56:44,436][WARN ][common.jna] Unknown mlockall error 11
>
> I have googled around and looked in the solaris documentation for the
> description of the error codes and I have been unsuccessful. The solaris
> docs are here
> 
>  but
> they only list 3 error codes. Is the error code 11 generated by ES?
>
> Our box has a total of 64 gigs of RAM and we give 32gigs to ES. We are
> running it using Oracle Java 1.7.13 64bit. Any help on the matter would be
> greatly appreciated!
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/bc9ba19f-c798-4e7f-9977-c9707709c47e%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624YbdrUjNe-S4UKvZcBQgnPup4QF3nidxLzLmDc82yd4mg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Solaris 10 mlockall error code

[James Pace]

We have been having issues running ES with the bootstrap.mlockall: true 
setting. We get the following error:

[2014-07-21 09:56:44,436][WARN ][common.jna] Unknown mlockall error 11

I have googled around and looked in the solaris documentation for the 
description of the error codes and I have been unsuccessful. The solaris 
docs are here 

 but 
they only list 3 error codes. Is the error code 11 generated by ES?

Our box has a total of 64 gigs of RAM and we give 32gigs to ES. We are 
running it using Oracle Java 1.7.13 64bit. Any help on the matter would be 
greatly appreciated!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/bc9ba19f-c798-4e7f-9977-c9707709c47e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Kibana settings for IPFIX/Netflow

[Janet Sullivan]

Every minute, we take a 1/4096 sample of traffic using IPFIX.  I want to graph 
this data as bits/sec in a histogram.  However, my math & kibana skills are 
failing me.

Here is how I think it should be set up, but it's always too low a value for 
Gbit/s:

Chart Value: total
Value Field: bytes (bytes per minute field)
Scale: 32768 (4096 * 8 bits in a byte)
Seconds, checked
Interval 1m
Y Format bytes

Help?  Maybe I'm missing the obvious, but its 2 a.m. and I'm mystified.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/73ae7aaffd5d44f290d16a14c679e2f8%40BN1PR07MB039.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

Re: LDAP authentication in Kibana

[naveen bajaj]

I configured ldap properties in httpd.conf.

AuthLDAPBindDN "uid=nabajaj,OU=Employee,OU=Cisco 
Users,DC=ds,DC=cisco,DC=com"
AuthLDAPBindPassword "password"
AuthLDAPURL "ldap://domain:389/OU=Employee,OU=Cisco 
Users,DC=ds,DC=cisco,DC=com?uid?sub?(objectClass=*)"
AuthType Basic
AuthBasicProvider "ldap"
authzldapauthoritative Off
AuthName "some text for login prompt"
  require valid-user

But it giving me error like

[error] [client x.x.x.x] user nabajaj: authentication failure for 
"/kibana": Password Mismatch

Please help me here.

On Wednesday, 18 June 2014 19:10:47 UTC+5:30, dharmendra pratap singh wrote:
>
> Hello Friends,
> Hope yo are doing good.
>
> In my application, I want to do the authentication in kibana using LDAP. 
> if anyone has done it before, please help me to come out of this.
>
> Appreciate your help.
>
> Regards
> Dharmendra
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7be86d2f-9c67-4ccd-92d2-37d154ecc6d8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.