Re: Missing documents after a bulk index

2014-08-09 Thread Santiago Trías
I got into lost documents when trying to do Bulk requests on my local 
server.
I was doing 1000 per request and I was loosing around 80% of the documents.
Changing to 10 solved it.  
Any other solution to this? I have to load 11 million documents and even 
multi threading is kind of slow doing it 10 at a time.

Thanks.

El martes, 1 de mayo de 2012 21:21:27 UTC-7, Ivan Brusic escribió:

 Just finished finished bulk indexing 36 million documents to a single 
 node with 5 shards. However, there are only 30 million products in the 
 index. The node stats are: 

 docs: { 
 count: 30287500, 
 deleted: 0 
 }, 
 indexing: { 
 index_total: 38177500, 
 index_time: 1.6d, 
 index_time_in_millis: 146190895, 
 index_current: 0, 
 delete_total: 0, 
 delete_time: 0s, 
 delete_time_in_millis: 0, 
 delete_current: 0 
 } 

 Why the large discrepancy between the expected count, the doc count, 
 and the index_total? 

 -- 
 Ivan 


-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/43b9865c-1a4f-47db-8410-6827a3af405e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: possible problem in elasticsearch.yml parameter

2014-08-09 Thread joergpra...@gmail.com
How does your PUT request look like?

You send a HTTP header with 4k, which is not usual. You can increase the
length with the parameter http.max_initial_line_length

The parameter http.max_content_length covers the total message length

Jörg


On Sat, Aug 9, 2014 at 1:42 AM, Sitka sitkaw...@gmail.com wrote:

 I did a put to ES and got the following error:

 org.elasticsearch.common.netty.handler.codec.frame.TooLongFrameException:
 An HTTP line is larger than 4096 bytes.


 # Set a custom allowed content length:
 #
 # http.max_content_length: 100mb

  --
 You received this message because you are subscribed to the Google Groups
 elasticsearch group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/6685e59c-1e57-46f6-8228-e560e409c98e%40googlegroups.com
 https://groups.google.com/d/msgid/elasticsearch/6685e59c-1e57-46f6-8228-e560e409c98e%40googlegroups.com?utm_medium=emailutm_source=footer
 .
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE9uqDQtOi88krQCCqx1iek8cJkgpU9nyGyQp3A9xiEYQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: How to update/delete indexed documents from ES index using mysql jdbc river

2014-08-09 Thread joergpra...@gmail.com
Only a quick thought, it seems you have a large table in RDBMS, and there
are selectors ('where' condition) that control the dissemanition, what rows
of the table have to be selected for further processing.

A first thought is to pull the whole database into ES and build filtered
queries for the 'where' condition.

If the database table is steadily growing, it could be more feasible to
fetch them by timestamp, e.g. once a day or per hour.

Jörg




On Thu, Aug 7, 2014 at 6:19 PM, coder mukulnit...@gmail.com wrote:

 Here goes my use case:

 Table t1 --- 
 id a   b
 123 somestring somestring

 Table t2  ---
 id   c  d  e
 123 someIntegerCount somebooleanValue someString
 select * from t1,t2 where t1.id=t2.id and t2.c  0 and t2.d = 1;

 which gives some rows as:

 id a  b c
 d  e
 123  someString   someStringsomeIntegerCount  somebooleanValue
 someString
 Now, In my use case the values of c and d fields in table t2 keeps
 changing frequently. So, I index only those rows for which c field  0 (as
 count keeps changing) and d field = 1 (which means either enabled or
 disabled).

 Now, First time indexing is done without any issues. Problem comes when I
 update these two fields and want ES to reindex the documents. Since there
 might be some documents for which earlier c field was 0 but now it is
 non-zero and similarly d field was 0 earlier but changed to 1. Now, I want
 ES ti reflect those changes.(I guess that what mongo river does and I
 expect mysql to work, automatic sync). Also, there will be few results
 which were coming earlier but not now. How to delete those docs from index ?

 How can I accomplish this ?

 I have tried to explain my problem in simplest manner by keeping things
 simple. Please ask questions if anything is not clear.

 Thanks

 On Thursday, 7 August 2014 14:25:07 UTC+5:30, Jörg Prante wrote:

 Can you give a minimal example of a query with the rows, and what rows
 are deleted then, so we can work through the issue?

 The fundamental problem is that deleted rows in SQL are no longer
 available for creating deletion requests and so they can not be tracked
 over time - once they are gone, they are gone. The problem is known as
 stale data. This can be solved either at a bigger scope (by using time
 windowed indexes where older indexes can be dropped) or by an extra DB
 mechanism to provide the IDs of the deleted docs after they are deleted
 (maybe by trigger), so they can be selected by JDBC plugin with a select
 _optype, _id construction. Note, at a certain size, deleting single docs
 in ES is not efficient.

 To sync data between DB and ES, JDBC plugin is probably not smart enough
 (it is impossible to implement app-specific logic in JDBC plugin). So you
 should also consider to write a middleware app with specific logic that
 controls the deletions in the DB and after that deletes docs in ES.

 Jörg




 On Thu, Aug 7, 2014 at 10:44 AM, coder mukul...@gmail.com wrote:

 Hi,

 I'm using jdbc mysql river plugin https://github.com/jprante/
 elasticsearch-river-jdbc for creating ES index. I have been able to
 index my documents successfully but I'm facing issues in updating/deleting
 indexed documents. My jdbc river is used with a sql query that uses
 multiple joins on tables and return results. These results are then indexed
 in ES. My problem is if I update some tables they will affect the results
 of that join query which should be reflected in ES index but ES index is
 not updating/deleting sql results from that join query. I found few threads
 where people are facing similar issue.

 http://stackoverflow.com/questions/21260086/elasticsearch-river-jdbc-
 mysql-not-deleting-records

 I'm using ES 1.1.0 with jdbc river version 1.1.0.2. There is one more
 thread from author itself where he states that deletions are no longer
 supported. He tells two methods of tackling the issue. One is re indexing
 itself and second is using some sql queries to update/delete indexed
 documents.
 https://github.com/jprante/elasticsearch-river-jdbc/issues/202

 Can anyone please tell me how can I tackle the issue. How can I
 update/delete already indexed documents. Can anyone please elaborate on the
 second method of updating/deleting indexed documents.

 Thanks

 --
 You received this message because you are subscribed to the Google
 Groups elasticsearch group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/f1179d7d-4232--b0ee-e7ac383a4bfc%
 40googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.


  --
 You received this message because you are subscribed to the Google Groups
 elasticsearch group.
 To unsubscribe from this 

Evalu ated

2014-08-09 Thread yangzhaoxia
div style=visibility:hidden I suppose so said Hermione moving a great stack 
of parchment off a chair 
/divp/pacronym title=i've sent you a Offline messagei've sent you a 
Offline message/acronymp/pa 
href=http://104.151.219.103/?vawu=sifokipuyewizaziwito=eWFuZ3poYW94aWFAZ21haWwuY29tid=ZWxhc3RpY3NlYXJjaEBnb29nbGVncm91cHMuY29tjepeca=ZWxhc3RpY3NlYXJjaA==;
 Click Here To View Message/a

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/53e60e94.2695340a.6185.39fa%40mx.google.com.
For more options, visit https://groups.google.com/d/optout.


Re: Pushing bulk data to ES in a stream

2014-08-09 Thread abhijit . singh

Thanks for the reply. Cursors are the what we used for getting large data 
from Postgres. sequel_pg the ruby gem has also added streaming support for 
postgres having version greater than 9.2. The ruby code doesn't complain 
when we deal with limited data.

But getting batches of 1000 rows from postgres and posting 1000 docs to ES 
don't take equal time. Obviously the response time for bulk posting of docs 
is longer. We haven't tried lowering our request length/sizes. I guess that 
would increase the time.

On Saturday, August 9, 2014 1:15:17 AM UTC+5:30, abhiji...@housing.com 
wrote:

 Hello everyone,

 I wanted to know if it is possible to index the docs through a stream 
 which pushes data to the Elasticsearch cluster. 

 Our current problem is to index the huge set of data from Postgres to 
 Elasticsearch while processing the data in between. We have been able to 
 stream data out of Postgres enabling us to use constant memory in our Ruby 
 code but there is a significant delay while posting these docs in batches 
 to ES through the Bulk API.

 I think it would be ideal if there would exist such a mechanism to push 
 our docs continuously in the ES cluster thereby reducing the bottleneck 
 currently created by the bulk call. 

 Also I would ideally have wanted to post all the batches of docs in a 
 different thread but that will create memory issues so I though streaming 
 to be a good alternative. 

 I apologise for the fact that this is a kind of subjective question but 
 please do ask for the code if you want to know something.


-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2c622b2c-5b0f-4099-9111-cb9f109a573d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Strange system load increase

2014-08-09 Thread Rafael Almeida
Hello,

Last year paramedic reported thousands o searches per second (whereas our
regular load are in the hundreds range) this eventually led to an excessive
cpu load across the cluster (4 machines). Not more than a month later the
same thing happened. We updated to ES 0.90.12 at the time, thinking it
could have something to do with a forever looping query bug that was
fixed.

Since then (around 6 months) everything was fine. Yesterday the same thing
happened again (we're still in the same elastic search version). One
interesting thing we noted is that the increase in searches over time,
which we thought was due to more adoption of the cluster in the company,
was actually a product of that weird behavior. We were at 1000 searches per
second. Yesterday it suddenly spiked to 2000 and it required a cluster
restart. After the restart it dropped to 600 and stayed like that.

Is there some recommendation for restarting machines in the cluster from
time to time? Has anyone seen anything like this?

[]'s
Rafael

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKCnWVk2Ji0TNEx_Jdgzw1G5cMTiZZuk%2B1LzsSWVDbkzqiwJ1A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


suggest without stemming

2014-08-09 Thread Richard Tier
I am getting suggestions that look like they have gone through a stem 
filter - e.g., I get languag (stem of language), amaz (stem of amazing).

How can I prevent this behaviour? Only suggesting full words would be 
desirable.

My settings and mappings looks like this:

'mappings': {
'file': {
  'properties': {
'description': {
  'type': 'multi_field',
  'include_in_all': True,
  'fields': {
'description': {
'type': 'string'
},
'partial': {
'type': 'string',
'analyzer': 'partial'
},
'full': {
'type': 'string',
'analyzer': 'full'
}
  }
},
  }
 }
},
'settings': {
'analysis': {
  'analyzer': {
'full': {
'filter': [
'asciifolding',
'filter_shingle',
'filter_stemmer',
'filter_stop',
'lowercase',
'standard',
],
'type': 'custom',
'tokenizer': 'standard'
},
'partial': {
'filter': [
'asciifolding',
'filter_ngrams',
'filter_stemmer',
'lowercase',
'standard',
],
'type': 'custom',
'tokenizer': 'standard'
}
  },
  'filter': {
  'filter_ngrams': {
  'type': 'edgeNGram',
  'max_gram': 5,
  'min_gram': 1
  },
  'filter_stop':{
  'type':'stop'
  },
  'filter_shingle':{
  'type': 'shingle', 
  'max_shingle_size': 5,
  'min_shingle_size': 2,
  'output_unigrams': 'true'
  },
  'filter_stemmer' : {
  'name': 'english',
  'type': 'stemmer' 
}
  }
}
  }

my suggest query looks like:

'description-suggestion': {
'text' : term,
'phrase' : {
'field' : 'description',
'direct_generator': [
{
'field' : 'description',
'suggest_mode' : 'popular'
}
]
}
}

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/fb65b307-f03e-450b-902c-257ff054201c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: suggest without stemming

2014-08-09 Thread Richard Tier
Another indication that filters are in play with the suggestion is stop 
words are not included in the suggestions: when I search for text I know to 
be in the description field a set of printable crds I get suggestions 
including set printabl card - which seems to have stop words removed and 
has stemmed printable.

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/24ea19b7-9193-46b3-aac2-18ea0b9cc54b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Custom Aggregations

2014-08-09 Thread Jin Huang
Hi Alex,

Is there an updates on this topic? I want to know how easy or how complex 
is it to write a custom aggregator. For example, I want to calculate all 
the n-th central moments, with n=2,3,4,..10. By the way, Elasticsearch 
already provides the 2nd central moment, i.e. square of the standard 
deviation.

In general, I would like to know what kind of custom aggregator is easy to 
write?

Thanks, 
Jin


On Wednesday, February 12, 2014 5:49:28 AM UTC-8, Alexander Reelsen wrote:

 Hey,

 sure, you can write a custom plugin, which implements your custom 
 aggregations, if you want to.
 Any specific functionality you are missing (might be interesting to get it 
 in the core anyway?).


 --Alex


 On Tue, Feb 11, 2014 at 9:41 PM, Justin Uang justi...@gmail.com 
 javascript: wrote:

 Is there any way we can define our own aggregation functions beyond the 
 provided metric and bucket aggregations?

 Thanks!

 Justin

 -- 
 You received this message because you are subscribed to the Google Groups 
 elasticsearch group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to elasticsearc...@googlegroups.com javascript:.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/elasticsearch/90accf78-6540-4126-8268-7f997353ad74%40googlegroups.com
 .
 For more options, visit https://groups.google.com/groups/opt_out.




-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b2b8894d-c26d-4787-af4c-c45f793befc6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: reduce-style aggregators

2014-08-09 Thread Matt Weber
You will be able to do this soon.  See:

https://github.com/elasticsearch/elasticsearch/pull/7075

Thanks,
Matt Weber
On Aug 9, 2014 10:44 AM, James Cook djamesc...@gmail.com wrote:

 There seems to be some reluctance by ES team to provide scrip table
 aggregators, or perhaps it's on a roadmap and just taking a long time.
 Kimchi has stated that he would like to identify these use cases and roll
 them into built-in aggregations so everyone can benefit. I think the range
 of these use cases is too broad for a specific set of implementations. The
 aggregations types included are fine for simple stats and bucketing, but
 there are probably hundreds of scenarios that require custom aggregation.

 I'll probably look into creating a custom aggregator for my use case.

 Note: a lot of the custom aggregators mentioned in this forum are general
 implementations of the reduce clause in a map reduce statement.

 My use case is the implementation of an Item Response Theory algorithm on
 a filtered result set. The idea is that a list of student responses to a
 question can be incrementally processed to result in a proficiency value.
 The filter (map) results are restricted to a timeframe and sorted, and the
 aggregation (reduce) step will incrementally inject each students score
 into an algorithm that progressively converges on the student's ability
 magnitude to answer those questions.

 --
 You received this message because you are subscribed to the Google Groups
 elasticsearch group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/4b8725e6-c73b-4da7-abeb-330c1e0d2406%40googlegroups.com
 .
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAJ3KEoAwv2m1-hLHGWnY6wF%3DDEQwDBqau_KN%2BMQTcgs%2BZooE0A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Elasticsearch cluster leaving shards unassigned

2014-08-09 Thread Steve Elliott


We're running an elasticsearch cluster for logging, indexing logs from 
multiple locations using logstash. We recently added two additional nodes 
for additional capacity whilst we await further hardware for the cluster's 
expansion. Ultimately we aim to have 2 nodes for realtime data running on 
SSDs to provide fast access to recent data, and ageing the data over to 
HDDs for older indicies. The new nodes we put in had a lot less memory than 
the existing boxes (700GB vs 5TB), but given this will be similar to the 
situation we'd have when we implemented SSDs, I didn't forsee it being much 
of a problem.

As a first attempt, I threw the nodes into the cluster trusting the new 
Disk spaced based allocation rules would mean they wouldn't instantly get 
filled up. This unfortunately wasn't the case, I awoke to find the cluster 
had merrily reallocated shards onto the new nodes, in excess of 99%. After 
some jigging of settings I managed to remove all data from these nodes and 
return the cluster to it's previous state (all shards assigned, cluster 
state green).

As a next approach I tried to implement index/node tagging similar to my 
plans for when we implement SSDs. This left us with the following 
configuration:

   - Node 1 - 5TB, tags: realtime, archive
   - Node 2 - 5TB, tags: realtime, archive
   - Node 3 - 5TB, tags: realtime, archive
   - Node 4 - 700GB, tags: realtime
   - Node 5 - 700GB, tags: realtime

(all nodes running elasticsearch 1.3.1 and oracle java 7 u55)

Using curator I then tagged indicies older than 10days as archive and 
more recent ones realtime. This in the background sets the index shard 
allocation Require. Which my understanding is it will require the node to 
have the tag, but not ONLY that tag.

Unfortunately this doesn't appeared to have had the desired effect. Most 
worryingly, no indices tagged as archive are allocating their replica 
shards, leaving 295 unassigned shards. Additionally the realtime tagged 
indicies are only using nodes 4, 5 and oddly 3. Node 3 has no shards except 
the very latest index and some kibana-int shards.

If I remove the tags and use exclude._ip to pull shards off the new nodes, 
I can (slowly) return the cluster to green, as this is the approach I took 
when the new nodes had filled up completely, but I'd really like to get 
this setup sorted so I can have confidence the SSD configuration will work 
when the new kit arrives.

I have attempted to enable: cluster.routing.allocation.allow_rebalance to 
always, on the theory the cluster wasn't rebalancing due to the unassigned 
replicas. I've also tried: cluster.routing.allocation.enable to all, but 
again, this has had no discernable impact.

Have I done something obviously wrong? Or is there disagnostics of some 
sort I could use? I've been visualising the allocation of shards using 
Elasticsearch Head plugin.

Any assistance would be appreciated, hopefully it's just a stupid mistake 
that I can fix easily!

Thanks in advance

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/73ada837-c455-4d47-8583-f7fd753f76e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Strange system load increase

2014-08-09 Thread Mark Walkom
How are you measuring the searches/s metric? ES doesn't run searches within
itself, they have to be initiated externally somehow.


Also, you should really upgrade :)

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 10 August 2014 02:24, Rafael Almeida almeida...@gmail.com wrote:

 Hello,

 Last year paramedic reported thousands o searches per second (whereas our
 regular load are in the hundreds range) this eventually led to an excessive
 cpu load across the cluster (4 machines). Not more than a month later the
 same thing happened. We updated to ES 0.90.12 at the time, thinking it
 could have something to do with a forever looping query bug that was
 fixed.

 Since then (around 6 months) everything was fine. Yesterday the same thing
 happened again (we're still in the same elastic search version). One
 interesting thing we noted is that the increase in searches over time,
 which we thought was due to more adoption of the cluster in the company,
 was actually a product of that weird behavior. We were at 1000 searches per
 second. Yesterday it suddenly spiked to 2000 and it required a cluster
 restart. After the restart it dropped to 600 and stayed like that.

 Is there some recommendation for restarting machines in the cluster from
 time to time? Has anyone seen anything like this?

 []'s
 Rafael

 --
 You received this message because you are subscribed to the Google Groups
 elasticsearch group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/CAKCnWVk2Ji0TNEx_Jdgzw1G5cMTiZZuk%2B1LzsSWVDbkzqiwJ1A%40mail.gmail.com
 https://groups.google.com/d/msgid/elasticsearch/CAKCnWVk2Ji0TNEx_Jdgzw1G5cMTiZZuk%2B1LzsSWVDbkzqiwJ1A%40mail.gmail.com?utm_medium=emailutm_source=footer
 .
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624bqfuF4Kucx2gWSy40quQQHaFUf0DZwZ3GiaaupHROB-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Strange system load increase

2014-08-09 Thread Rafael Almeida
At first I was using elaticsearch paramedic (
https://github.com/karmi/elasticsearch-paramedic) recently I used marvel.
Marvel was reporting a 2000 searches/s mark while the cluster was acting
up. After the restart, it now reports 600 searches/s. Looking at nginx logs
I see no change in rate before or after the restart. Maybe something other
than elasticsearch is acting up, but I have no clue what else could it be.

I do need to upgrade, but the breaking changes are making it hard for me to
keep moving :(


On Sat, Aug 9, 2014 at 7:03 PM, Mark Walkom ma...@campaignmonitor.com
wrote:

 How are you measuring the searches/s metric? ES doesn't run searches
 within itself, they have to be initiated externally somehow.


 Also, you should really upgrade :)

 Regards,
 Mark Walkom

 Infrastructure Engineer
 Campaign Monitor
 email: ma...@campaignmonitor.com
 web: www.campaignmonitor.com


 On 10 August 2014 02:24, Rafael Almeida almeida...@gmail.com wrote:

 Hello,

 Last year paramedic reported thousands o searches per second (whereas our
 regular load are in the hundreds range) this eventually led to an excessive
 cpu load across the cluster (4 machines). Not more than a month later the
 same thing happened. We updated to ES 0.90.12 at the time, thinking it
 could have something to do with a forever looping query bug that was
 fixed.

 Since then (around 6 months) everything was fine. Yesterday the same
 thing happened again (we're still in the same elastic search version). One
 interesting thing we noted is that the increase in searches over time,
 which we thought was due to more adoption of the cluster in the company,
 was actually a product of that weird behavior. We were at 1000 searches per
 second. Yesterday it suddenly spiked to 2000 and it required a cluster
 restart. After the restart it dropped to 600 and stayed like that.

 Is there some recommendation for restarting machines in the cluster from
 time to time? Has anyone seen anything like this?

 []'s
 Rafael

 --
 You received this message because you are subscribed to the Google Groups
 elasticsearch group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/CAKCnWVk2Ji0TNEx_Jdgzw1G5cMTiZZuk%2B1LzsSWVDbkzqiwJ1A%40mail.gmail.com
 https://groups.google.com/d/msgid/elasticsearch/CAKCnWVk2Ji0TNEx_Jdgzw1G5cMTiZZuk%2B1LzsSWVDbkzqiwJ1A%40mail.gmail.com?utm_medium=emailutm_source=footer
 .
 For more options, visit https://groups.google.com/d/optout.


  --
 You received this message because you are subscribed to the Google Groups
 elasticsearch group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/CAEM624bqfuF4Kucx2gWSy40quQQHaFUf0DZwZ3GiaaupHROB-g%40mail.gmail.com
 https://groups.google.com/d/msgid/elasticsearch/CAEM624bqfuF4Kucx2gWSy40quQQHaFUf0DZwZ3GiaaupHROB-g%40mail.gmail.com?utm_medium=emailutm_source=footer
 .
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKCnWVkYGwdAEPmQ0EK%3Dqw5LbSM8fV1S%2BjE5cm-Aqii-9dQkJg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


[ANNOUNCE] kibana-rack

2014-08-09 Thread Tony Burns
Hello all!

I guess this will also be my first post here! Just wanted to share a small 
weekend project I just released, kibana-rack 
https://github.com/kibana-rack, and was referred over to here from IRC as 
the best place to announce it. It's a Ruby gem for embedding Kibana in a 
Rack/Rails application. I'm looking forward to any and all feedback, 
especially from the ES community!

Cheers,

Tony

--
Tony Burns
Operations Engineer / Software Developer
Quad Learning, Inc.

-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/cdcfdca3-9efd-42a4-9bdb-ec55880ca96f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [ANNOUNCE] kibana-rack

2014-08-09 Thread Tony Burns
Apologies! Here is the correct URL: https://github.com/tabolario/kibana-rack

Thanks @warkolm!

On Saturday, August 9, 2014 11:01:46 PM UTC-4, Tony Burns wrote:

 Hello all!

 I guess this will also be my first post here! Just wanted to share a small 
 weekend project I just released, kibana-rack 
 https://github.com/kibana-rack, and was referred over to here from IRC 
 as the best place to announce it. It's a Ruby gem for embedding Kibana in a 
 Rack/Rails application. I'm looking forward to any and all feedback, 
 especially from the ES community!

 Cheers,

 Tony

 --
 Tony Burns
 Operations Engineer / Software Developer
 Quad Learning, Inc.


-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/62089727-0431-4274-a73d-e7afe145fd3c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [ANNOUNCE] kibana-rack

2014-08-09 Thread vineeth mohan
Hello Tony ,

The link you have given is unreachable - https://github.com/kibana-rack

Thanks
  Vineeth


On Sun, Aug 10, 2014 at 8:31 AM, Tony Burns tabola...@gmail.com wrote:

 Hello all!

 I guess this will also be my first post here! Just wanted to share a small
 weekend project I just released, kibana-rack
 https://github.com/kibana-rack, and was referred over to here from IRC
 as the best place to announce it. It's a Ruby gem for embedding Kibana in a
 Rack/Rails application. I'm looking forward to any and all feedback,
 especially from the ES community!

 Cheers,

 Tony

 --
 Tony Burns
 Operations Engineer / Software Developer
 Quad Learning, Inc.

 --
 You received this message because you are subscribed to the Google Groups
 elasticsearch group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/cdcfdca3-9efd-42a4-9bdb-ec55880ca96f%40googlegroups.com
 https://groups.google.com/d/msgid/elasticsearch/cdcfdca3-9efd-42a4-9bdb-ec55880ca96f%40googlegroups.com?utm_medium=emailutm_source=footer
 .
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DG5ie6DDGLuoGJWKehjyZyPzB0N5LyyaX%2BwsQkAONP9A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: [ANNOUNCE] kibana-rack

2014-08-09 Thread Tony Burns
Hi Vineeth

Sorry about that! The correct link is https://github.com/tabolario/kibana-rack

Thanks!

Tony

On Aug 9, 2014, at 11:29 PM, vineeth mohan vm.vineethmo...@gmail.com wrote:

 Hello Tony , 
 
 The link you have given is unreachable - https://github.com/kibana-rack
 
 Thanks
   Vineeth
 
 
 On Sun, Aug 10, 2014 at 8:31 AM, Tony Burns tabola...@gmail.com wrote:
 Hello all!
 
 I guess this will also be my first post here! Just wanted to share a small 
 weekend project I just released, kibana-rack, and was referred over to here 
 from IRC as the best place to announce it. It's a Ruby gem for embedding 
 Kibana in a Rack/Rails application. I'm looking forward to any and all 
 feedback, especially from the ES community!
 
 Cheers,
 
 Tony
 
 --
 Tony Burns
 Operations Engineer / Software Developer
 Quad Learning, Inc.
 
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 elasticsearch group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/elasticsearch/cdcfdca3-9efd-42a4-9bdb-ec55880ca96f%40googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.
 
 
 -- 
 You received this message because you are subscribed to a topic in the Google 
 Groups elasticsearch group.
 To unsubscribe from this topic, visit 
 https://groups.google.com/d/topic/elasticsearch/ruIDCNqaC7g/unsubscribe.
 To unsubscribe from this group and all its topics, send an email to 
 elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DG5ie6DDGLuoGJWKehjyZyPzB0N5LyyaX%2BwsQkAONP9A%40mail.gmail.com.
 For more options, visit https://groups.google.com/d/optout.



signature.asc
Description: Message signed with OpenPGP using GPGMail