Re: Importing Large Amounts of Data to Production Indices

2015-01-29 Thread webish
it (refresh = -1), then once it's been sent to ES turn indexing back on. On 28 January 2015 at 11:45, webish gre...@yoursports.com javascript: wrote: I have some production indices that needs to a large amount of data imported into them fairly frequently. Each time we import data the ES nodes

Performance with n Indices in Aliases

2015-01-27 Thread webish
I was wondering what is going on behind the scenes when adding n number of indices to an alias. Are there any performance implications? So an alias with a single index that has a single shard will allocate a single process to scan the index... So that would mean, with the same data, when

Importing Large Amounts of Data to Production Indices

2015-01-27 Thread webish
I have some production indices that needs to a large amount of data imported into them fairly frequently. Each time we import data the ES nodes become a huge bottleneck. I honestly expected a lot better performance out of them. Regardless, I would like to import data in a production ES

Elasricsearch Documentation on Functional Big O Notation

2015-01-27 Thread webish
I have often found myself looking into the performance of different functionality with Elasticsearch. I feel like this is a huge missing piece of the documentation with ES. The Redis documentation attempts to identify performance of functionality using Big O Notation or what they refer to as

Re: Pagination of continuously updating data

2014-10-09 Thread webish
Any luck Ben? Looks like I posted a similar question. https://groups.google.com/forum/#!topic/elasticsearch/mUwfSQvKnDw On Sunday, November 3, 2013 1:33:20 PM UTC-5, Ben wrote: I am new to ES and have a requirement to store real-time data streams. This is time-stamp based data. Can anyone

pagination with growing index

2014-10-08 Thread webish
I have come upon an interesting problem with pagination that I was wondering if anyone else solved elegantly. The problem can best be described by twitter's dev docs: https://dev.twitter.com/rest/public/timelines. Essentially, using the from and size parameters

WebServer on Same Machine As ES Instances

2014-05-14 Thread webish
Has anyone experimented with putting web servers (nginx) on the same machine as each ES node? For APIs that require many queries and network round trips this could reduce network latency -- You received this message because you are subscribed to the Google Groups elasticsearch group. To

Re: elasticsearch performance issue with load increasing reponse time is increasing

2014-05-13 Thread webish
Did you ever solve this? On Thursday, February 27, 2014 4:32:29 AM UTC-5, kondapal...@gmail.com wrote: Hi All We had elasticsearch with 4 Nodes the the Cluster each 8gb memory. when we are calling the elastic search query from java with 25 threads first thread is taking 400 msec where as

Time Date: Giant Index w/Shard Routing VS Small Indices w/Little Shards and Aliasing

2014-05-13 Thread webish
I am attempting to optimize time based data such as that of a newsfeed. I've been running tests with data broken into indices based on month, week, day. I'm using aliases to query the entire set or smaller ranges such as last-month, last-quarter. I'm still trying to figure out what will be

Re: Time Date: Giant Index w/Shard Routing VS Small Indices w/Little Shards and Aliasing

2014-05-13 Thread webish
comparison it's probably worth looking to increase the number of nodes, you'll get better much performance improvements. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com javascript: web: www.campaignmonitor.com On 14 May 2014 14:52, webish gre

Index Shard Number in Poor Performing Setup

2014-05-09 Thread webish
I'm seeing 40ms response times for queries using term and a custom ID field such as myId: 1234 I've inherited an Elastic Search database and application code. Currently there are two nodes and two main indexes. Index User is user profile data and Index News is event date in time series.

Elastic Search Document Index Numerical Rank/Index

2014-05-07 Thread webish
Knowns: 1. Large set of entity documents 2. Entity documents have large sets of relationship documents 3. Relationship documents contain a type field and field linking to another entity document that has the relationship 4. Entity documents use increment functions to maintain

Elastic Search Get Numerical Sort Index/Rank of Document

2014-05-07 Thread webish
Knowns: 1. Large set of entity documents 2. Entity documents have large sets of relationship documents (child docs) 3. Relationship documents contain a type field and field linking to another entity document that has the relationship 4. Entity documents use increment functions