Re: Elasticsearch Aggregation time
Hi Adrian,
thanks,
we are already using count type , the filter will be an actual filter ,
we want different filters on each aggregation so it would not be possible
to do a filtered query.
Can we improve using more replications or more sharding .
On Wednesday, 12 November 2014 04:16:54 UTC+5:30, Adrien Grand wrote:
Hi Ankur,
I assume that your revenueFilter aggregation uses an actual filter and not
a match_all filter? Otherwise you could just remove it.
Are you actually interested in the top hits that match your query? If not,
you could switch to the `count` search type and move the filter from your
aggregation to the filtered_query, this would be faster.
On Mon, Nov 10, 2014 at 11:53 AM, Ankur Goel ankr...@gmail.com
javascript: wrote:
query : {
filtered : {
query : {
match_all : { }
},
filter : {
bool : {
must : {
bool : {
must : {
terms : {
isActive : [ true ]
}
}
}
}
}
}
}
},
aggregations : {
revenueFilter : {
filter : {
match_all : { }
},
aggregations : {
revenue : {
range : {
field : revenue,
ranges : [ {
to : 1.0
}, {
from : 1.0,
to : 5.0
}, {
from : 5.0,
to : 50.0
}, {
from : 50.0,
to : 100.0
}, {
from : 100.0,
to : 1000.0
}, {
from : 1000.0
} ]
}
}
}
}
}
}
On Wednesday, 5 November 2014 19:38:42 UTC+5:30, Adrien Grand wrote:
Can you please show the json of the request that you send to
elasticsearch?
On Wed, Nov 5, 2014 at 10:52 AM, Ankur Goel ankr...@gmail.com wrote:
hi ,
we are trying to run some aggregation over around 5 million documents
with cardinality of the fields of the order of 1000 , the aggregation is
a
filter aggregation which wraps underlying term aggregation . Right now
it's taking around 1.2 secs on an average to compute it , the time
increases when no. of documents are increased or I try to do multiple
aggregations. we have aws extra large machines, shards 3 and replication 2
.
1.) can we improve this time (will like it to get it within 1 sec) , I
can see very little if any of field cache being used
2.) how does this scale , it increases with number of documents , how
can I offset that (increasing nodes , replication , sharding ??)
3.) are there any better options (plugins or a different platform for
aggregating data )
regards
Ankur Goel
--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/fb73f5bd-24a4-4065-9253-39aa8dd9dfe0%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fb73f5bd-24a4-4065-9253-39aa8dd9dfe0%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
Adrien Grand
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c83b1ddc-6a4b-4f24-ba3d-f48a8cb108c2%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c83b1ddc-6a4b-4f24-ba3d-f48a8cb108c2%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
Adrien Grand
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e26c7ab9-2923-4e93-bbf6-a74530f3df1d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Elasticsearch Aggregation time
query : {
filtered : {
query : {
match_all : { }
},
filter : {
bool : {
must : {
bool : {
must : {
terms : {
isActive : [ true ]
}
}
}
}
}
}
}
},
aggregations : {
revenue : {
filter : {
match_all : { }
},
aggregations : {
revenueUSD : {
range : {
field : revenueUSD,
ranges : [ {
to : 1.0
}, {
from : 1.0,
to : 5.0
}, {
from : 5.0,
to : 50.0
}, {
from : 50.0,
to : 100.0
}, {
from : 100.0,
to : 1000.0
}, {
from : 1000.0
} ]
}
}
}
}
}
}
this is a sample , the match all is usually replaced by some query
On Wednesday, 5 November 2014 19:38:42 UTC+5:30, Adrien Grand wrote:
Can you please show the json of the request that you send to elasticsearch?
On Wed, Nov 5, 2014 at 10:52 AM, Ankur Goel ankr...@gmail.com
javascript: wrote:
hi ,
we are trying to run some aggregation over around 5 million documents
with cardinality of the fields of the order of 1000 , the aggregation is a
filter aggregation which wraps underlying term aggregation . Right now
it's taking around 1.2 secs on an average to compute it , the time
increases when no. of documents are increased or I try to do multiple
aggregations. we have aws extra large machines, shards 3 and replication 2
.
1.) can we improve this time (will like it to get it within 1 sec) , I
can see very little if any of field cache being used
2.) how does this scale , it increases with number of documents , how can
I offset that (increasing nodes , replication , sharding ??)
3.) are there any better options (plugins or a different platform for
aggregating data )
regards
Ankur Goel
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fb73f5bd-24a4-4065-9253-39aa8dd9dfe0%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fb73f5bd-24a4-4065-9253-39aa8dd9dfe0%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
Adrien Grand
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/deb3e7e4-751a-4d7e-92d5-28be42b11e76%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Elasticsearch Aggregation time
query : {
filtered : {
query : {
match_all : { }
},
filter : {
bool : {
must : {
bool : {
must : {
terms : {
isActive : [ true ]
}
}
}
}
}
}
}
},
aggregations : {
revenueFilter : {
filter : {
match_all : { }
},
aggregations : {
revenue : {
range : {
field : revenue,
ranges : [ {
to : 1.0
}, {
from : 1.0,
to : 5.0
}, {
from : 5.0,
to : 50.0
}, {
from : 50.0,
to : 100.0
}, {
from : 100.0,
to : 1000.0
}, {
from : 1000.0
} ]
}
}
}
}
}
}
On Wednesday, 5 November 2014 19:38:42 UTC+5:30, Adrien Grand wrote:
Can you please show the json of the request that you send to elasticsearch?
On Wed, Nov 5, 2014 at 10:52 AM, Ankur Goel ankr...@gmail.com
javascript: wrote:
hi ,
we are trying to run some aggregation over around 5 million documents
with cardinality of the fields of the order of 1000 , the aggregation is a
filter aggregation which wraps underlying term aggregation . Right now
it's taking around 1.2 secs on an average to compute it , the time
increases when no. of documents are increased or I try to do multiple
aggregations. we have aws extra large machines, shards 3 and replication 2
.
1.) can we improve this time (will like it to get it within 1 sec) , I
can see very little if any of field cache being used
2.) how does this scale , it increases with number of documents , how can
I offset that (increasing nodes , replication , sharding ??)
3.) are there any better options (plugins or a different platform for
aggregating data )
regards
Ankur Goel
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fb73f5bd-24a4-4065-9253-39aa8dd9dfe0%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fb73f5bd-24a4-4065-9253-39aa8dd9dfe0%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
Adrien Grand
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c83b1ddc-6a4b-4f24-ba3d-f48a8cb108c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Elasticsearch Aggregation time
hi , we are trying to run some aggregation over around 5 million documents with cardinality of the fields of the order of 1000 , the aggregation is a filter aggregation which wraps underlying term aggregation . Right now it's taking around 1.2 secs on an average to compute it , the time increases when no. of documents are increased or I try to do multiple aggregations. we have aws extra large machines, shards 3 and replication 2 . 1.) can we improve this time (will like it to get it within 1 sec) , I can see very little if any of field cache being used 2.) how does this scale , it increases with number of documents , how can I offset that (increasing nodes , replication , sharding ??) 3.) are there any better options (plugins or a different platform for aggregating data ) regards Ankur Goel -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fb73f5bd-24a4-4065-9253-39aa8dd9dfe0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
BackUp and restore
Hi All, We have been using an elastic search cluster with 3 nodes , running on aws machines , we have been using EBS for work and data directory of elasticsearch . I was experimenting with backup using EBS snapshots , here is what I did 1.) created a snapshot of one ebs volume (say, alpha) 2.) deleted the index 3.) shutdown the cluster 4.) unmounted the ebs volume on alpha (say /mnt/data ) 5.) created a new volume from snapshot and mounted on alpha in the same location (/mnt/data) 6.) restarted elastic search on one node only cluster went to red state , all shards in unassigned state 7.) restarted another node with blank data (/mnt/data ) directory cluster went to yellow state , all shards STILL in UNASSIGNED STATE 8.) manually allocated shards to node recovered no documents , few kbs of data got restored :( can anyone please help me debug this , what am I doing wrong , is there a better way to do this ?? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6381f2e3-706d-4bc8-86f5-283396e37560%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
ES backup using EBS
Hi All, We have been using an elastic search cluster with 3 nodes , running on aws machines , we have been using EBS for work and data directory of elasticsearch . I was experimenting with backup using EBS snapshots , here is what I did 1.) created a snapshot of one ebs volume (say, alpha) 2.) deleted the index 3.) shutdown the cluster 4.) unmounted the ebs volume on alpha (say /mnt/data ) 5.) created a new volume from snapshot and mounted on alpha in the same location (/mnt/data) 6.) restarted elastic search on one node only cluster went to red state , all shards in unassigned state 7.) restarted another node with blank data (/mnt/data ) directory cluster red state , all shards STILL in UNASSIGNED STATE 8.) manually allocated shards to node recovered no documents , few kbs of data got restored :( can anyone please help me debug this , what am I doing wrong , is there a better way to do this ?? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52a4da1e-7131-4026-b644-4a73b7fe97a4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: cluster reroute and potential data loss
Hi ,
I am also facing the same issue , did you get it resolved ??
I am facing it while doing a ebs snapshot based recovery .
On Tuesday, 4 February 2014 03:46:04 UTC+5:30, Mark Conlin wrote:
So during a cluster restart sometimes we get nodes that have unallocated
shards, both the primary and replica will be unallocated.
They stay stuck in this state, leaving the cluster red.
If I force allocation, with allow_primary=true, I get a new blank shard,
all docs lost.
If I force allocation, with allow_primary=false, I get an error:
{
error:
RemoteTransportException[[yournodename][inet[/10.1.1.1:9300]][cluster/reroute]];
nested: ElasticSearchIllegalArgumentException[[allocate] trying to allocate
a primary shard [yourindexname][4]], which is disabled]; ,
status: 400
}
Once the cluster gets to this state, am I just out of luck on recovering
the data in these shards?
Mark
On Mon, Feb 3, 2014 at 4:57 PM, Nikolas Everett nik...@gmail.comjavascript:
wrote:
If all replicas of a particular shard are unallocated and you
allow_primary allocate one then it'll allocate empty. If a node that had
some data for that shard comes back it won't be able to use that data
because the shard has been allocated empty.
On Mon, Feb 3, 2014 at 4:42 PM, Mark Conlin mark@gmail.comjavascript:
wrote:
I was reading some ES
docohttp://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-reroute.htmland
stumbled upon this part of the Cluster Reroute API:
allocate:
*Allocate an unassigned shard to a node. It also accepts the
allow primary flag to explicitly specify that it is allowed to explicitly
allocate a primary shard (might result in data loss).*
Why might this result in data loss?
If I use:
POST /_cluster/reroute
{
commands : [ {
cancel :
{
index : myindex, shard : 4, node: somenode,
allow_primary:true
}
}
]
}
To get a node that has unallocated shards back to green, how will I know
if data loss has occured?
How/why is the data being lost?
Thanks,
Mark
--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0829b11c-d18a-4f6e-9cf4-67a94fd55daa%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to a topic in the
Google Groups elasticsearch group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/jeaefaiC6d8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1e_y78fPt_N%3DUcTc4Grad_L4fVMLzT%2By3dQ232dsEfEQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/aa1e3bfc-bf61-4f92-b12d-9b8ab414834e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
S3 gateway issues
hi, I am using s3 gateway in a application , elastic search version 1.x , I had a strange exception while starting my nodes , please take a look Error injecting constructor, java.lang.UnsupportedOperationException at org.elasticsearch.gateway.s3.S3Gateway.init(Unknown Source) while locating org.elasticsearch.gateway.s3.S3Gateway while locating org.elasticsearch.gateway.Gateway Caused by: java.lang.UnsupportedOperationException at org.elasticsearch.cluster.metadata.RestoreMetaData$Factory.fromXContent(RestoreMetaData.java:462) at org.elasticsearch.cluster.metadata.RestoreMetaData$Factory.fromXContent(RestoreMetaData.java:400) at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1323) at org.elasticsearch.gateway.blobstore.BlobStoreGateway.readMetaData(BlobStoreGateway.java:213) at org.elasticsearch.gateway.blobstore.BlobStoreGateway.findLatestIndex(BlobStoreGateway.java:198) at org.elasticsearch.gateway.blobstore.BlobStoreGateway.initialize(BlobStoreGateway.java:73) at org.elasticsearch.gateway.s3.S3Gateway.init(S3Gateway.java:97) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54) at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86) at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98) at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52) at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45) at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837) at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42) at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57) at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45) at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200) at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193) at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830) at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193) at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175) at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110) at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93) at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70) at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59) at org.elasticsearch.node.internal.InternalNode.init(InternalNode.java:187) at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159) I am trying to understand what is happening here , the exception looks like it has happened while trying to recover index data but beyond that but I cannot get a clue , please help -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/26712b30-3338-4621-831b-67100a02545a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: S3 gateway issues
Hi David , thanks for replying , I am using version number : 1.0.0, we have AWS plugin, we have removed S3 gateway for now , will switch to local but just wanted to make sure why we are getting this error, It will be really helpful to avoid any surprises in future. On Thursday, 10 April 2014 18:11:48 UTC+5:30, Ankur Goel wrote: hi, I am using s3 gateway in a application , elastic search version 1.x , I had a strange exception while starting my nodes , please take a look Error injecting constructor, java.lang.UnsupportedOperationException at org.elasticsearch.gateway.s3.S3Gateway.init(Unknown Source) while locating org.elasticsearch.gateway.s3.S3Gateway while locating org.elasticsearch.gateway.Gateway Caused by: java.lang.UnsupportedOperationException at org.elasticsearch.cluster.metadata.RestoreMetaData$Factory.fromXContent(RestoreMetaData.java:462) at org.elasticsearch.cluster.metadata.RestoreMetaData$Factory.fromXContent(RestoreMetaData.java:400) at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1323) at org.elasticsearch.gateway.blobstore.BlobStoreGateway.readMetaData(BlobStoreGateway.java:213) at org.elasticsearch.gateway.blobstore.BlobStoreGateway.findLatestIndex(BlobStoreGateway.java:198) at org.elasticsearch.gateway.blobstore.BlobStoreGateway.initialize(BlobStoreGateway.java:73) at org.elasticsearch.gateway.s3.S3Gateway.init(S3Gateway.java:97) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54) at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86) at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98) at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52) at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45) at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837) at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42) at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57) at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45) at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200) at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193) at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830) at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193) at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175) at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110) at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93) at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70) at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59) at org.elasticsearch.node.internal.InternalNode.init(InternalNode.java:187) at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159) I am trying to understand what is happening here , the exception looks like it has happened while trying to recover index data but beyond that but I cannot get a clue , please help -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d493a384-f782-4dc5-81d1-e5a07f7e347f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
