API to get bounded port
When starting a cluster via testing, I want to get the bounded port since I am letting choose and there might be a conflict. Here is the ugly and brittle code I came up to do this: https://gist.github.com/ppearcy/c5d969326b9e6ace8046 Is there a nicer API than having to regex out the connection string? Thanks, Paul -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2a10a6e4-5f01-45f4-a9b8-0be5d35546a0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: plan for river
I see. Thanks for your quick reply. Have a nice day. Thanks, - Mungeol On Monday, October 27, 2014 12:40:12 PM UTC+9, vineeth mohan wrote: Hello Mungeol , As far as i know , the plan is to depreciate rivers and move them to logstash. Thanks Vineeth On Mon, Oct 27, 2014 at 5:19 AM, Mungeol Heo munge...@gmail.com javascript: wrote: Hi, My question is that will es remove all river related plugin in the future? If it will, I'd like to know that is there substitution for JDBC? Thanks. Best regards, - Mungeol -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cc6d541f-1609-4218-932b-064a27e9692a%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/cc6d541f-1609-4218-932b-064a27e9692a%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/db28801f-5fa1-49c9-abcf-448292c62300%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: plan for river
Yes, there is already a substitution, the JDBC feeder in the JDBC river repo. Future versions of JDBC river will no longer rely on the river API. Jörg On Mon, Oct 27, 2014 at 12:49 AM, Mungeol Heo mungeol@gmail.com wrote: Hi, My question is that will es remove all river related plugin in the future? If it will, I'd like to know that is there substitution for JDBC? Thanks. Best regards, - Mungeol -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cc6d541f-1609-4218-932b-064a27e9692a%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/cc6d541f-1609-4218-932b-064a27e9692a%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFk3x%2BTzscHBohYbiHUb-By%2BVv9w5OGx7vj8hj0oE7MRQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Connecting to ES via a http proxy in perl client
I'm trying to connect to my ES via a proxy using a client written in perl.
What's the best way to do this?
Here's what I have, and it works, but I suspect there's a more straight
forward approach:
$e = Search::Elasticsearch-new(
cxn = 'LWP',
nodes = 'node1:9200' );
$ENV{HTTP_proxy} = http://proxy:3128;;
$e-transport-cxn_pool-next_cxn-handle-env_proxy;
--
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/79f08fa4-6bac-4876-8c92-85f3e89a9220%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
What happens to data in an existing type if we update the mapping to specify 'path's for _id and _routing
We are using ES 1.3.2. We have a need to specify custom id and routing values when indexing. We've been doing this using Java APIs, however we would now like to update the mapping to specify 'path's for _id and _routing. The question we have is: 1) Since this type already has a huge number of documents, can we change the mapping? When we tried it, we got a 'acknowledged: true' response, but it doesn't seem to be working when we tried indexing. 2) In case there is a way to achieve this, will it affect only the new documents being indexed? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1c5516bd-7738-4969-8bee-b979aa89b65b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: allow_explicit_index and _bulk
Sorry my friends,
my mistake, after one week I have discovered that the correct syntax is:
POST /foo/bar/_bulk
{ index: {_id : 1} }
{baz: foobar }
It works! Thanks
Niccolo
Il giorno giovedì 23 ottobre 2014 16:39:39 UTC+1, Niccolò Becchi ha scritto:
Thanks Jörg for the prompt answer. However, for security reason we have
disabled the option rest.action.multi.allow_
explicit_index (
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/url-access-control.html
http://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Furl-access-control.htmlsa=Dsntz=1usg=AFQjCNEtPcfFx_JMPzCxDP2x2dyHm8T_GA
).
So, in this case using:
POST /_bulk
{ index : { _index: foo, _type: bar, _id : 1 } }
{ baz: foobar }
I am getting this exception:
{
error: ElasticsearchIllegalArgumentException[explicit index in bulk
is not allowed],
status: 400
}
Is there any way to insert a document, specifying the id, in the bulk mode
with no explicit index in bulk, as this code doesn't work:
POST /foo/bar/_bulk
{ index: {} }
{ _id : 1, baz: foobar }
Else is this a bug/feature to be implemented?
Il giorno giovedì 23 ottobre 2014 10:27:34 UTC+1, Jörg Prante ha scritto:
The bulk format does not allow _id in the payload line, you confuse the
indexer.
Use something like
POST /_bulk
{ index : { _index: foo, _type: bar, _id : 1 } }
{ baz: foobar }
Jörg
On Wed, Oct 22, 2014 at 8:47 PM, Niccolò Becchi niccolo...@gmail.com
wrote:
This issue looks to be fixed on
https://github.com/elasticsearch/elasticsearch/issues/4668
However, on elasticsearch-1.3.4, running the example with
rest.action.multi.allow_explicit_index: false:
```
POST /foo/bar/_bulk
{ index: {} }
{ _id : 1, baz: foobar }
```
I am getting the exception:
```
{
took: 1,
errors: true,
items: [
{
create: {
_index: foo,
_type: bar,
_id: oX0Xp8dzRbySZiKX8QI0zw,
status: 400,
error: MapperParsingException[failed to parse [_id]];
nested: MapperParsingException[Provided id [oX0Xp8dzRbySZiKX8QI0zw] does
not match the content one [1]];
}
}
]
}
```
Am I doing something wrong or something has changed?
Il giorno giovedì 9 gennaio 2014 15:38:46 UTC, Gabe Gorelick-Feldman ha
scritto:
Opened an issue: https://github.com/elasticsearch/elasticsearch/
issues/4668
On Thursday, January 9, 2014 3:39:39 AM UTC-5, Alexander Reelsen wrote:
Hey,
after having a very quick look, it looks like a bug (or wrong
documentation, need to check further). Can you create a github issue?
Thanks!
--Alex
On Wed, Jan 8, 2014 at 11:08 PM, Gabe Gorelick-Feldman
gabego...@gmail.com wrote:
The documentation on URL-based access control
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/url-access-control.html
implies
that _bulk still works if you set
rest.action.multi.allow_explicit_index:
false, as long as you specify the index in the URL. However, I can't
get it to work.
POST /foo/bar/_bulk
{ index: {} }
{ _id: 1234, baz: foobar }
returns
explicit index in bulk is not allowed
Should this work?
--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/a0d1fa2f-0c28-4142-9f6d-4b28a1695bb3%
40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a9aff019-33c0-4743-9e14-fe3913bcda1c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/a9aff019-33c0-4743-9e14-fe3913bcda1c%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b5f5d61c-93c5-4705-9d95-a84ef2a168e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: API to get bounded port
Not sure why you convert the publish address to a string, in Java you can
find the current transport port with
// find node address
NodesInfoRequest nodesInfoRequest = new
NodesInfoRequest().transport(true);
NodesInfoResponse response =
client.admin().cluster().nodesInfo(nodesInfoRequest).actionGet();
Object obj =
response.iterator().next().getTransport().getAddress().publishAddress();
if (obj instanceof InetSocketTransportAddress) {
InetSocketTransportAddress address =
(InetSocketTransportAddress) obj;
// ... process host and port from address
}
if (obj instanceof LocalTransportAddress) {
LocalTransportAddress address = (LocalTransportAddress) obj;
// process local address, no host/port
}
which is, I admit, still not pretty.
Note, if you have a node set to local(true), there is no host/port, but a
LocalTransportAddress.
Jörg
On Mon, Oct 27, 2014 at 7:36 AM, ppearcy ppea...@gmail.com wrote:
When starting a cluster via testing, I want to get the bounded port since
I am letting choose and there might be a conflict.
Here is the ugly and brittle code I came up to do this:
https://gist.github.com/ppearcy/c5d969326b9e6ace8046
Is there a nicer API than having to regex out the connection string?
Thanks,
Paul
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2a10a6e4-5f01-45f4-a9b8-0be5d35546a0%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/2a10a6e4-5f01-45f4-a9b8-0be5d35546a0%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHDHKvokqgZ50AAVK%3D4Mvie73ifs%3DhbkcO%3Dw%2BCEWMtFKQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: Migration of 0.90.3 cluster to new cluster running 1.3.4
This is very strange. I shut down the old cluster while copying the files. For some reason I'm seeing duplicate docs again with ~3.2M docs on the old cluster and ~6.3M docs on the new cluster (using Kopf to compare). Am I missing something obvious? At one point I _think_ I got the document count to match up but obviously I'm not able to reach this state again. On Friday, October 24, 2014 11:42:27 PM UTC+2, Jörg Prante wrote: The plan to move from a 2 node to a 3 node cluster is as follows - backup your old data files (in case you want to go back, once upgraded, there is no way back) - shutdown old cluster - move the data file folder of the old cluster nodes to the new cluster nodes data folders. One node gets no data folder. No rsync required. - check minimum_master_nodes = 2. This is essential for 3 nodes. - start up cluster, all nodes. See the shards rebalancing. No need to worry about primary shards. Jörg On Fri, Oct 24, 2014 at 8:03 PM, Magnus Persson magnus.e...@gmail.com javascript: wrote: Oh, didn't know about optimize so I'll definitely keep that in mind. The reason I was asking about primary shards is that I saw, when starting from a rsync'd datafolder off of one of the nodes, double the amount of documents. It wasn't immediatly apparent but when I later on tried with two rsyncs matching up old node 1 with new node 1 and old node 2 with new node 2 the duplicates went away... and the cluster recovered significantly faster. But reading this, it seems to be sufficient just to rsync the data folder from any 1 node in the old cluster and things will just work? Is there a way to verify the consistency of my cluster? Something like index checksums, or somesuch? On 24 October 2014 17:54, Ivan Brusic iv...@brusic.com javascript: wrote: Unless you are moving to new hardware, there is no need to rsync your data. Both Elasticsaerch 0.90.x and 1.3.x are based on Lucene 4, so the underlying data is compatible. Of course, you should backup your data before such an upgrade. After restarting your new cluster with your old data, I would run an optimize on your indices so that Lucene can upgrade all your segments into the new format. There have been some issues with Lucene format incompatibilities, but they usually deal with indices with beta Lucene versions. You cannot bring up a mixed cluster between 0.90 and 1.x, so you would need to stop all your VMs. Why are you interested in primary shards? Elasticsearch is not like most database where the primary node has an extra special connotation. I have not played around with shard allocation much, but here is an old article: http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/ Cheers, Ivan On Thu, Oct 23, 2014 at 4:18 PM, Magnus Persson magnus.e...@gmail.com javascript: wrote: Ah, slight typo in regard to the old cluster. It is 1 replica per index. On Thursday, October 23, 2014 10:13:57 PM UTC+2, Magnus Persson wrote: So I'm about to upgrade to 1.3.4, but due to some unfortunate circumstances I need to migrate my ES cluster to new VMs. The environment is fairly simple. At the top I have logstash agent pulling messages off of a Redis server and feeding it to my 2 node cluster (2 replicas, 2 shards per index). So for what it's worth I can stop logstash and the cluster will essentially stop indexing data, allowing me to shut it down without issue. Once I have the old cluster shut down, I intend to rsync it over to the new cluster which is 3 nodes (2 replicas, 3 shards per index). What is the best approach here? I was thinking that I could rsync the data folder from 1 of my 2 VMs running on the old cluster but then I realized that the primary shard for each index might not be on that VM. Can I manually set the primary shard somehow? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ee5aa6d1-3339-4d45-8cd6-76614269e501%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/ee5aa6d1-3339-4d45-8cd6-76614269e501%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/8MWsKqDIKpA/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQB_R8bj9mNSASWJVpGZwR5JYJSdu6bk_5DvzxPgtbU-Bg%40mail.gmail.com
indexing and searching for string '???'
Hello,
when trying a match query on a string field to match the string '???' i am
getting nothing back from elasticsearch.
It seems like the standard analyzer is just stripping this string out when
tokenizing. probably because its treating a ? as a end of word and
filtering it out?
when doing _analyze?analyzer=standardpretty' -d 'this is a ??? test'
I get back the response below which seems to confirm that. Is there any
way where I could still be filtering out ? at the end of words, but if
there are multiple '??' it doesn't strip them?
{
tokens : [ {
token : this,
start_offset : 0,
end_offset : 4,
type : ALPHANUM,
position : 1
}, {
token : is,
start_offset : 5,
end_offset : 7,
type : ALPHANUM,
position : 2
}, {
token : a,
start_offset : 8,
end_offset : 9,
type : ALPHANUM,
position : 3
}, {
token : test,
start_offset : 15,
end_offset : 19,
type : ALPHANUM,
position : 4
} ]
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALdNed%2BCGeR_92B%3DH%2BnS3FY%3DuiXH0Q6ShJV_Jg_awbQ2bH3sbQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
analyzer settings for breaking up words on hyphens
Hello, I have a field that is using the whitespace tokenizer, but I also want to tokenize on hyphens (-) like the standard analyzer does. I'm having trouble figuring out what additional custom settings I would have to put in there in order to be able to tokenize off of hyphens as well. Thanks, Mike -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALdNedLtdAWEiQN%2BoUV17J5e8DowMbDva2pJn1S%3Dr9w1qtP9bA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: indexing and searching for string '???'
Exactly default standard analyzer is meant for text analysis. ??? is not a word
either a number so it's removed.
If you need to analyze that, you should try to use another analyzer like:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-whitespace-analyzer.html#analysis-whitespace-analyzer
--
David Pilato | Technical Advocate | elasticsearch.com
david.pil...@elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs
Le 27 octobre 2014 à 12:53:01, Mike Topper (top...@gmail.com) a écrit:
Hello,
when trying a match query on a string field to match the string '???' i am
getting nothing back from elasticsearch.
It seems like the standard analyzer is just stripping this string out when
tokenizing. probably because its treating a ? as a end of word and filtering it
out?
when doing _analyze?analyzer=standardpretty' -d 'this is a ??? test'
I get back the response below which seems to confirm that. Is there any way
where I could still be filtering out ? at the end of words, but if there are
multiple '??' it doesn't strip them?
{
tokens : [ {
token : this,
start_offset : 0,
end_offset : 4,
type : ALPHANUM,
position : 1
}, {
token : is,
start_offset : 5,
end_offset : 7,
type : ALPHANUM,
position : 2
}, {
token : a,
start_offset : 8,
end_offset : 9,
type : ALPHANUM,
position : 3
}, {
token : test,
start_offset : 15,
end_offset : 19,
type : ALPHANUM,
position : 4
} ]
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALdNed%2BCGeR_92B%3DH%2BnS3FY%3DuiXH0Q6ShJV_Jg_awbQ2bH3sbQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.544e3533.216231b.91d6%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.
Re: Failed to start shard
Hi Mohamed, Did you find a solution to this problem? On Wednesday, July 23, 2014 10:06:16 AM UTC+5:30, Mohamed Lrhazi wrote: I went ahead and upgraded to elasticsearch-1.2.2-1.noarch, but the same errors continues after cluster start. I notice that the node inquestion is the master, and is supposed to be a non data node. it should not be starting any shards, right? Thanks a lot, Mohamed. On Tue, Jul 22, 2014 at 11:55 PM, Mohamed Lrhazi ml...@georgetown.edu javascript: wrote: Hello, Our master is constantly logging this error, about twice per second I saw similar error mentioned in the release notes for a previous verion, this cluster is running 1.0.1. Is it the same issue? What work arounds do we have? [2014-07-22 23:50:28,533][WARN ][cluster.action.shard ] [rap-es2] [de_2007][0] received shard failed for [de_2007][0], node[EtDIFKT7Qi-nAuvG47tGYQ], [P], s[INITIALIZING], indexUUID [yLrHy5k7RqO6rLnl_b4kww], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[de_2007][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[de_2007][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/data/elasticsearch/data/Odine2/nodes/0/indices/de_2007/0/index), type=MERGE, rate=20.0)]): files: []]; ]] -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/OVc_0Kqm2yk/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52f5a6c8-847c-4294-a2eb-308eb9aa3fe7%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Migration of 0.90.3 cluster to new cluster running 1.3.4
https://gist.github.com/magnusp/515a5c3debed12802d1f is the configuration im running on the new cluster. The old cluster is the default that came with 0.90.3 (replicas and shards were set via templates I guess) On Monday, October 27, 2014 12:37:48 PM UTC+1, Magnus Persson wrote: This is very strange. I shut down the old cluster while copying the files. For some reason I'm seeing duplicate docs again with ~3.2M docs on the old cluster and ~6.3M docs on the new cluster (using Kopf to compare). Am I missing something obvious? At one point I _think_ I got the document count to match up but obviously I'm not able to reach this state again. On Friday, October 24, 2014 11:42:27 PM UTC+2, Jörg Prante wrote: The plan to move from a 2 node to a 3 node cluster is as follows - backup your old data files (in case you want to go back, once upgraded, there is no way back) - shutdown old cluster - move the data file folder of the old cluster nodes to the new cluster nodes data folders. One node gets no data folder. No rsync required. - check minimum_master_nodes = 2. This is essential for 3 nodes. - start up cluster, all nodes. See the shards rebalancing. No need to worry about primary shards. Jörg On Fri, Oct 24, 2014 at 8:03 PM, Magnus Persson magnus.e...@gmail.com wrote: Oh, didn't know about optimize so I'll definitely keep that in mind. The reason I was asking about primary shards is that I saw, when starting from a rsync'd datafolder off of one of the nodes, double the amount of documents. It wasn't immediatly apparent but when I later on tried with two rsyncs matching up old node 1 with new node 1 and old node 2 with new node 2 the duplicates went away... and the cluster recovered significantly faster. But reading this, it seems to be sufficient just to rsync the data folder from any 1 node in the old cluster and things will just work? Is there a way to verify the consistency of my cluster? Something like index checksums, or somesuch? On 24 October 2014 17:54, Ivan Brusic iv...@brusic.com wrote: Unless you are moving to new hardware, there is no need to rsync your data. Both Elasticsaerch 0.90.x and 1.3.x are based on Lucene 4, so the underlying data is compatible. Of course, you should backup your data before such an upgrade. After restarting your new cluster with your old data, I would run an optimize on your indices so that Lucene can upgrade all your segments into the new format. There have been some issues with Lucene format incompatibilities, but they usually deal with indices with beta Lucene versions. You cannot bring up a mixed cluster between 0.90 and 1.x, so you would need to stop all your VMs. Why are you interested in primary shards? Elasticsearch is not like most database where the primary node has an extra special connotation. I have not played around with shard allocation much, but here is an old article: http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/ Cheers, Ivan On Thu, Oct 23, 2014 at 4:18 PM, Magnus Persson magnus.e...@gmail.com wrote: Ah, slight typo in regard to the old cluster. It is 1 replica per index. On Thursday, October 23, 2014 10:13:57 PM UTC+2, Magnus Persson wrote: So I'm about to upgrade to 1.3.4, but due to some unfortunate circumstances I need to migrate my ES cluster to new VMs. The environment is fairly simple. At the top I have logstash agent pulling messages off of a Redis server and feeding it to my 2 node cluster (2 replicas, 2 shards per index). So for what it's worth I can stop logstash and the cluster will essentially stop indexing data, allowing me to shut it down without issue. Once I have the old cluster shut down, I intend to rsync it over to the new cluster which is 3 nodes (2 replicas, 3 shards per index). What is the best approach here? I was thinking that I could rsync the data folder from 1 of my 2 VMs running on the old cluster but then I realized that the primary shard for each index might not be on that VM. Can I manually set the primary shard somehow? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ee5aa6d1-3339-4d45-8cd6-76614269e501%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/ee5aa6d1-3339-4d45-8cd6-76614269e501%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/8MWsKqDIKpA/unsubscribe . To unsubscribe from this group and all
histogram aggregation with float interval 1.0 gives ArithmeticException
Hi, the following query (note bolded line):
{
query: {
filtered: {
query: {
term: {
self_and_ancestors: diamonds
}
},
filter: {
terms: {
attr_types.diamond#color: [
d
]
}
}
}
},
sort: [
{
sell_offer_cents: {
order: asc
}
}
],
fields: _source,
script_fields: {
gap_cents: {
script: custom_score_item_bid_ask_gap,
params: {
individual_price_item_ids: [],
individual_price_item_cents: [],
pb_amount_below_cents: 0
},
lang: native
}
},
aggs: {
all_items: {
global: {},
aggs: {
gem#carats: {
filter: {
terms: {
attr_types.diamond#polish: [
ex,
0001vg
]
}
},
aggs: {
gems#carats: {
histogram: {
field: attr_types.gem#carats,
* interval: 0.1,*
min_doc_count: 0
}
},
gem#carats_stats: {
stats: {
field: attr_types.gem#carats
}
}
}
}
}
}
}
}
Gives the following error:
{
error: SearchPhaseExecutionException[Failed to execute phase [query],
all shards failed; shardFailures
{[D6h8GKwjRqO_Yem09Hu_VA][development-liquidibles::application-items][4]:
QueryPhaseExecutionException[[development-liquidibles::application-items][4]:
query[filtered(filtered(self_and_ancestors:diamonds)-cache(attr_types.diamond#color:d))-cache(_type:item)],from[0],size[10],sort[custom:\sell_offer_cents\:
org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@11ce49a]:
Query Failed [Failed to execute global aggregators]]; nested:
ArithmeticException;
}{[D6h8GKwjRqO_Yem09Hu_VA][development-liquidibles::application-items][3]:
QueryPhaseExecutionException[[development-liquidibles::application-items][3]:
query[filtered(filtered(self_and_ancestors:diamonds)-cache(attr_types.diamond#color:d))-cache(_type:item)],from[0],size[10],sort[custom:\sell_offer_cents\:
org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@8c9d82]:
Query Failed [Failed to execute global aggregators]]; nested:
ArithmeticException;
}{[D6h8GKwjRqO_Yem09Hu_VA][development-liquidibles::application-items][2]:
QueryPhaseExecutionException[[development-liquidibles::application-items][2]:
query[filtered(filtered(self_and_ancestors:diamonds)-cache(attr_types.diamond#color:d))-cache(_type:item)],from[0],size[10],sort[custom:\sell_offer_cents\:
org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@73a7e5]:
Query Failed [Failed to execute global aggregators]]; nested:
ArithmeticException;
}{[D6h8GKwjRqO_Yem09Hu_VA][development-liquidibles::application-items][1]:
QueryPhaseExecutionException[[development-liquidibles::application-items][1]:
query[filtered(filtered(self_and_ancestors:diamonds)-cache(attr_types.diamond#color:d))-cache(_type:item)],from[0],size[10],sort[custom:\sell_offer_cents\:
org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@15d1b1a]:
Query Failed [Failed to execute global aggregators]]; nested:
ArithmeticException;
}{[D6h8GKwjRqO_Yem09Hu_VA][development-liquidibles::application-items][0]:
QueryPhaseExecutionException[[development-liquidibles::application-items][0]:
query[filtered(filtered(self_and_ancestors:diamonds)-cache(attr_types.diamond#color:d))-cache(_type:item)],from[0],size[10],sort[custom:\sell_offer_cents\:
org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@1b8c216]:
Query Failed [Failed to execute global aggregators]]; nested:
*ArithmeticException*; }],
status: 500
}
If I change the interval to be 1.0 or greater, it works. But, I want
intervals of 0.1...
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b8dad371-dfef-4c57-b7d8-433ee1c308c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Customizing Directory and IndexWriter behavior via custom ES plug-in
Hi there, in the last couple of years we managed to customize Apache Lucene (through its public API) to support branching, tagging and compare in a concurrent fashion for our server application. We managed to achieve this by using a couple of custom Directory, exactly one IndexDeletionPolicy and one MergePolicy implementations. Currently we are considering to replace Lucene with Elasticsearch on the server-side. Before we jumped into the details of collecting the differences between the two technologies in respect of the search and indexing functionality and for instance how to port our custom collectors and how to replace NDVs, we would like to make sure if it is possible at all. I've just checked out the source and realized that the registration of the services are done via various module implementations and the actual configured service implementations are injected into the constructors. For the sake of simplicity is there a way for example to create an Elasticsearch module which forces the underlying IndexWriter to use the FooCustomDeletionPolicy instead of the default KeepOnlyLastDeletionPolicy? I assume if this is straightforward we could use or custom implementations for the directory and the IndexWriter what we are currently using with pure Lucene. After doing some research I found this [1] thread. Am I close to the answer/solution? I have to notice we would like to achieve this without forking the public repository. Thanks in advance for the feedbacks. Cheers, Akos [1]: https://groups.google.com/forum/#!topic/elasticsearch/rFaLnI5FRf4 -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0345efea-3134-488d-b13d-199a24642422%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: nested aggregation against key value pairs
Here is some sample data:
PUT index1
PUT index1/type1/_mapping
{
type1: {
properties: {
kvp: {
type: nested,
properties: {
key: {
type: string
},
value: {
type: string
}
}
}
}
}
}
POST index1/type1
{
kvp: [
{
key: key1,
value: value1
}
]
}
POST index1/type1
{
kvp: [
{
key: key1,
value: value1
},
{
key: key1,
value: value2
},
{
key: key2,
value: value1
},
{
key: key2,
value: value2
}
]
}
The result I'd like combines the fields kvp.key and kvp.value:
key1 - value1: DocCount = 2
key1 - value2: DocCount = 1
key2 - value2: DocCount = 1
I'm starting to think that I need to re-index the data and combine the
kvp.key and kvp.value fields into a single field so that I can
aggregate on it.
On Friday, October 24, 2014 2:17:04 PM UTC-5, Jay Hilden wrote:
I have an ES type with a nested KeyValuePair type. What I'm trying to do
is a terms aggregation on both the key and value fields such that I'd get
the following results:
Key1 - Value1: DocCount = 10
Key1 - Value2: DocCount = 9
Key2 - Value3: DocCount = 4
Here is my mapping:
{
index123 : {
mappings : {
type123 : {
properties : {
authEventID : {
type : long
},
authInput : {
properties : {
uIDExtensionFields : {
type : nested,
properties : {
key : {
type : string
},
value : {
type : string
}
}
}
}
}
}
}
}
}
}
Is there a way to do this?
Thank you.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d9db4ea8-68af-4cc5-a6dc-876f218b58f7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Failed to start shard
Sorry dont recall! but I find this in an email sent on the same thread think: For me the change that seemed to have worked, I keep my fingers crossed, is to add node.master=False to all nodes except one... black magic! Other things we did involve: Adding more nodes and deleting indexes with zero documents. HTH, Mohamed. On Mon, Oct 27, 2014 at 8:12 AM, Anirudh Perugu anir...@unbxd.com wrote: Hi Mohamed, Did you find a solution to this problem? On Wednesday, July 23, 2014 10:06:16 AM UTC+5:30, Mohamed Lrhazi wrote: I went ahead and upgraded to elasticsearch-1.2.2-1.noarch, but the same errors continues after cluster start. I notice that the node inquestion is the master, and is supposed to be a non data node. it should not be starting any shards, right? Thanks a lot, Mohamed. On Tue, Jul 22, 2014 at 11:55 PM, Mohamed Lrhazi ml...@georgetown.edu wrote: Hello, Our master is constantly logging this error, about twice per second I saw similar error mentioned in the release notes for a previous verion, this cluster is running 1.0.1. Is it the same issue? What work arounds do we have? [2014-07-22 23:50:28,533][WARN ][cluster.action.shard ] [rap-es2] [de_2007][0] received shard failed for [de_2007][0], node[EtDIFKT7Qi-nAuvG47tGYQ], [P], s[INITIALIZING], indexUUID [yLrHy5k7RqO6rLnl_b4kww], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[de_2007][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[de_2007][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/data/elasticsearch/data/Odine2/nodes/0/indices/de_2007/0/index), type=MERGE, rate=20.0)]): files: []]; ]] -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/ topic/elasticsearch/OVc_0Kqm2yk/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearc...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a% 40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/OVc_0Kqm2yk/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52f5a6c8-847c-4294-a2eb-308eb9aa3fe7%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/52f5a6c8-847c-4294-a2eb-308eb9aa3fe7%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEU_gmdtPD8Cxg%2BJmqJhtZRgMv%3DQ8SNo7qds1GT%2BOdegkGKQ-w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Customizing Directory and IndexWriter behavior via custom ES plug-in
I stand corrected - there is also the possibility of the setting index.store.type, by setting this to a Java class name, you can use this as the current index store implementation from a plugin. So, no patching/forking required. Jörg On Mon, Oct 27, 2014 at 2:26 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Regarding the deletion policy, you can set the class name of your deletion policy implementation in the setting index.deletionpolicy.type For custom Directory, you have to patch org.elasticsearch.index.store.IndexStoreModule with your custom index store. The index store is something like an IndexWriter / Lucene Directory on steroids. At the moment, it is not possible to add custom index stores from a plugin (see the fixed enumeration of implementations in IndexStoreModule) Jörg On Mon, Oct 27, 2014 at 1:22 PM, Ákos Kitta kittaa...@gmail.com wrote: Hi there, in the last couple of years we managed to customize Apache Lucene (through its public API) to support branching, tagging and compare in a concurrent fashion for our server application. We managed to achieve this by using a couple of custom Directory, exactly one IndexDeletionPolicy and one MergePolicy implementations. Currently we are considering to replace Lucene with Elasticsearch on the server-side. Before we jumped into the details of collecting the differences between the two technologies in respect of the search and indexing functionality and for instance how to port our custom collectors and how to replace NDVs, we would like to make sure if it is possible at all. I've just checked out the source and realized that the registration of the services are done via various module implementations and the actual configured service implementations are injected into the constructors. For the sake of simplicity is there a way for example to create an Elasticsearch module which forces the underlying IndexWriter to use the FooCustomDeletionPolicy instead of the default KeepOnlyLastDeletionPolicy? I assume if this is straightforward we could use or custom implementations for the directory and the IndexWriter what we are currently using with pure Lucene. After doing some research I found this [1] thread. Am I close to the answer/solution? I have to notice we would like to achieve this without forking the public repository. Thanks in advance for the feedbacks. Cheers, Akos [1]: https://groups.google.com/forum/#!topic/elasticsearch/rFaLnI5FRf4 -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0345efea-3134-488d-b13d-199a24642422%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/0345efea-3134-488d-b13d-199a24642422%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFQHUNDA6aER6jHkEFcsHS8PEpCPcszq%2BNGqbp_os%2B9_Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: ES 1.3.4: sometimes plugins return empty page
Anyone an idea regarding to this issue? Seems an ungraceful ES shutdown affects the installed ES plugins ... Some days ago I updated my 2-node ES Cluster to 1.3.4. Since the update the installed plugins sometimes are not applicable and for http://localhost:9200/_plugin/head returns an empty page. This effect is seen very often after restarting a node, e.g. after SHUTDOWN or simply killing the server process. After restart the ES cluster itself runs an is applicable to ES java client, but none of the installed plugins is applicable. Looks like the ES server does not deliver the plugins web resources. When opening the index.html from head-plugin from filesystem, the plugin runs fine. Probably an issue with 1.3.4? My experience with earlier releases indicated a very fault tolerant cluster, but with current release there are much issues to plugin availablity. Any ideas? Markus -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7f343b0e-d423-43a6-95d4-1a0eea008e9f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: completion suggester caching
I ran into same caching problem with latest ES 1.3.4.
I updated documents which just hold completion field to use with completion
suggester, even if i deleted docs via id and they can't be found via
_search endpoint anymore i still get suggests for those deleted documents.
I use a cluster with 2 nodes. Any suggestions?!
Am Freitag, 7. Februar 2014 16:04:01 UTC+1 schrieb Jorge Sanchez:
Hello,
I am using completion suggester for user search on my website. I basically
followed the howto from elasticsearch webpage.
Created the mapping:
post /gruppu/user/_mapping
{
user : {
properties : {
name : { type : string },
suggest : { type : completion,
index_analyzer : simple,
search_analyzer : simple,
payloads : true
}
}
}
}
And I put something to the ES:
post /gruppu/user/10001?refresh=true
{
name : jorge,
suggest : {
input: [ jjorge],
payload : { userid : 10001, grouporuser: group }
}
}'
*The problem is that when I delete or update above document, the
suggester still seems to cache the data. My question is what is the correct
way to update the documents in ES to have the suggester working correctly? *
For example the above document, after updating the document and changing
the input , when I search for it I get the latest version:
{_index:gruppu,_type:user,_id:10001,_version:4,exists:true,
_source : {
name : jorge,
suggest : {
input: [ *jorge group*],
payload : { userid : 10001, grouporuser: group }
}
}'
}
When I run the suggester I get following:
user-suggest: [
{
text: j,
offset: 0,
length: 1,
options: [
{
text: jorge group,
score: 2,
payload: {
userid: 10001,
grouporuser: group
}
},
{
text: *jjorge*,
score: 1,
payload: {
userid: 10001,
grouporuser: group
}
},
So what is the correct way of updating the ES documents when using the
suggester ?
Thanks,
Jorge Sanchez
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f4bcada2-f0a4-4ea3-a2d7-5de54920fda1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: completion suggester caching
Addendum: And against a 1.3.2 cluster with 2 nodes it works as expected,
but not in 1.3.4 ?
Am Montag, 27. Oktober 2014 15:09:10 UTC+1 schrieb Tom:
I ran into same caching problem with latest ES 1.3.4.
I updated documents which just hold completion field to use with
completion suggester, even if i deleted docs via id and they can't be found
via _search endpoint anymore i still get suggests for those deleted
documents.
I use a cluster with 2 nodes. Any suggestions?!
Am Freitag, 7. Februar 2014 16:04:01 UTC+1 schrieb Jorge Sanchez:
Hello,
I am using completion suggester for user search on my website. I
basically followed the howto from elasticsearch webpage.
Created the mapping:
post /gruppu/user/_mapping
{
user : {
properties : {
name : { type : string },
suggest : { type : completion,
index_analyzer : simple,
search_analyzer : simple,
payloads : true
}
}
}
}
And I put something to the ES:
post /gruppu/user/10001?refresh=true
{
name : jorge,
suggest : {
input: [ jjorge],
payload : { userid : 10001, grouporuser: group }
}
}'
*The problem is that when I delete or update above document, the
suggester still seems to cache the data. My question is what is the correct
way to update the documents in ES to have the suggester working correctly? *
For example the above document, after updating the document and changing
the input , when I search for it I get the latest version:
{_index:gruppu,_type:user,_id:10001,_version:4,exists:true,
_source : {
name : jorge,
suggest : {
input: [ *jorge group*],
payload : { userid : 10001, grouporuser: group }
}
}'
}
When I run the suggester I get following:
user-suggest: [
{
text: j,
offset: 0,
length: 1,
options: [
{
text: jorge group,
score: 2,
payload: {
userid: 10001,
grouporuser: group
}
},
{
text: *jjorge*,
score: 1,
payload: {
userid: 10001,
grouporuser: group
}
},
So what is the correct way of updating the ES documents when using the
suggester ?
Thanks,
Jorge Sanchez
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/54e3a018-c903-47dd-9e4a-4e420ea37964%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Migration of 0.90.3 cluster to new cluster running 1.3.4
When using the count API the document count seems to more reasonably match up. Might possibly be that Kopf is counting documents differently on 0.90 than on 1.3.. seems far fetched though. On Monday, October 27, 2014 1:16:40 PM UTC+1, Magnus Persson wrote: https://gist.github.com/magnusp/515a5c3debed12802d1f is the configuration im running on the new cluster. The old cluster is the default that came with 0.90.3 (replicas and shards were set via templates I guess) On Monday, October 27, 2014 12:37:48 PM UTC+1, Magnus Persson wrote: This is very strange. I shut down the old cluster while copying the files. For some reason I'm seeing duplicate docs again with ~3.2M docs on the old cluster and ~6.3M docs on the new cluster (using Kopf to compare). Am I missing something obvious? At one point I _think_ I got the document count to match up but obviously I'm not able to reach this state again. On Friday, October 24, 2014 11:42:27 PM UTC+2, Jörg Prante wrote: The plan to move from a 2 node to a 3 node cluster is as follows - backup your old data files (in case you want to go back, once upgraded, there is no way back) - shutdown old cluster - move the data file folder of the old cluster nodes to the new cluster nodes data folders. One node gets no data folder. No rsync required. - check minimum_master_nodes = 2. This is essential for 3 nodes. - start up cluster, all nodes. See the shards rebalancing. No need to worry about primary shards. Jörg On Fri, Oct 24, 2014 at 8:03 PM, Magnus Persson magnus.e...@gmail.com wrote: Oh, didn't know about optimize so I'll definitely keep that in mind. The reason I was asking about primary shards is that I saw, when starting from a rsync'd datafolder off of one of the nodes, double the amount of documents. It wasn't immediatly apparent but when I later on tried with two rsyncs matching up old node 1 with new node 1 and old node 2 with new node 2 the duplicates went away... and the cluster recovered significantly faster. But reading this, it seems to be sufficient just to rsync the data folder from any 1 node in the old cluster and things will just work? Is there a way to verify the consistency of my cluster? Something like index checksums, or somesuch? On 24 October 2014 17:54, Ivan Brusic iv...@brusic.com wrote: Unless you are moving to new hardware, there is no need to rsync your data. Both Elasticsaerch 0.90.x and 1.3.x are based on Lucene 4, so the underlying data is compatible. Of course, you should backup your data before such an upgrade. After restarting your new cluster with your old data, I would run an optimize on your indices so that Lucene can upgrade all your segments into the new format. There have been some issues with Lucene format incompatibilities, but they usually deal with indices with beta Lucene versions. You cannot bring up a mixed cluster between 0.90 and 1.x, so you would need to stop all your VMs. Why are you interested in primary shards? Elasticsearch is not like most database where the primary node has an extra special connotation. I have not played around with shard allocation much, but here is an old article: http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/ Cheers, Ivan On Thu, Oct 23, 2014 at 4:18 PM, Magnus Persson magnus.e...@gmail.com wrote: Ah, slight typo in regard to the old cluster. It is 1 replica per index. On Thursday, October 23, 2014 10:13:57 PM UTC+2, Magnus Persson wrote: So I'm about to upgrade to 1.3.4, but due to some unfortunate circumstances I need to migrate my ES cluster to new VMs. The environment is fairly simple. At the top I have logstash agent pulling messages off of a Redis server and feeding it to my 2 node cluster (2 replicas, 2 shards per index). So for what it's worth I can stop logstash and the cluster will essentially stop indexing data, allowing me to shut it down without issue. Once I have the old cluster shut down, I intend to rsync it over to the new cluster which is 3 nodes (2 replicas, 3 shards per index). What is the best approach here? I was thinking that I could rsync the data folder from 1 of my 2 VMs running on the old cluster but then I realized that the primary shard for each index might not be on that VM. Can I manually set the primary shard somehow? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ee5aa6d1-3339-4d45-8cd6-76614269e501%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/ee5aa6d1-3339-4d45-8cd6-76614269e501%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout.
How long do logs stay in Elasticsearch?
I was wondering how long logs or events stay in Elasticsearch? Is there a way we can change the default value? Thanks, Aaron Lee -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5bf81d3e-2a63-4197-8cfd-65af52fbc2b8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: completion suggester caching
Answer: On the failing cluster we had accidently same node names for two
nodes. After fixing that everything is fine.
I am wondering: Shouldn't ES come up with some kind of warning when same
node name is used twice in a cluster or do names haven't to be unique
actually?
Am Montag, 27. Oktober 2014 15:13:30 UTC+1 schrieb Tom:
Addendum: And against a 1.3.2 cluster with 2 nodes it works as expected,
but not in 1.3.4 ?
Am Montag, 27. Oktober 2014 15:09:10 UTC+1 schrieb Tom:
I ran into same caching problem with latest ES 1.3.4.
I updated documents which just hold completion field to use with
completion suggester, even if i deleted docs via id and they can't be found
via _search endpoint anymore i still get suggests for those deleted
documents.
I use a cluster with 2 nodes. Any suggestions?!
Am Freitag, 7. Februar 2014 16:04:01 UTC+1 schrieb Jorge Sanchez:
Hello,
I am using completion suggester for user search on my website. I
basically followed the howto from elasticsearch webpage.
Created the mapping:
post /gruppu/user/_mapping
{
user : {
properties : {
name : { type : string },
suggest : { type : completion,
index_analyzer : simple,
search_analyzer : simple,
payloads : true
}
}
}
}
And I put something to the ES:
post /gruppu/user/10001?refresh=true
{
name : jorge,
suggest : {
input: [ jjorge],
payload : { userid : 10001, grouporuser: group }
}
}'
*The problem is that when I delete or update above document, the
suggester still seems to cache the data. My question is what is the correct
way to update the documents in ES to have the suggester working correctly? *
For example the above document, after updating the document and changing
the input , when I search for it I get the latest version:
{_index:gruppu,_type:user,_id:10001,_version:4,exists:true,
_source : {
name : jorge,
suggest : {
input: [ *jorge group*],
payload : { userid : 10001, grouporuser: group }
}
}'
}
When I run the suggester I get following:
user-suggest: [
{
text: j,
offset: 0,
length: 1,
options: [
{
text: jorge group,
score: 2,
payload: {
userid: 10001,
grouporuser: group
}
},
{
text: *jjorge*,
score: 1,
payload: {
userid: 10001,
grouporuser: group
}
},
So what is the correct way of updating the ES documents when using the
suggester ?
Thanks,
Jorge Sanchez
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0209a805-5150-4564-b53e-88d113b14348%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Migration of 0.90.3 cluster to new cluster running 1.3.4
This was confirmed as a bug in Kopf master (https://github.com/lmenezes/elasticsearch-kopf/issues/190) On Monday, October 27, 2014 3:21:24 PM UTC+1, Magnus Persson wrote: When using the count API the document count seems to more reasonably match up. Might possibly be that Kopf is counting documents differently on 0.90 than on 1.3.. seems far fetched though. On Monday, October 27, 2014 1:16:40 PM UTC+1, Magnus Persson wrote: https://gist.github.com/magnusp/515a5c3debed12802d1f is the configuration im running on the new cluster. The old cluster is the default that came with 0.90.3 (replicas and shards were set via templates I guess) On Monday, October 27, 2014 12:37:48 PM UTC+1, Magnus Persson wrote: This is very strange. I shut down the old cluster while copying the files. For some reason I'm seeing duplicate docs again with ~3.2M docs on the old cluster and ~6.3M docs on the new cluster (using Kopf to compare). Am I missing something obvious? At one point I _think_ I got the document count to match up but obviously I'm not able to reach this state again. On Friday, October 24, 2014 11:42:27 PM UTC+2, Jörg Prante wrote: The plan to move from a 2 node to a 3 node cluster is as follows - backup your old data files (in case you want to go back, once upgraded, there is no way back) - shutdown old cluster - move the data file folder of the old cluster nodes to the new cluster nodes data folders. One node gets no data folder. No rsync required. - check minimum_master_nodes = 2. This is essential for 3 nodes. - start up cluster, all nodes. See the shards rebalancing. No need to worry about primary shards. Jörg On Fri, Oct 24, 2014 at 8:03 PM, Magnus Persson magnus.e...@gmail.com wrote: Oh, didn't know about optimize so I'll definitely keep that in mind. The reason I was asking about primary shards is that I saw, when starting from a rsync'd datafolder off of one of the nodes, double the amount of documents. It wasn't immediatly apparent but when I later on tried with two rsyncs matching up old node 1 with new node 1 and old node 2 with new node 2 the duplicates went away... and the cluster recovered significantly faster. But reading this, it seems to be sufficient just to rsync the data folder from any 1 node in the old cluster and things will just work? Is there a way to verify the consistency of my cluster? Something like index checksums, or somesuch? On 24 October 2014 17:54, Ivan Brusic iv...@brusic.com wrote: Unless you are moving to new hardware, there is no need to rsync your data. Both Elasticsaerch 0.90.x and 1.3.x are based on Lucene 4, so the underlying data is compatible. Of course, you should backup your data before such an upgrade. After restarting your new cluster with your old data, I would run an optimize on your indices so that Lucene can upgrade all your segments into the new format. There have been some issues with Lucene format incompatibilities, but they usually deal with indices with beta Lucene versions. You cannot bring up a mixed cluster between 0.90 and 1.x, so you would need to stop all your VMs. Why are you interested in primary shards? Elasticsearch is not like most database where the primary node has an extra special connotation. I have not played around with shard allocation much, but here is an old article: http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/ Cheers, Ivan On Thu, Oct 23, 2014 at 4:18 PM, Magnus Persson magnus.e...@gmail.com wrote: Ah, slight typo in regard to the old cluster. It is 1 replica per index. On Thursday, October 23, 2014 10:13:57 PM UTC+2, Magnus Persson wrote: So I'm about to upgrade to 1.3.4, but due to some unfortunate circumstances I need to migrate my ES cluster to new VMs. The environment is fairly simple. At the top I have logstash agent pulling messages off of a Redis server and feeding it to my 2 node cluster (2 replicas, 2 shards per index). So for what it's worth I can stop logstash and the cluster will essentially stop indexing data, allowing me to shut it down without issue. Once I have the old cluster shut down, I intend to rsync it over to the new cluster which is 3 nodes (2 replicas, 3 shards per index). What is the best approach here? I was thinking that I could rsync the data folder from 1 of my 2 VMs running on the old cluster but then I realized that the primary shard for each index might not be on that VM. Can I manually set the primary shard somehow? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ee5aa6d1-3339-4d45-8cd6-76614269e501%40googlegroups.com
Requirement on web application administrator
*Please send Resume to suni...@sreeinfotech.com suni...@sreeinfotech.com* *Position:Web Sphere* *Location: Columbus, Ohio http://en.wikipedia.org/wiki/Columbus,_Ohio* *Duration:6 months* *Top Three Skills:* * Websphere, JBoss, Linux* *Job Description:* We are seeking a web application administrator to design, monitor and enhance various web applications platforms . This individual will need excellent organization, communication and planning skills to assist a team of developers with application rollouts. This individual must have experience with the installation and configuration of tomcat and apache web servers. Ability to manage time effectively and prioritize task for multiple projects is a must. *Technical Requirements*: - 5 years experience with Tomcat or Jboss web servers - Strong Linux skill is a must - Troubleshooting and problem solving with clustered web environments - Experience with websphere application server 7 or 8 - Experience with websphere portal is a plus - Shell Scripting - Experience with F5 load balancing solutions - Demonstrator knowledge of Linux RHEL 5 - Familiarity with application monitoring - Familiarity with WCM content management - Familiarity with SSO authentication products such as WebSEAL Bachelor's Degree in Computer Science or related field preferred; Thanks Regards*…..**?* *Sunitha * *Sree Infotech LLC* 9901 E. Valley Ranch Parkway, Suite 3008, Irving, TX 75063 Phone: 972-332-3471 Extn 114 Email Id: suni...@sreeinfotech.com Gtalk: archana.sreeinfo Yahoo ID:archana.sreeinfo -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANB0eHE0FJ8p0QF_gzed8nqJaLCfYCZtQnWNeKkKd-B18Qp2nw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: How long do logs stay in Elasticsearch?
Hi Aaron, Unless documents are configured with a TTL[1], elasticsearch does not delete documents automatically. You are probably looking for a tool like curator[2] that helps deal with time-based indices? [1] http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html [2] https://github.com/elasticsearch/curator On Mon, Oct 27, 2014 at 3:31 PM, Aaron Lee arnnl...@gmail.com wrote: I was wondering how long logs or events stay in Elasticsearch? Is there a way we can change the default value? Thanks, Aaron Lee -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5bf81d3e-2a63-4197-8cfd-65af52fbc2b8%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/5bf81d3e-2a63-4197-8cfd-65af52fbc2b8%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Adrien Grand -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5rNHmLFqLiAHvZVEb%2BarzsDusZr0z9Bw1-ycfG-PfO6Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: plan for river
There is nothing magical about rivers. With some Java code changes, most rivers can be made to run as standalone Java processes. The only thing the rivers do is (weakly) guarantee that only one river instance is run per cluster. Cheers, Ivan On Mon, Oct 27, 2014 at 4:11 AM, joergpra...@gmail.com joergpra...@gmail.com wrote: Yes, there is already a substitution, the JDBC feeder in the JDBC river repo. Future versions of JDBC river will no longer rely on the river API. Jörg On Mon, Oct 27, 2014 at 12:49 AM, Mungeol Heo mungeol@gmail.com wrote: Hi, My question is that will es remove all river related plugin in the future? If it will, I'd like to know that is there substitution for JDBC? Thanks. Best regards, - Mungeol -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cc6d541f-1609-4218-932b-064a27e9692a%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/cc6d541f-1609-4218-932b-064a27e9692a%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFk3x%2BTzscHBohYbiHUb-By%2BVv9w5OGx7vj8hj0oE7MRQ%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFk3x%2BTzscHBohYbiHUb-By%2BVv9w5OGx7vj8hj0oE7MRQ%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBOYCE7rMYC6j46owedA0TVPid3tV5RXAVSTyJfPmXiow%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: nested aggregation against key value pairs
Hi Jay,
Reindexing and having a key that combines the key and value fields would
certainly be the fastest option.
On Mon, Oct 27, 2014 at 1:52 PM, Jay Hilden jay.hil...@gmail.com wrote:
Here is some sample data:
PUT index1
PUT index1/type1/_mapping
{
type1: {
properties: {
kvp: {
type: nested,
properties: {
key: {
type: string
},
value: {
type: string
}
}
}
}
}
}
POST index1/type1
{
kvp: [
{
key: key1,
value: value1
}
]
}
POST index1/type1
{
kvp: [
{
key: key1,
value: value1
},
{
key: key1,
value: value2
},
{
key: key2,
value: value1
},
{
key: key2,
value: value2
}
]
}
The result I'd like combines the fields kvp.key and kvp.value:
key1 - value1: DocCount = 2
key1 - value2: DocCount = 1
key2 - value2: DocCount = 1
I'm starting to think that I need to re-index the data and combine the
kvp.key and kvp.value fields into a single field so that I can
aggregate on it.
On Friday, October 24, 2014 2:17:04 PM UTC-5, Jay Hilden wrote:
I have an ES type with a nested KeyValuePair type. What I'm trying to do
is a terms aggregation on both the key and value fields such that I'd get
the following results:
Key1 - Value1: DocCount = 10
Key1 - Value2: DocCount = 9
Key2 - Value3: DocCount = 4
Here is my mapping:
{
index123 : {
mappings : {
type123 : {
properties : {
authEventID : {
type : long
},
authInput : {
properties : {
uIDExtensionFields : {
type : nested,
properties : {
key : {
type : string
},
value : {
type : string
}
}
}
}
}
}
}
}
}
}
Is there a way to do this?
Thank you.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d9db4ea8-68af-4cc5-a6dc-876f218b58f7%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d9db4ea8-68af-4cc5-a6dc-876f218b58f7%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
Adrien Grand
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7j2Shpz1G7GhpFA5Oqy8ReCWTHrF_D3QJPNjsHQGMcQw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: histogram aggregation with float interval 1.0 gives ArithmeticException
Hi Mark,
This is unfortunately a known limitation of histograms today: they only
work with integer intervals.
https://github.com/elasticsearch/elasticsearch/issues/4847
On Mon, Oct 27, 2014 at 1:17 PM, LiquidMark mark.e.mol...@gmail.com wrote:
Hi, the following query (note bolded line):
{
query: {
filtered: {
query: {
term: {
self_and_ancestors: diamonds
}
},
filter: {
terms: {
attr_types.diamond#color: [
d
]
}
}
}
},
sort: [
{
sell_offer_cents: {
order: asc
}
}
],
fields: _source,
script_fields: {
gap_cents: {
script: custom_score_item_bid_ask_gap,
params: {
individual_price_item_ids: [],
individual_price_item_cents: [],
pb_amount_below_cents: 0
},
lang: native
}
},
aggs: {
all_items: {
global: {},
aggs: {
gem#carats: {
filter: {
terms: {
attr_types.diamond#polish: [
ex,
0001vg
]
}
},
aggs: {
gems#carats: {
histogram: {
field: attr_types.gem#carats,
* interval: 0.1,*
min_doc_count: 0
}
},
gem#carats_stats: {
stats: {
field: attr_types.gem#carats
}
}
}
}
}
}
}
}
Gives the following error:
{
error: SearchPhaseExecutionException[Failed to execute phase
[query], all shards failed; shardFailures
{[D6h8GKwjRqO_Yem09Hu_VA][development-liquidibles::application-items][4]:
QueryPhaseExecutionException[[development-liquidibles::application-items][4]:
query[filtered(filtered(self_and_ancestors:diamonds)-cache(attr_types.diamond#color:d))-cache(_type:item)],from[0],size[10],sort[custom:\sell_offer_cents\:
org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@11ce49a]:
Query Failed [Failed to execute global aggregators]]; nested:
ArithmeticException;
}{[D6h8GKwjRqO_Yem09Hu_VA][development-liquidibles::application-items][3]:
QueryPhaseExecutionException[[development-liquidibles::application-items][3]:
query[filtered(filtered(self_and_ancestors:diamonds)-cache(attr_types.diamond#color:d))-cache(_type:item)],from[0],size[10],sort[custom:\sell_offer_cents\:
org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@8c9d82]:
Query Failed [Failed to execute global aggregators]]; nested:
ArithmeticException;
}{[D6h8GKwjRqO_Yem09Hu_VA][development-liquidibles::application-items][2]:
QueryPhaseExecutionException[[development-liquidibles::application-items][2]:
query[filtered(filtered(self_and_ancestors:diamonds)-cache(attr_types.diamond#color:d))-cache(_type:item)],from[0],size[10],sort[custom:\sell_offer_cents\:
org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@73a7e5]:
Query Failed [Failed to execute global aggregators]]; nested:
ArithmeticException;
}{[D6h8GKwjRqO_Yem09Hu_VA][development-liquidibles::application-items][1]:
QueryPhaseExecutionException[[development-liquidibles::application-items][1]:
query[filtered(filtered(self_and_ancestors:diamonds)-cache(attr_types.diamond#color:d))-cache(_type:item)],from[0],size[10],sort[custom:\sell_offer_cents\:
org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@15d1b1a]:
Query Failed [Failed to execute global aggregators]]; nested:
ArithmeticException;
}{[D6h8GKwjRqO_Yem09Hu_VA][development-liquidibles::application-items][0]:
QueryPhaseExecutionException[[development-liquidibles::application-items][0]:
query[filtered(filtered(self_and_ancestors:diamonds)-cache(attr_types.diamond#color:d))-cache(_type:item)],from[0],size[10],sort[custom:\sell_offer_cents\:
org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@1b8c216]:
Query Failed [Failed to execute global aggregators]]; nested:
*ArithmeticException*; }],
status: 500
}
If I change the interval to be 1.0 or greater, it works. But, I want
intervals of 0.1...
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b8dad371-dfef-4c57-b7d8-433ee1c308c6%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b8dad371-dfef-4c57-b7d8-433ee1c308c6%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
Adrien Grand
--
You received this message because you are subscribed to the Google Groups
elasticsearch
Re: nested aggregation against key value pairs
Thanks Adrien.
On Mon, Oct 27, 2014 at 12:12 PM, Adrien Grand
adrien.gr...@elasticsearch.com wrote:
Hi Jay,
Reindexing and having a key that combines the key and value fields would
certainly be the fastest option.
On Mon, Oct 27, 2014 at 1:52 PM, Jay Hilden jay.hil...@gmail.com wrote:
Here is some sample data:
PUT index1
PUT index1/type1/_mapping
{
type1: {
properties: {
kvp: {
type: nested,
properties: {
key: {
type: string
},
value: {
type: string
}
}
}
}
}
}
POST index1/type1
{
kvp: [
{
key: key1,
value: value1
}
]
}
POST index1/type1
{
kvp: [
{
key: key1,
value: value1
},
{
key: key1,
value: value2
},
{
key: key2,
value: value1
},
{
key: key2,
value: value2
}
]
}
The result I'd like combines the fields kvp.key and kvp.value:
key1 - value1: DocCount = 2
key1 - value2: DocCount = 1
key2 - value2: DocCount = 1
I'm starting to think that I need to re-index the data and combine the
kvp.key and kvp.value fields into a single field so that I can
aggregate on it.
On Friday, October 24, 2014 2:17:04 PM UTC-5, Jay Hilden wrote:
I have an ES type with a nested KeyValuePair type. What I'm trying to
do is a terms aggregation on both the key and value fields such that I'd
get the following results:
Key1 - Value1: DocCount = 10
Key1 - Value2: DocCount = 9
Key2 - Value3: DocCount = 4
Here is my mapping:
{
index123 : {
mappings : {
type123 : {
properties : {
authEventID : {
type : long
},
authInput : {
properties : {
uIDExtensionFields : {
type : nested,
properties : {
key : {
type : string
},
value : {
type : string
}
}
}
}
}
}
}
}
}
}
Is there a way to do this?
Thank you.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d9db4ea8-68af-4cc5-a6dc-876f218b58f7%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d9db4ea8-68af-4cc5-a6dc-876f218b58f7%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
Adrien Grand
--
You received this message because you are subscribed to a topic in the
Google Groups elasticsearch group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/NJRzaH6FUfY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7j2Shpz1G7GhpFA5Oqy8ReCWTHrF_D3QJPNjsHQGMcQw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7j2Shpz1G7GhpFA5Oqy8ReCWTHrF_D3QJPNjsHQGMcQw%40mail.gmail.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAD3qxy53ggTWrK4taryaLe7jPu%3DSPJMNeUzbnzbO%2B%3D_EzbdTBQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: Sorting weirdness
Hi Michael,
This is because your field is tokenized and elasticsearch will take the
minimum value as a sort value. For the first document, this would be 5,
which compares lower than B in the second string. To make it work as
expected, you need to index your field as `not_analyzed` (
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#string
).
On Sat, Oct 25, 2014 at 8:37 PM, Michael Irwin m...@livej.am wrote:
I have a mapping like this:
venue: {
type: nested,
include_in_parent: true,
properties: {
name: {
type: string
}
}
If I'm sorting by 'venue.name' ascending, why would a name like 'Terminal
5' be sorted before 'B.B. King Blues Club Grill'? Does it have
something to do with the number '5' in the name?
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/159fe507-bb01-4504-9ba7-e6b7e7bb964c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/159fe507-bb01-4504-9ba7-e6b7e7bb964c%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
Adrien Grand
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6eGjjezKR592k7_8WUf151j_CxE7aR%2BuMrB0Fgo8U-HA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: Help: Flattened aggregations (with limiting and sorting)
Hi Matt, I don't understand what the problem is, can you maybe try to elaborate a bit? Thanks. On Fri, Oct 24, 2014 at 4:00 PM, Matt Traynham skitch...@gmail.com wrote: Thoughts, anybody? I saw that you can somewhat do this with scripts and letting the top aggregation encompass all term fields, but is that any more performant? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c79e42a7-bd82-4aed-9e1d-4e882566f630%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/c79e42a7-bd82-4aed-9e1d-4e882566f630%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Adrien Grand -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j65OMxJQkCxRBORW9V2%3Dy2wUNxDmbWiWKdJh_27qjsX1g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Modify the index setting after the index created ? what's the function of search_quote_analyzer ?
Bump, I'm having the same problem.
On Thursday, June 12, 2014 10:32:14 PM UTC-5, Ivan Ji wrote:
Hi all,
I want to modify one field's search analyzer from standard to keyword
after the index created. So I try to PUT mapping :
$ curl -XPUT 'http://localhost:9200/qindex/main/_mapping' -d '
{
main : {
properties : {
name : { type: string, index: analyzed,
index_analyzer: filename_ngram, search_analyzer: keyword}
}
}
}
'
The operation seems succeed. Because I expect it might conflict, what
would the situations that conflict might occur? This is my first question.
Anyway then I try to get the mapping out: (partial)
name: {
type: string,
index_analyzer: filename_ngram,
search_analyzer: keyword,
include_in_all: true,
search_quote_analyzer: standard
}
So I am wondering whether my operation succeeded? and what is the
search_quote_analyzer function? And it still remains standard, does it
matter?
Could anyone answer me these questions?
Cheers,
Ivan
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d3ec9347-931e-43bf-a199-d667a43f42a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Find the 100 closest neighbors to a point (lng, lat)
Hi Michael, You can do that using geo-distance sorting: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_geo_distance_sorting It would probably not be optimal in the sense that elasticsearch will need to compute the distance for every matching document, but maybe it would still be fast enough? On Fri, Oct 24, 2014 at 11:11 AM, Michael Lumbroso mich...@sportintown.com wrote: Hello, sorry if this question has already been asked, but I didn't find much material during my search. Basically, what I need to do, is find the exact 100 closest objects around a spatial point (longitude, latitude), among a 1 million geolocalized object all around the world. Is there an efficient way to do that? (performance is the most important parameter here) Are there plugins/libraries to help me do so? Are there better options than Elasticsearch for this very problem? Thanks for your help, and keep up the good work on this wonderful tool Michael -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/04ce1fbd-c88f-4517-9d56-044bb235c29c%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/04ce1fbd-c88f-4517-9d56-044bb235c29c%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Adrien Grand -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j45xYH%2BGmF%2BQ3t5m1OYLKZD7Vp6p0HxpmkD7-Q%2B7Zu1hQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Having problems with hit count from OR filter
Hi Lenin,
This looks like a bug indeed... Did you manage to nail down this issue?
Could you run the same terms aggregation on the or query to see the
distribution of terms?
On Fri, Oct 24, 2014 at 4:05 AM, Lenin lsubraman...@maverixbio.com wrote:
I have the following aggregation which show all the count by values for a
particular field values.
http://localhost:8200/index1/collection1/_search?search_type=count
{
aggs : {
effects : {
terms : {
field : type
}
}
}
}
Output is
{
took: 2,
timed_out: false,
_shards: {
total: 5,
successful: 5,
failed: 0
},
hits: {
total: 133490,
max_score: 0,
hits: []
},
aggregations: {
effects: {
buckets: [
{
key: snp,
doc_count: 112918
},
{
key: indel,
doc_count: 15725
},
{
key: mnp,
doc_count: 3751
},
{
key: mixed,
doc_count: 1096
}
]
}
}
}
When i count the individual count, the total tallies to 133490 (which is
the total number of docs in the colleciton.
But when i do the following query, i don't get the exact result count ( I
am using all the possible values which returned above and converted to an
OR query ) :
{
query: {
filtered: {
filter: {
and: [
{
query: {
filtered: {
filter: {
or: { filters : [
{
query: {
match: {
type: SNP
}
}
},
{
query: {
match: {
type: INS
}
}
},
{
query: {
match: {
type: DEL
}
}
},
{
query: {
match: {
type: COMPLEX
}
}
},
{
query: {
match: {
type: MNP
}
}
},
{
query: {
match: {
type: MIXED
}
}
}
] }
}
}
}
}
]
}
}
}
}
Output :
{
took: 3,
timed_out: false,
_shards: {
total: 5,
successful: 5,
failed: 0
},
hits: {
total: 117765,
max_score: 1,
hits: [
.]
}
}
As you can see the result hit count doesn't match the number of documents.
When i convert the above query from a match to terms based one, i get the
exact count.
{
query: {
filtered: {
filter: {
and: [
{
query: {
filtered: {
filter: {
and : [{
query: {
terms: {
type: [snp, mixed, indel, mnp]
}
}
}]
}
}
}
}
]
}
}
}
}
Is this an issue with the OR query ?
Also, is there a suitable alternative with the match query where i could
easily represent the above query like :
{
query : {
match : { type : [ snp, mixed, indel, mnp ] }
}
}
Any help is appreciated.
Thanks.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b466f820-d5cc-4a3b-a77a-79fe5aaa8ada%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b466f820-d5cc-4a3b-a77a-79fe5aaa8ada%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options,
Re: Having problems with hit count from OR filter
Hi Adrien,
Thanks for getting back. I was able indeed fix the issue, it was a data
problem in my end.
But I ran into another issue with OR filter while i was able to figure the
above one.
I have posted it as github.
https://github.com/elasticsearch/elasticsearch/issues/8246
I have a testdata to simulate the same as well. Please let me know if you
need anything more.
Thanks.
-Lenin
On Monday, October 27, 2014 10:32:20 AM UTC-7, Adrien Grand wrote:
Hi Lenin,
This looks like a bug indeed... Did you manage to nail down this issue?
Could you run the same terms aggregation on the or query to see the
distribution of terms?
On Fri, Oct 24, 2014 at 4:05 AM, Lenin lsubra...@maverixbio.com
javascript: wrote:
I have the following aggregation which show all the count by values for a
particular field values.
http://localhost:8200/index1/collection1/_search?search_type=count
{
aggs : {
effects : {
terms : {
field : type
}
}
}
}
Output is
{
took: 2,
timed_out: false,
_shards: {
total: 5,
successful: 5,
failed: 0
},
hits: {
total: 133490,
max_score: 0,
hits: []
},
aggregations: {
effects: {
buckets: [
{
key: snp,
doc_count: 112918
},
{
key: indel,
doc_count: 15725
},
{
key: mnp,
doc_count: 3751
},
{
key: mixed,
doc_count: 1096
}
]
}
}
}
When i count the individual count, the total tallies to 133490 (which is
the total number of docs in the colleciton.
But when i do the following query, i don't get the exact result count (
I am using all the possible values which returned above and converted to an
OR query ) :
{
query: {
filtered: {
filter: {
and: [
{
query: {
filtered: {
filter: {
or: { filters : [
{
query: {
match: {
type: SNP
}
}
},
{
query: {
match: {
type: INS
}
}
},
{
query: {
match: {
type: DEL
}
}
},
{
query: {
match: {
type: COMPLEX
}
}
},
{
query: {
match: {
type: MNP
}
}
},
{
query: {
match: {
type: MIXED
}
}
}
] }
}
}
}
}
]
}
}
}
}
Output :
{
took: 3,
timed_out: false,
_shards: {
total: 5,
successful: 5,
failed: 0
},
hits: {
total: 117765,
max_score: 1,
hits: [
.]
}
}
As you can see the result hit count doesn't match the number of
documents. When i convert the above query from a match to terms based
one, i get the exact count.
{
query: {
filtered: {
filter: {
and: [
{
query: {
filtered: {
filter: {
and : [{
query: {
terms: {
type: [snp, mixed, indel, mnp]
}
}
}]
}
}
}
}
]
}
}
}
}
Is this an issue with the OR query ?
Also, is there a suitable alternative with the match query where i could
easily represent the above query like :
{
query : {
match : { type : [ snp, mixed, indel, mnp ] }
}
}
Any help is appreciated.
Thanks.
--
You received this message because
Tribe node setup, /_cat/nodes unresponsive
Hi all,
I have a tribe node setup across two clusters, on server0 ($IP0) and
server1 ($IP1), using unicast discovery.
server0 also runs the elasticsearch tribe node instance.
This setup works fine most of the time, and I can see logs from all
clusters on tribe node Kibana.
The issue is that sometimes the elasticsearch instances go unresponsive.
More specifically, it's the nodes API (including /_cat/nodes) that do not
respond, which results in Kibana throwing the following error message:
Could not reach http://.../_nodes. If you are using a proxy, ensure it is
configured correctly
At first, I thought it was just the tribe node that was doing this, but
just today I noticed this on one of the clusters as well.
Other URLs don't seem to be affected (e.g. /, /_cluster/health, etc).
After some time, the nodes API comes back and produces the following
output, and goes unresponsive again.
I've also tried restarting the elasticsearch processes. While it fixes the
unresponsiveness in the short term, it then goes back to the same cycle.
jun@server0 ~ % curl -XGET $IP0:9201/_cat/nodes?pretty=truev=true
hostip heap.percent ram.percent load node.role master name
server0 $IP0 11 5 2.97 c x
tribe
server1 $IP1 16 38 2.74 d x
s1-elasticsearch
server0 $IP0 17 5 2.97 d x
s0-elasticsearch
server1 $IP1 c x
s1-logstash
server0 $IP0 c x
tribe
server0 $IP0 c x
tribe
server0 $IP0 c x
tribe
server0 $IP0 c x
s0-logstash
jun@server0 ~ % curl -XGET $IP0:9200/_cat/nodes?pretty=truev=true
{
error : NullPointerException[null],
status : 500
}
jun@server0 ~ % curl -XGET $IP1:9200/_cat/nodes?pretty=truev=true
{
error : NullPointerException[null],
status : 500
}
jun@server0 ~ % curl -XGET '$IP0:9201/_nodes/process?pretty=truev=true'
{
cluster_name : elasticsearch,
nodes : {
gPX5VOT5TdGPOF6wq9ErKA : {
name : tribe,
transport_address : inet[/$IP0:9300],
host : server0,
ip : $IP0,
version : 1.3.4,
build : a70f3cc,
http_address : inet[/$IP0:9201],
attributes : {
client : true,
data : false
},
process : {
refresh_interval_in_millis : 1000,
id : 187050,
max_file_descriptors : 65535,
mlockall : false
}
},
liMvaS8cSu2PafCM653LHA : {
name : s1-elasticsearch,
transport_address : inet[/$IP1:9301],
host : server1,
ip : $IP1,
version : 1.3.4,
build : a70f3cc,
http_address : inet[/$IP1:9200],
process : {
refresh_interval_in_millis : 1000,
id : 32465,
max_file_descriptors : 65535,
mlockall : false
}
},
z-GOP2v7Q7-FvlaS8bIBJQ : {
name : s0-elasticsearch,
transport_address : inet[/$IP0:9301],
host : server0,
ip : $IP0,
version : 1.3.4,
build : a70f3cc,
http_address : inet[/$IP0:9200],
process : {
refresh_interval_in_millis : 1000,
id : 165512,
max_file_descriptors : 65535,
mlockall : false
}
}
}
}
Any help would be greatly appreciated.
Thanks,
Jun
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/314fd788-a883-45c6-9c94-226dd29f9d5c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: analyzer settings for breaking up words on hyphens
Thanks! i'll go ahead and try the pattern tokenizer route. On Mon, Oct 27, 2014 at 1:22 PM, Ivan Brusic i...@brusic.com wrote: You can either use a pattern tokenizer with your patterns being whitespace + hypen, or further decompose your token post tokenization with the word delimiter token filter, which is much harder to use (and might be an overkill for your use case). http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-pattern-tokenizer.html http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-word-delimiter-tokenfilter.html Cheers, Ivan On Mon, Oct 27, 2014 at 7:55 AM, Mike Topper top...@gmail.com wrote: Hello, I have a field that is using the whitespace tokenizer, but I also want to tokenize on hyphens (-) like the standard analyzer does. I'm having trouble figuring out what additional custom settings I would have to put in there in order to be able to tokenize off of hyphens as well. Thanks, Mike -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALdNedLtdAWEiQN%2BoUV17J5e8DowMbDva2pJn1S%3Dr9w1qtP9bA%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CALdNedLtdAWEiQN%2BoUV17J5e8DowMbDva2pJn1S%3Dr9w1qtP9bA%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDeFdP4-imY0ReSZTkSAnfQ8o6_hWp9MAB0YcMOgDo9rA%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDeFdP4-imY0ReSZTkSAnfQ8o6_hWp9MAB0YcMOgDo9rA%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALdNedK9EfeL-FGbavnKO4t%3DkrQ%2BxeQ-O2p2wL-P_iqGSrhrsg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: histogram aggregation with float interval 1.0 gives ArithmeticException
Hi Adrien,
Thanks for letting me know!
-Mark
On Mon, Oct 27, 2014 at 12:14 PM, Adrien Grand
adrien.gr...@elasticsearch.com wrote:
Hi Mark,
This is unfortunately a known limitation of histograms today: they only
work with integer intervals.
https://github.com/elasticsearch/elasticsearch/issues/4847
On Mon, Oct 27, 2014 at 1:17 PM, LiquidMark mark.e.mol...@gmail.com
wrote:
Hi, the following query (note bolded line):
{
query: {
filtered: {
query: {
term: {
self_and_ancestors: diamonds
}
},
filter: {
terms: {
attr_types.diamond#color: [
d
]
}
}
}
},
sort: [
{
sell_offer_cents: {
order: asc
}
}
],
fields: _source,
script_fields: {
gap_cents: {
script: custom_score_item_bid_ask_gap,
params: {
individual_price_item_ids: [],
individual_price_item_cents: [],
pb_amount_below_cents: 0
},
lang: native
}
},
aggs: {
all_items: {
global: {},
aggs: {
gem#carats: {
filter: {
terms: {
attr_types.diamond#polish: [
ex,
0001vg
]
}
},
aggs: {
gems#carats: {
histogram: {
field: attr_types.gem#carats,
* interval: 0.1,*
min_doc_count: 0
}
},
gem#carats_stats: {
stats: {
field: attr_types.gem#carats
}
}
}
}
}
}
}
}
Gives the following error:
{
error: SearchPhaseExecutionException[Failed to execute phase
[query], all shards failed; shardFailures
{[D6h8GKwjRqO_Yem09Hu_VA][development-liquidibles::application-items][4]:
QueryPhaseExecutionException[[development-liquidibles::application-items][4]:
query[filtered(filtered(self_and_ancestors:diamonds)-cache(attr_types.diamond#color:d))-cache(_type:item)],from[0],size[10],sort[custom:\sell_offer_cents\:
org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@11ce49a]:
Query Failed [Failed to execute global aggregators]]; nested:
ArithmeticException;
}{[D6h8GKwjRqO_Yem09Hu_VA][development-liquidibles::application-items][3]:
QueryPhaseExecutionException[[development-liquidibles::application-items][3]:
query[filtered(filtered(self_and_ancestors:diamonds)-cache(attr_types.diamond#color:d))-cache(_type:item)],from[0],size[10],sort[custom:\sell_offer_cents\:
org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@8c9d82]:
Query Failed [Failed to execute global aggregators]]; nested:
ArithmeticException;
}{[D6h8GKwjRqO_Yem09Hu_VA][development-liquidibles::application-items][2]:
QueryPhaseExecutionException[[development-liquidibles::application-items][2]:
query[filtered(filtered(self_and_ancestors:diamonds)-cache(attr_types.diamond#color:d))-cache(_type:item)],from[0],size[10],sort[custom:\sell_offer_cents\:
org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@73a7e5]:
Query Failed [Failed to execute global aggregators]]; nested:
ArithmeticException;
}{[D6h8GKwjRqO_Yem09Hu_VA][development-liquidibles::application-items][1]:
QueryPhaseExecutionException[[development-liquidibles::application-items][1]:
query[filtered(filtered(self_and_ancestors:diamonds)-cache(attr_types.diamond#color:d))-cache(_type:item)],from[0],size[10],sort[custom:\sell_offer_cents\:
org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@15d1b1a]:
Query Failed [Failed to execute global aggregators]]; nested:
ArithmeticException;
}{[D6h8GKwjRqO_Yem09Hu_VA][development-liquidibles::application-items][0]:
QueryPhaseExecutionException[[development-liquidibles::application-items][0]:
query[filtered(filtered(self_and_ancestors:diamonds)-cache(attr_types.diamond#color:d))-cache(_type:item)],from[0],size[10],sort[custom:\sell_offer_cents\:
org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@1b8c216]:
Query Failed [Failed to execute global aggregators]]; nested:
*ArithmeticException*; }],
status: 500
}
If I change the interval to be 1.0 or greater, it works. But, I want
intervals of 0.1...
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b8dad371-dfef-4c57-b7d8-433ee1c308c6%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b8dad371-dfef-4c57-b7d8-433ee1c308c6%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit
Re: Date histogram fixed interval
Hi Jeremy, It is not possible. One work-around could be to over request (for example create per-minute buckets) and finally merge them to get the desired number of buckets. I believe this is something that could be achieved through https://github.com/elasticsearch/elasticsearch/issues/8110. Another option could be to run two requests: a first one to figure the min/max timestamps and a second one with an interval that would generate the desired number of buckets. On Mon, Oct 27, 2014 at 7:18 PM, jnortey jeremy.nor...@gmail.com wrote: Is it possible to specify a fixed interval for a date histogram facet/aggregation? Here is what I mean by fixed: Lets say that I created an aggregation that returned data for between a certain time range. This could be for the last 24 hours, the last 30 days, etc. Regardless of the time range, I would like to get exactly six buckets of data regardless of the size of the time range. So if the time range was for the last 24 hours, I would get a bucket for each four hour period. If the time range were 30 days, I would get a bucket for each 5 day period. Is it possible to specify something like this in the date time histogram? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6c0c6fcd-9f4e-4aa3-91cc-5a586f88811e%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/6c0c6fcd-9f4e-4aa3-91cc-5a586f88811e%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Adrien Grand -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j698OgAmvmThRcha-opcS32VNrqb9mf2s4jCvj1-JNPCA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: FacetPhaseExecutionException with new Marvel installation
Hi again,
Yep, I had added the required settings to the yaml files first.
I tried the steps you described above, and it did not have any effect --
still no template present, and still getting the error. Since it wasn't
too much trouble, I started over from scratch -- rebuilt the monitoring
cluster, but also uninstalled then reinstalled the plugin in the production
cluster, and restarted. After this, I saw a bunch of update_mapping calls.
The template was present, and the errors went away. It seems that some
state regarding Marvel is kept in the production cluster, and whatever it
was got cleared when I reinstalled the plugin there. That may be something
worth mentioning in the installation docs.
In any case, thanks for your help -- it's all working now!
Cheers,
Ross
On Tuesday, 28 October 2014 09:48:12 UTC+11, Boaz Leskes wrote:
Hey,
You probably did but just double checking- did you change the settings in
the yaml files before restarting the nodes?
There is an easier way to fix this than a full restart: first restart a
single node on production. That will cause the agent to check again for the
template. Verify that the template was added. The delete all .marvel-2014*
indices on the monitoring cluster and let them be recreated base on the
template.
Boaz
—
Sent from Mailbox https://www.dropbox.com/mailbox
On Mon, Oct 27, 2014 at 11:25 PM, Ross Simpson simp...@gmail.com
javascript: wrote:
Hi Boaz,
To install, I ran
bin/plugin --install elasticsearch/marvel/latest
on each node in both clusters, then restarted both clusters.
Since then, I have tried several things, including deleting the indexes
from the monitoring cluster and reinstalling the plugin on the monitoring
cluster. I'll try now to delete all the marvel indexes, uninstall, then
reinstall marvel into both clusters.
I'm a bit stumped otherwise, so I'm all ears for any other suggestions.
Cheers,
Ross
On Tuesday, 28 October 2014 08:30:54 UTC+11, Boaz Leskes wrote:
It looks like something is wrong is indeed wrong with your marvel index
template which should be there before data is indexed. How did you install
marvel? Did you perhaps delete the data folder of the monitoring cluster
after production was already shipping data?
Cheers,
Boaz
On Monday, October 27, 2014 7:45:34 AM UTC+1, Ross Simpson wrote:
To troubleshoot a little more, I rebuilt the monitoring cluster to use
ElasticSearch 1.1.1, which matches the ES version used in the production
cluster. No luck.
On the Overview dashboard, I can see some data (summary, doc count,
search and indexing rates are all populated [screenshot attached]), but
but
both the nodes and indices sections are empty other than the errors
mentioned in the previous post. Cluster pulse doesn't show any events at
all; node stats and index stats do both show data.
Any further suggestions would be greatly appreciated :)
Cheers,
Ross
On Monday, 27 October 2014 11:15:42 UTC+11, Ross Simpson wrote:
I've got a brand-new Marvel installation, and am having some
frustrating issues with it: on the overview screen, I am constantly
getting
errors like:
*Oops!* FacetPhaseExecutionException[Facet [timestamp]: failed to
find mapping for node.ip_port.raw]
*Production cluster:*
* ElasticSearch 1.1.1
* Marvel 1.2.1
* Running in vSphere
*Monitoring cluster:*
* ElasticSearch 1.3.4
* Marvel 1.2.1
* Running in AWS
After installing the plugin and bouncing all nodes in both clusters,
Marvel seems to be working -- an index has been created in the monitoring
cluster (.marvel-2014.10.26), and I see thousands of documents in
there. There are documents with the following types: cluster_state,
cluster_stats, index_stats, indices_stats, node_stats. So, it does
seem that data is being shipped from the prod cluster to the monitoring
cluster.
I've seen in the user group that other people have had similar issues.
Some of those mention problems with the marvel index template. I don't
seem to have any at all templates in my monitoring cluster:
$ curl -XGET localhost:9200/_template/
{}
I tried manually adding the default template (as described in
http://www.elasticsearch.org/guide/en/marvel/current/#config-marvel-indices),
but that didn't seem to have any effect.
So far, I've seen just two specific errors in Marvel:
* FacetPhaseExecutionException[Facet [timestamp]: failed to find
mapping for node.ip_port.raw]
* FacetPhaseExecutionException[Facet [timestamp]: failed to find
mapping for index.raw]
I've also looked through the logs on both the production and
monitoring clusters, and the only errors are in the monitoring cluster
resulting from queries from the Marvel UI, like this:
[2014-10-27 11:08:13,427][DEBUG][action.search.type ] [ip-10-4-
1-187] [.marvel-2014.10.27][1], node[SR_hriFmTCav-8ofbKU-8g], [R], s[
STARTED]: Failed to execute
Re: analyzer settings for breaking up words on hyphens
Or you could cheat and use a character filter to turn the hyphen into spaces. Lots of ways to skin a cat. On Mon, Oct 27, 2014 at 7:07 PM, Mike Topper top...@gmail.com wrote: Thanks! i'll go ahead and try the pattern tokenizer route. On Mon, Oct 27, 2014 at 1:22 PM, Ivan Brusic i...@brusic.com wrote: You can either use a pattern tokenizer with your patterns being whitespace + hypen, or further decompose your token post tokenization with the word delimiter token filter, which is much harder to use (and might be an overkill for your use case). http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-pattern-tokenizer.html http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-word-delimiter-tokenfilter.html Cheers, Ivan On Mon, Oct 27, 2014 at 7:55 AM, Mike Topper top...@gmail.com wrote: Hello, I have a field that is using the whitespace tokenizer, but I also want to tokenize on hyphens (-) like the standard analyzer does. I'm having trouble figuring out what additional custom settings I would have to put in there in order to be able to tokenize off of hyphens as well. Thanks, Mike -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALdNedLtdAWEiQN%2BoUV17J5e8DowMbDva2pJn1S%3Dr9w1qtP9bA%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CALdNedLtdAWEiQN%2BoUV17J5e8DowMbDva2pJn1S%3Dr9w1qtP9bA%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDeFdP4-imY0ReSZTkSAnfQ8o6_hWp9MAB0YcMOgDo9rA%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDeFdP4-imY0ReSZTkSAnfQ8o6_hWp9MAB0YcMOgDo9rA%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALdNedK9EfeL-FGbavnKO4t%3DkrQ%2BxeQ-O2p2wL-P_iqGSrhrsg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CALdNedK9EfeL-FGbavnKO4t%3DkrQ%2BxeQ-O2p2wL-P_iqGSrhrsg%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1oEgb55Y0tVU6VNzDXEF6RJQRRFZ%3DW2_iKrRmJBMVW2Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
ElasticSearch - significant term aggregation with range
I am interested to know how can I add a range for a significant terms
aggregations query. For example:
{
query: {
terms: {
text_content: [
searchTerm
]
},
range: {
dateField: {
from: date1,
to: date2
}
}
},
aggregations: {
significantQTypes: {
significant_terms: {
field: field1,
size: 10
}
}
},
size: 0
}
will not work. Any suggestions on how to specify the range?
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/50f0ba07-9f76-4472-9bbd-15d774cc63f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: delete/update nested documents with elasticsearch Java API
script: removeObjects = new java.util.ArrayList(); foreach (item :
ctx._source.list) {if (item['tweet_id'] == tweet_id)
{removeObjects.add(item) } }
for(removeObject:removeObjects){ctx._source.list.remove(removeObject);},
2014년 10월 8일 수요일 오후 8시 12분 55초 UTC+9, Rajit Garg 님의 말:
Hey
*I figured it out, I am able to update but I am not able to remove object
that satisfies the condition*
PUT twitter/twit/1
{
list: [
{
tweet_id: 1,
a: b
},
{
tweet_id: 123,
a: f
}
]
}
POST /twitter/twit/1/_update
{
script: foreach (item : ctx._source.list) {if (item['tweet_id'] ==
tweet_id) { item['new_field'] = 'ghi'; } },
params: {
tweet_id: 123
}
}
*this is working*
*for remove i am doing this*
POST /twitter/twit/1/_update
{
script: foreach (item : ctx._source.list) {if (item['tweet_id'] ==
tweet_id) {ctx._source.list.remove(item); } },
params: {
tweet_id: 123
}
}
*but this is not working and giving this error, *
ElasticsearchIllegalArgumentException[failed to execute script]; nested:
ConcurrentModificationException;
Error: ElasticsearchIllegalArgumentException[failed to execute script];
nested: ConcurrentModificationException;
..
...
*please help*
On Wednesday, April 10, 2013 12:22:14 PM UTC+5:30, Andrei Tolnai wrote:
Hello.
I use *Elastic Search Java API *for basic *CRUD *operations on ES
documents; with *root type* documents it is working fine.
However when I want to delete a nested document, I don't know how to use
the *Java API* and scripting that *Elastic Search *provides.
I have the following *root type* document which has in it a nested *users
*property (nested document).
{
customer:
{
_all : {enabled : false},
_source : {enabled : true},
store : true,
properties:
{
location : { type: string, store: yes,
index:not_analyzed },
comments : { type: string, store: yes,
index:not_analyzed },
* users:*
* {*
* type:nested,*
* store:yes, *
* properties:*
*{*
* _id : { type:string, store:yes,
index:not_analyzed },*
* name : { type:string, store:yes,
index:not_analyzed },*
* birthDate : { type:date, store:yes,
index:not_analyzed }*
*}*
* }*
}
}
}
If I want to add a user I do the following in *Java */ *Scala *code:
val json =
{\_id\:\1\,\name\:\Anthony\,\birthDate\:\2013-04-10T06:45:26.186Z\}
val updateRequest = new UpdateRequest(indexName, customer, customerId)
updateRequest.script( if (ctx._source.users == null) { ctx._source.users
= +json+ } else { ctx._source.users += +json+ } )
This works OK for adding nested *users *AKA updating the *customer root
type* document.
How is it to be done for *deleting* or *updating *a specific user ID
from the nested *users* nested document?
Thanks!
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8ef09dca-306d-4fa7-8936-a20e86125dac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Kibana: Deploy without Ruby / Warbler / Gems / Java etc
In Kibana 3, it was possible to deploy Kibana as a set of static files. In Kibana 4, the default mode is to deploy using a full stack of Ruby code on top of Warbler and JRuby etc. It seems from my experimentation that this is not really required beyond - automatically populating the /config endpoint with some configuration settings - creating an Elasticsearch proxy (presumably to avoid dealing with CORS and XSS limitations) - enumerating plugins (for reference I got Kibana 4 working in a static deployment by fixing a few pathing issues and adding CORS support to ES) While having a preconfigured stack that launches from a single command is awesome, it also bloats the deployment requirement for more sophisticated deployment. What I'd like to know is if the general development direction is that more dynamic configuration will be done in the Ruby code - creating an ever increasing requirement on the Ruby stack in the future? e.g. Generating a plugin manifest could easily be done using a bash / js script - it does not need a full Ruby stack. Thanks, Ben -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f316bdf1-01d4-4e20-8037-a36baf650263%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
