Re: Very regular disconnect and recover - every 2 hours
You can try winding out the timeouts, see http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#fault-detection On 31 March 2015 at 16:57, Neil Andrassy neil.andra...@thefilter.com wrote: It's probably something like that, but it only seems to be a problem with the more up to date version of ES. I'm keen to work out if there's a configuration option I can tweak in 1.4.4 to make ES more robust in this scenario or whether there's an issue around recovering dropped TCP connections between nodes in more recent versions. On Tuesday, 31 March 2015 03:33:18 UTC+1, Mark Walkom wrote: It's not the VPN reconnecting is it? On 31 March 2015 at 01:32, Neil Andrassy neil.a...@thefilter.com wrote: Hi, I have two independent clusters running across more or less the same machines. They're split across a pretty high bandwidth and relatively low latency VPN link. One cluster is running v1.0.1 and seems to stay up all the time. The other cluster is currently running 1.4.4 (and was running 1.4.2 before that) and seems to disconnect like clockwork every two hours. The disconnect of the nodes on one side of the link is brief, they rejoin and the recovery proceeds as normal. Any ideas what might cause this? Could it be data related? The newer cluster has more indexes shards than the old, but the co-ordinators (3 of / min master count 2) don't seem particularly stressed. Any thoughts on what, specifically to look for or whether any particular setting or code change might make the cluster more susceptible to disconnect when there's a minor / brief network connectivity blip? (and yes, I know multi-site isn't a recommended configuration - there are other challenges for us with the tribe node approach too, though :( ) Thanks in advance for any ideas or insight. N -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2% 40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/793f13a8-9ca8-4d86-b194-47b4e9cd5125%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/793f13a8-9ca8-4d86-b194-47b4e9cd5125%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_fbZXbP_iY4ZpJODXPmumh15CntRT6S4HaJdrvqv593A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: mlockall
Hi, here is the info Distributor ID: Ubuntu Description:Ubuntu 14.04.1 LTS Release:14.04 Codename: trusty I have tried to set mlockall = true on two server . i am able to successfully do it on one but not able to do it on other server On Tue, Mar 31, 2015 at 11:56 AM, Mark Walkom markwal...@gmail.com wrote: Which linux? On 31 March 2015 at 16:16, Cheten Dev cheten@carwale.com wrote: Hi, I am on linux server with elasticsearch 1.5.0 On Tue, Mar 31, 2015 at 8:07 AM, Mark Walkom markwal...@gmail.com wrote: A bit more info would be useful. What version of ES, what OS? On 30 March 2015 at 22:24, Chetan Dev cheten@carwale.com wrote: Hi, i have tried everything written on the document to set mlackall = true but nothing seems working here what am i missing here ? Thanks -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5748d9fb-120a-4521-8caa-62b780ea0594%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/5748d9fb-120a-4521-8caa-62b780ea0594%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/R41hW2QaL0w/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-92m9Qw4j4T8gwUV6%2BumkvbAOENTTsiupMurpGf8NZSw%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-92m9Qw4j4T8gwUV6%2BumkvbAOENTTsiupMurpGf8NZSw%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAG_NmL9T4dZeHUYpN0_5gwbfppL7%2BOJyNc6t3SQUpSe9N5oN2Q%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAG_NmL9T4dZeHUYpN0_5gwbfppL7%2BOJyNc6t3SQUpSe9N5oN2Q%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/R41hW2QaL0w/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_5X8ubgUVPd6ZV%2Bh_EP1m72Noht4DOvMCysUieWrQp9Q%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_5X8ubgUVPd6ZV%2Bh_EP1m72Noht4DOvMCysUieWrQp9Q%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAG_NmL-ADaAv%2BUx76DUVmVc7q0CJrT%2BV5VOJnUCYtfa-7CcW2w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: What is the best configuration to run on linux VPS server without crashing?
Hi Mark, Our server have 7.5GB RAM, 4vCPUs (in Amazon EC2), our data about 50 thousand records (about 50MB) I think ElasticSearch (ES) server handle 10 connections at the same time (per second) I have assign 2GB RAM for HEAP_SIZE, our server will run normally for a while about 5 minutes and after that, the ES server don't respond anymore connection - It seems ElasticSearch Service dead. Could you tell me the best configuration? Vào 22:29:36 UTC+7 Thứ Sáu, ngày 20 tháng 3 năm 2015, Mark Walkom đã viết: By the looks of things, you should look for a new provider as they are doing some things on the underlying hypervisor restricting ES from locking memory access. However your heap size is very small, how much data is in your cluster. On 20 March 2015 at 07:13, Yashin Soraballee yashin.s...@gmail.co Mam javascript: wrote: Hello guys, I am trying to run elasticsearch on a VPS server running CentOS with 4GB of RAM. It starts successfully but with the following errors and warning message below. # sudo service elasticsearch start error: permission denied on key 'vm.max_map_count' Starting elasticsearch: [ OK ] root@vps15042 [~]# log4j:WARN No appenders could be found for logger (common) log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for morefo. As you can notice, there is no permission to set the vm.max_map_count setting on this VPS server ( even with root access ). So the only way to get rid of the message is by commenting the max_map_count setting out. It then runs for a maximum of 4 hours and then tell me Elastic service dead but pid still exists. It is most probably running out of memory and getting killed. The following is a log report as such : Mar 18 22:37:51 ecvp156s kernel: [3712702.735583] 22234 (java) invoked oom-killer in ub 15042 generation 0 gfp 0x200d2 Mar 18 22:37:51 ecvp156s kernel: [3712702.736531] UB-15042-Mem-Info: Mar 18 22:37:51 ecvp156s kernel: [3712702.756570] Out of memory in UB 15042: OOM killed process 22232 (java) score 0 vm:7815680kB, rss:3786544kB, swap:0kB Mar 18 22:37:52 ecvp156s kernel: [3712703.677851] oom-killer in ub 15042 generation 0 ends: task died Mar 18 22:38:21 ecvp156s kernel: [3712732.546844] 25904 (java) invoked oom-killer in ub 15042 generation 1 gfp 0x200d2 Mar 18 22:38:21 ecvp156s kernel: [3712732.547766] UB-15042-Mem-Info: Mar 18 22:38:21 ecvp156s kernel: [3712732.567524] Out of memory in UB 15042: OOM killed process 25901 (java) score 0 vm:7815680kB, rss:3781624kB, swap:0kB Mar 18 22:38:22 ecvp156s kernel: [3712733.475601] oom-killer in ub 15042 generation 1 ends: task died Mar 19 00:24:42 ecvp156s kernel: [3719119.150426] UB-9340-Mem-Info: My host provider is telling me to increase the amount of RAM. However I am running ES on a windows surface pro tablet with 4GB of RAM and core i3, and I am getting absolutely no issues with not a so powerful machine. Can I change some configurations to make it work perfectly? I've already tried the following and other combinations, but nothing keeps it running for long : /etc/security/limits.conf elasticsearch hard memlock 30 /etc/sysconfig/elasticsearch ES_HEAP_SIZE 1g MAX_LOCKED_MEMORY to 30 /etc/elasticsearch/elasticsearch.yml index.number_of_shards: 1 index.number_of_replicas: 0 index.term_index_interval: 256 index.term_index_divisor: 5 bootstrap.mlockall: true Thank you for your time. I am stuck with this and I'll have to change server if I cannot resolve it. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/73d39af1-9254-4bce-9590-789554db90fb%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/73d39af1-9254-4bce-9590-789554db90fb%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/46f892d9-1c9f-495f-87ce-c90a49b241c9%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: mlockall
Which linux? On 31 March 2015 at 16:16, Cheten Dev cheten@carwale.com wrote: Hi, I am on linux server with elasticsearch 1.5.0 On Tue, Mar 31, 2015 at 8:07 AM, Mark Walkom markwal...@gmail.com wrote: A bit more info would be useful. What version of ES, what OS? On 30 March 2015 at 22:24, Chetan Dev cheten@carwale.com wrote: Hi, i have tried everything written on the document to set mlackall = true but nothing seems working here what am i missing here ? Thanks -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5748d9fb-120a-4521-8caa-62b780ea0594%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/5748d9fb-120a-4521-8caa-62b780ea0594%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/R41hW2QaL0w/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-92m9Qw4j4T8gwUV6%2BumkvbAOENTTsiupMurpGf8NZSw%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-92m9Qw4j4T8gwUV6%2BumkvbAOENTTsiupMurpGf8NZSw%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAG_NmL9T4dZeHUYpN0_5gwbfppL7%2BOJyNc6t3SQUpSe9N5oN2Q%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAG_NmL9T4dZeHUYpN0_5gwbfppL7%2BOJyNc6t3SQUpSe9N5oN2Q%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_5X8ubgUVPd6ZV%2Bh_EP1m72Noht4DOvMCysUieWrQp9Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Elastic Search - How to delete old records and improve performance
Hi Mark,
thanks for taking the time to reply
1- yes I believe all the data is in one index .. I had attached the
index_status output ... and a screenshot for the head_plugin output
2- I am considering the upgrade to latest version .. but would that impose
a risk (if any)?
thanks again
cheers!
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d5529d52-0a21-402a-8265-78da41b2b05f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
{
index: {
primary_size: 56.6gb
primary_size_in_bytes: 60849552973
size: 113.3gb
size_in_bytes: 121703182315
}
translog: {
operations: 0
}
docs: {
num_docs: 35606359
max_doc: 35640403
deleted_docs: 34044
}
merges: {
current: 0
current_docs: 0
current_size: 0b
current_size_in_bytes: 0
total: 0
total_time: 0s
total_time_in_millis: 0
total_docs: 0
total_size: 0b
total_size_in_bytes: 0
}
refresh: {
total: 212728
total_time: 45.6m
total_time_in_millis: 2737970
}
flush: {
total: 35680
total_time: 2.6m
total_time_in_millis: 156884
}
shards: {
0: [
{
routing: {
state: STARTED
primary: true
node: CvAQ3xeNRm-GuIzTkps_3A
relocating_node: null
shard: 0
index: dw_20130416
}
state: STARTED
index: {
size: 11.5gb
size_in_bytes: 12431838486
}
translog: {
id: 1366132941614
operations: 0
}
docs: {
num_docs: 7348576
max_doc: 7352282
deleted_docs: 3706
}
merges: {
current: 0
current_docs: 0
current_size: 0b
current_size_in_bytes: 0
total: 0
total_time: 0s
total_time_in_millis: 0
total_docs: 0
total_size: 0b
total_size_in_bytes: 0
}
refresh: {
total: 21265
total_time: 4.8m
total_time_in_millis: 288507
}
flush: {
total: 3568
total_time: 15.9s
total_time_in_millis: 15919
}
}
{
routing: {
state: STARTED
primary: false
node: sAz51CSkSfeiDeo1CYoMyw
relocating_node: null
shard: 0
index: dw_20130416
}
state: STARTED
index: {
size: 11.5gb
size_in_bytes: 12432400616
}
translog: {
id: 1366132941616
operations: 0
}
docs: {
num_docs: 7348576
max_doc: 7352094
deleted_docs: 3518
}
merges: {
current: 0
current_docs: 0
current_size: 0b
current_size_in_bytes: 0
total: 0
total_time: 0s
total_time_in_millis: 0
total_docs: 0
total_size: 0b
total_size_in_bytes: 0
}
refresh: {
total: 21282
total_time: 4.3m
total_time_in_millis: 262458
}
flush: {
total: 3568
total_time: 15.3s
total_time_in_millis: 15345
}
}
]
1: [
{
routing: {
state: STARTED
primary: true
node: CvAQ3xeNRm-GuIzTkps_3A
relocating_node: null
shard: 1
index: dw_20130416
}
state: STARTED
index: {
size: 11.5gb
size_in_bytes: 12453837813
}
translog: {
id:
Re: I want to load dashboard developed by another developer in kibana4 in my system using separate elasticsearch.It is possible in kibana 3 by using json file.is it possible in kibana 4?
This is currently not possible, but it's being worked on. See https://github.com/elastic/kibana/issues/2310, it might also have something to help in the meantime. On 31 March 2015 at 16:09, Priya G g.shanmugapriy...@gmail.com wrote: I want to load dashboard developed by another developer in kibana4 in my system . *For example*: I am using separate elastic search,kibana in my system and another developer also using seperate elastic search,kibana in his system.I want to view his kibana 4 dashboard using different data of my elastic search. In kibana 3 it is possible by loading json file . But in kibana 4 is it possible?Can anyone tell me?how to do? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/71045947-d47f-43e6-8d7f-fe6e3ae74efc%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/71045947-d47f-43e6-8d7f-fe6e3ae74efc%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_oFv%2BbdbcPLT5Rfdus9aXUnAdnUnJs7oFgjr5mY9Uk8A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Security Suggestion In Elasticsearch
Hi, I am trying to deploy my ES server in Digital ocean. But Digital ocean had some hacker attack as i didn't add any security to ES, So they blocked my droplet and gave me warning. So i tried to find out how can i secure my ES server. I have found out that Shield , using proxy or some other ways can be solution. But i am confused. What is the best option (to secure ES) i have right now. 1) I have one ES server (i may add more in future) 2) One server that run the java code. From this server i do all ES operation. 3) There are some more servers like postgresql, gearman but i think these are not relevant here. Thanks in advance. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4aa2f9f8-b4b7-4ce3-8ff3-66c3304d64a5%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Timestamps stored as seconds since Unix epoch
Though support may be coming -
https://github.com/elastic/elasticsearch/issues/10072
On 30 March 2015 at 09:16, Jean Marc Saffroy j...@scality.com wrote:
I thought I might have missed something, and it surprises me that the Unix
epoch isn't supported!
Well, I guess I'll have to work around that. Thanks!
JM
On Sun, Mar 29, 2015 at 10:21 PM, Mark Walkom markwal...@gmail.com
wrote:
Unix epoch isn't a supported format. Take a look at
http://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html
On 30 March 2015 at 02:12, Jean Marc Saffroy j...@scality.com wrote:
Here is a curl recreation, hopefully that will be clearer:
http://pastebin.com/DUhbpgze
JM
On Sun, Mar 29, 2015 at 4:18 PM, Jean Marc Saffroy j...@scality.com
wrote:
Of course I left a typo in my email: I do use the same field name
across mapping def, docs and queries, and it does not work.
JM
On Sun, Mar 29, 2015 at 4:17 PM, Jean Marc Saffroy j...@scality.com
wrote:
Hi all,
Not sure what I'm doing wrong, but I couldn't find a way to store my
docs with timestamps as seconds since the Unix epoch and query them
properly. I have my date/time field mapped like this:
start_time:{type: date },
I store documents like this:
{ start_time: 1427631731, ... }
And get no result when I do a range query like this:
query: {
range: { @timestamp: { gte: 2015-03-29 }}
}
I have good results when I store timestamps as strings in ISO format.
Is that supposed to work? Is there a specific date parser I should use?
JM
--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALZqptb5xZNgok4hiszEF9JP_KRuk6y%3DwdqfbjR0xxBesK4dBA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CALZqptb5xZNgok4hiszEF9JP_KRuk6y%3DwdqfbjR0xxBesK4dBA%40mail.gmail.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-v%3DM6ZVK1p%3Desb40QU_Va5beBpCVusGpKeoDpcht9c2Q%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-v%3DM6ZVK1p%3Desb40QU_Va5beBpCVusGpKeoDpcht9c2Q%40mail.gmail.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALZqptbGPcQDxUaFHfRfttdGyzFKVUogZEenj3nAPxOJYWitCw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CALZqptbGPcQDxUaFHfRfttdGyzFKVUogZEenj3nAPxOJYWitCw%40mail.gmail.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X882n7C8-C5T8DQOgqf36PG%3D6ZaJaQ6%3DxWFGVEOVmK%3DvQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
[ANN] Elasticsearch Twitter River plugin 2.5.0 released
Heya, We are pleased to announce the release of the Elasticsearch Twitter River plugin, version 2.5.0. The Twitter river indexes the public twitter stream, aka the hose, and makes it searchable. https://github.com/elastic/elasticsearch-river-twitter/ Release Notes - elasticsearch-river-twitter - Version 2.5.0 Fix: * [82] - Potential NPE when closing a river (https://github.com/elastic/elasticsearch-river-twitter/issues/82) Update: * [102] - Update Twitter4J to 4.0.3 (https://github.com/elastic/elasticsearch-river-twitter/issues/102) * [101] - Update to elasticsearch 1.5.0 (https://github.com/elastic/elasticsearch-river-twitter/issues/101) Issues, Pull requests, Feature requests are warmly welcome on elasticsearch-river-twitter project repository: https://github.com/elastic/elasticsearch-river-twitter/ For questions or comments around this plugin, feel free to use elasticsearch mailing list: https://groups.google.com/forum/#!forum/elasticsearch Enjoy, -The Elasticsearch team -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/551a4b0b.884bb40a.1c6f.ac31SMTPIN_ADDED_MISSING%40gmr-mx.google.com. For more options, visit https://groups.google.com/d/optout.
Re: Security Suggestion In Elasticsearch
The rule is not new. Do not expose Elasticsearch to the public internet, just like Postgresql and Gearman. Jörg On Tue, Mar 31, 2015 at 8:45 AM, Shohedul Hasan sha...@qianalysis.com wrote: Hi, I am trying to deploy my ES server in Digital ocean. But Digital ocean had some hacker attack as i didn't add any security to ES, So they blocked my droplet and gave me warning. So i tried to find out how can i secure my ES server. I have found out that Shield , using proxy or some other ways can be solution. But i am confused. What is the best option (to secure ES) i have right now. 1) I have one ES server (i may add more in future) 2) One server that run the java code. From this server i do all ES operation. 3) There are some more servers like postgresql, gearman but i think these are not relevant here. Thanks in advance. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4aa2f9f8-b4b7-4ce3-8ff3-66c3304d64a5%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/4aa2f9f8-b4b7-4ce3-8ff3-66c3304d64a5%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFZZbQ9x6MkdUC-TkCUHpfSh3Sieh1Rk_PE9YuwwSQvuw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Security Suggestion In Elasticsearch
Shield would be the best. But you can also use iptables and nginx to secure and proxy things. On 31 March 2015 at 17:45, Shohedul Hasan sha...@qianalysis.com wrote: Hi, I am trying to deploy my ES server in Digital ocean. But Digital ocean had some hacker attack as i didn't add any security to ES, So they blocked my droplet and gave me warning. So i tried to find out how can i secure my ES server. I have found out that Shield , using proxy or some other ways can be solution. But i am confused. What is the best option (to secure ES) i have right now. 1) I have one ES server (i may add more in future) 2) One server that run the java code. From this server i do all ES operation. 3) There are some more servers like postgresql, gearman but i think these are not relevant here. Thanks in advance. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4aa2f9f8-b4b7-4ce3-8ff3-66c3304d64a5%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/4aa2f9f8-b4b7-4ce3-8ff3-66c3304d64a5%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-Z6qH%2Bmh9Z7GB2UMChB6bwBrt-6tU_uVJWRGCy9c_gQg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[ANN] Elasticsearch Wikipedia River plugin 2.5.0 released
Heya, We are pleased to announce the release of the Elasticsearch Wikipedia River plugin, version 2.5.0. The Wikipedia River plugin allows index wikipedia. https://github.com/elastic/elasticsearch-river-wikipedia/ Release Notes - elasticsearch-river-wikipedia - Version 2.5.0 Update: * [47] - Update to elasticsearch 1.5.0 (https://github.com/elastic/elasticsearch-river-wikipedia/issues/47) Issues, Pull requests, Feature requests are warmly welcome on elasticsearch-river-wikipedia project repository: https://github.com/elastic/elasticsearch-river-wikipedia/ For questions or comments around this plugin, feel free to use elasticsearch mailing list: https://groups.google.com/forum/#!forum/elasticsearch Enjoy, -The Elasticsearch team -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/551a4f5a.e65ab40a.6f46.2f66SMTPIN_ADDED_MISSING%40gmr-mx.google.com. For more options, visit https://groups.google.com/d/optout.
Re: A cautionary tale about index autocreation
The problem with action.auto_create_index: false is that it disables index templates... So it is basically useless if you are using one -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d0a69a36-6497-4a51-a144-48f31a6b2312%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Very regular disconnect and recover - every 2 hours
Both clusters have the following settings so, if that's related, I think there must be another contributing factor... discovery.zen.fd.ping_interval : 1s, discovery.zen.fd.ping_timeout : 60s, discovery.zen.fd.ping_retries : 3, On 31 March 2015 at 07:32, Mark Walkom markwal...@gmail.com wrote: You can try winding out the timeouts, see http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#fault-detection On 31 March 2015 at 16:57, Neil Andrassy neil.andra...@thefilter.com wrote: It's probably something like that, but it only seems to be a problem with the more up to date version of ES. I'm keen to work out if there's a configuration option I can tweak in 1.4.4 to make ES more robust in this scenario or whether there's an issue around recovering dropped TCP connections between nodes in more recent versions. On Tuesday, 31 March 2015 03:33:18 UTC+1, Mark Walkom wrote: It's not the VPN reconnecting is it? On 31 March 2015 at 01:32, Neil Andrassy neil.a...@thefilter.com wrote: Hi, I have two independent clusters running across more or less the same machines. They're split across a pretty high bandwidth and relatively low latency VPN link. One cluster is running v1.0.1 and seems to stay up all the time. The other cluster is currently running 1.4.4 (and was running 1.4.2 before that) and seems to disconnect like clockwork every two hours. The disconnect of the nodes on one side of the link is brief, they rejoin and the recovery proceeds as normal. Any ideas what might cause this? Could it be data related? The newer cluster has more indexes shards than the old, but the co-ordinators (3 of / min master count 2) don't seem particularly stressed. Any thoughts on what, specifically to look for or whether any particular setting or code change might make the cluster more susceptible to disconnect when there's a minor / brief network connectivity blip? (and yes, I know multi-site isn't a recommended configuration - there are other challenges for us with the tribe node approach too, though :( ) Thanks in advance for any ideas or insight. N -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2% 40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/b00b8bda-9238-47e8-b0f2-3d4d6751b3c2%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/793f13a8-9ca8-4d86-b194-47b4e9cd5125%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/793f13a8-9ca8-4d86-b194-47b4e9cd5125%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/FLsYRpcADEk/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_fbZXbP_iY4ZpJODXPmumh15CntRT6S4HaJdrvqv593A%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_fbZXbP_iY4ZpJODXPmumh15CntRT6S4HaJdrvqv593A%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Neil Andrassy | CTO | The Filter phone | +44 (0)1225 588 004 skype | andrassynp -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABpTWLMn8k_hbuRkh3s-ZmWqZd9s1POHzMQEuFFxWVugS5%3Dxng%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Default template mapping at type level
Hallo, I have 6 types that I would like to keep in the same time-series index. 5 of these types are very similar and so I have defined the common fields in the _default_ mapping. The 6th is very different and shares very few fields with the other types: but since the _default_ mapping applies to all types in the index I have a lot of fields in the mapping for type6 that I don't need or want. Is there a way around this? I'm looking for a default type that I can derive from (rather than a default mapping at index level), but this does not seem to exist. Is this correct or is there a workaround? Regards, Andrew -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9f026249-dd48-4769-b435-7cdfc5517905%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
How to merge elasticsearch 1.4.2 data with already running elasticsearch 1.4.4?
I am having some data in elasticsearch 1.4.2.i am also having some data in elasticsearch 1.4.4. Now i want to take data stored in elasticsearch 1.4.2 and i have to store that data in elasticsearch1.4.4. Can anyone tell me how to do? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e7d87a86-03a0-4f34-a97c-8b28dc6e5ac3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Scoring and explain
FYI ExplainableSearchScript interface is back in 1.4 branch. Le mercredi 19 novembre 2014 04:04:05 UTC+1, Nick Tarleton a écrit : On Thursday, November 6, 2014 3:30:26 AM UTC-8, Guillaume H wrote: Hi guys, I'm developing a scoring plugin and I need to display some information about score computation. I need to display the reason(s) why I have this score, which criteria matched etc.. I currently implement ExplainableSearchScript interface and use Explain feature to retrieve that info. I think it's not the best way to get scoring reason because of overload during hit fetch and because this interface is not in 1.4 branch anymore. My first thought is to add a new field in response but it really doesn't seems obvious ... So what can I do to get scoring information efficiently ? I also have this issue. I've been using ExplainableSearchScript to add information computed in the scoring script to the explanation for debugging, but it's gone in 1.4. Is there any other way (doesn't have to be efficient) to get information out of a script into the search response? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/478f8ffc-a9dd-40fc-9b84-a0b292c15a37%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
IFrame options
Hi, I'm displaying Kibana graphs in another system using the generated iframe, but I need to get rid of both the Kibana loading screen as well as error messages when no data is found. I only need to display graphs. Are there any options for doing that? Thanks, Ole Hedegaard -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/111c01cd-78da-46c0-8140-c5895aeb85a9%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[ANN] Elasticsearch RabbitMQ River plugin 2.5.0 released
Heya, We are pleased to announce the release of the Elasticsearch RabbitMQ River plugin, version 2.5.0. The RabbitMQ River plugin allows index bulk format messages into elasticsearch.. https://github.com/elastic/elasticsearch-river-rabbitmq/ Release Notes - elasticsearch-river-rabbitmq - Version 2.5.0 Update: * [94] - Replace deprecated `JsonFactory#createJsonParser` by `JsonFactory#createParser` (https://github.com/elastic/elasticsearch-river-rabbitmq/issues/94) * [93] - Update to elasticsearch 1.5.0 (https://github.com/elastic/elasticsearch-river-rabbitmq/issues/93) * [91] - Deprecate `replication` parameter (https://github.com/elastic/elasticsearch-river-rabbitmq/issues/91) Issues, Pull requests, Feature requests are warmly welcome on elasticsearch-river-rabbitmq project repository: https://github.com/elastic/elasticsearch-river-rabbitmq/ For questions or comments around this plugin, feel free to use elasticsearch mailing list: https://groups.google.com/forum/#!forum/elasticsearch Enjoy, -The Elasticsearch team -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/551a6989.0817b40a.64b6.27b5SMTPIN_ADDED_MISSING%40gmr-mx.google.com. For more options, visit https://groups.google.com/d/optout.
Problems upgrading to 1.5.0
We just encountered some mysterious problems when upgrading from 1.1.1 to
1.5.0.
The cluster consists of three machines, two data nodes and one master-only
node. It hosts 86 indices which each has one replica.
I stopped writes, did a snapshot and stopped the entire cluster before I
upgraded the nodes and restarted them. The system came up and quickly
turned yellow, but it refused to become green. it failed to recover a
number of shards. The errors I got in the logs looked like this (there were
a lot):
[2015-03-31 07:33:39,704][WARN ][indices.cluster ] [NODE1]
[signal_bin][0] sending failed shard after recovery failure
org.elasticsearch.indices.recovery.RecoveryFailedException:
[signal_bin][0]: Recovery failed from
[NODE2][rpXLVgS8Qw2jgimXNYKn_A][NODE2][inet[/IP2:9300]]{aws_availability_zone=us-east-1d,
max_local_storage_nodes=1} into
[NODE1][tdXdf0MeS62DIO0KFZX-Rg][NODE1][inet[/IP1:9300]]{aws_availability_zone=us-east-1b,
max_local_storage_nodes=1}
at
org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:274)
at
org.elasticsearch.indices.recovery.RecoveryTarget.access$700(RecoveryTarget.java:69)
at
org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:550)
at
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.elasticsearch.transport.RemoteTransportException:
[NODE2][inet[/IP2:9300]][internal:index/shard/recovery/start_recovery]
Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
[signal_bin][0] Phase[1] Execution failed
at
org.elasticsearch.index.engine.InternalEngine.recover(InternalEngine.java:839)
at org.elasticsearch.index.shard.IndexShard.recover(IndexShard.java:684)
at
org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:125)
at
org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:49)
at
org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:146)
at
org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:132)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
at
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by:
org.elasticsearch.indices.recovery.RecoverFilesRecoveryException:
[signal_bin][0] Failed to transfer [11] files with total size of [1.4mb]
at
org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:413)
at
org.elasticsearch.index.engine.InternalEngine.recover(InternalEngine.java:834)
... 10 more
Caused by: org.elasticsearch.transport.RemoteTransportException:
[NODE1][inet[/IP1:9300]][internal:index/shard/recovery/clean_files]
Caused by: org.elasticsearch.indices.recovery.RecoveryFailedException:
[signal_bin][0]: Recovery failed from
[NODE2][rpXLVgS8Qw2jgimXNYKn_A][NODE2][inet[/IP2:9300]]{aws_availability_zone=us-east-1d,
max_local_storage_nodes=1} into
[NODE1][tdXdf0MeS62DIO0KFZX-Rg][NODE1][inet[/IP1:9300]]{aws_availability_zone=us-east-1b,
max_local_storage_nodes=1} (failed to clean after recovery)
at
org.elasticsearch.indices.recovery.RecoveryTarget$CleanFilesRequestHandler.messageReceived(RecoveryTarget.java:443)
at
org.elasticsearch.indices.recovery.RecoveryTarget$CleanFilesRequestHandler.messageReceived(RecoveryTarget.java:389)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
at
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.elasticsearch.ElasticsearchIllegalStateException: local
version: name [_yor.si], length [363], checksum [1jnqbzx], writtenBy [null]
is different from remote version after recovery: name [_yor.si], length
[363], checksum [null], writtenBy [null]
at
org.elasticsearch.index.store.Store.verifyAfterCleanup(Store.java:645)
at org.elasticsearch.index.store.Store.cleanupAndVerify(Store.java:613)
at
Re: ES/Lucene eating up entire memory!
Thanks Uwe. As I mentioned earlier, I did guess that VIRT doesn't indicate RAM consumption. What I am concerned about is the 3rd row which shows memory and indicates that out of the total 50g, 43g is in use. Once this number crosses 45g, my other databases start behaving badly. Problem is, even after I kill all the processes, this doesn't go down. (Attaching snapshot of top after killing all processes). Right now what I do is reboot the system every three days which is the time it takes to gradually fill the memory with something (I have no clue what that is). Though I think the max file descriptors wouldn't be the culprit for this? I haven't changed that yet. On Mon, Mar 30, 2015 at 3:19 AM, Uwe Schindler uwe.h.schind...@gmail.com wrote: You should read: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html Maybe this allows you to figure out what's going on! VIRT means nothing about consumption, you should look at RES. Thanks, Uwe Am Sonntag, 29. März 2015 22:23:00 UTC+2 schrieb Yogesh: Hi, I have a single node ES setup (50GB memory, 500GB disk, 4 cores) and I run the Twitter river on it. I've set the ES_HEAP_SIZE to 5g. However, when I do top, the ES process shows the VIRT memory to be around 34g. That would be I assume the max mapped memory. The %MEM though always hovers around 10% However, within a few days post-reboot, the memory used keeps going up. From 10g to almost 50g (as shown in the third line) because of which my other dbs start behaving badly. Below is the snapshot of top. Despite the fact that VIRT and %MEM still hover around the same 34g and 10% respectively. Please help me understand where is my memory going over time! My one guess is that Lucene is eating it up. How do I remedy it? Thanks-in-advance! https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AN8/aq8-wxm2bBg/s1600/top.png -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/kTDNDJwxOzA/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADM0w%3Di5%2B_Cd-swP3f58jm-cE%2B7ULsq6QwfafQjsmka47h3fkg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Script to configure Kibana 4 index pattern for logstash, import saved searches, visualizations dashboards
Hi, Is there a script available to configure Kibana 4. I'm starting Elasticsearch, Logstash Kibana 4 inside Docker. I'm looking for a way to automatically configure: - the Kibana 4 index pattern for logstash - import Kibana saved searches, visualizations dashboards -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/284e6911-f34b-46b6-904f-ff7842bc411f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: EC2 client node cluster discovery
Seems like not many other people have this problem... In my case this was only a problem using unicast (which cloud-aws does), so I changed the cloud plugin to work for my case. The file that needed to be changed was AwsEc2UnicastHostsProvider.java. Look for UnicastZenPing.LIMIT_PORTS_COUNT. That const is hard coded to 1 in ES v1.5. I changed it to read a config value or use the const if no config was found. This worked fine for me with the es-1.5 2.5.0 version of the plugin. Seems like anyone running more than one node on a server not using multicast could have this issue. On Friday, March 27, 2015 at 11:02:05 PM UTC-4, Jerry Pattenaude wrote: Still trying to figure out a way past this issue but I actually know what's going on now. The first cluster takes port 9300 and anything looking for a different cluster on that same machine stops when it sees 9300 is in use by a different cluster. Elasticsearch used to scan the port range but they removed that code because it's expensive to check so many ports. There is a pull request to make the number of ports to scan configurable. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ccafba78-3c68-40e8-9e0b-0f6618517abc%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: How getting document match rate?
Hi guys,
The Explain api returns coord in bool query. This could work.
On Saturday, March 28, 2015 at 6:33:53 PM UTC+2, Terra Sacer wrote:
Hello everyone,
For example my data
[
{ id : 1, type : article, title : About the Java Technology
},
{ id : 2, type : article, title : How does ElasticSearch
work },
{ id : 3, type : article, title : How to get the count of
results ElasticSearch }
]
When we run the following command
GET myindex/article/_search
{
query: {
bool: {
should: [
{
match: {
title: how count elasticsearch
}
}
]
}
}
}
We get these results
{
took: 776,
timed_out: false,
_shards: {
total: 5,
successful: 5,
failed: 0
},
hits: {
total: 2,
max_score: 0.16608897,
hits: [
{
_index: myindex,
_type: article,
_id: 3,
_score: 0.16608897,
_source: {
title: How to get the count of results ElasticSearch
}
},
{
_index: myindex,
_type: article,
_id: 2,
_score: 0.05758412,
_source: {
title: How does ElasticSearch work
}
}
]
}
}
Now the question is: Contains a maximum term(how, count and elasticsearch)
the third document, match 100%
*How can get this information from ElasticSearch?*
I can benefit from the score and max_scor information when query match is
100% but when it is not 100% match?
Please help me :)
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8546ca39-30f9-4159-b205-99263034ac13%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[ANN] Elasticsearch CouchDB River plugin 2.5.0 released
Heya, We are pleased to announce the release of the Elasticsearch CouchDB River plugin, version 2.5.0. The CouchDB River plugin allows to hook into couchdb _changes feed and automatically index it into elasticsearch.. https://github.com/elastic/elasticsearch-river-couchdb/ Release Notes - elasticsearch-river-couchdb - Version 2.5.0 Update: * [95] - Update to elasticsearch 1.5.0 (https://github.com/elastic/elasticsearch-river-couchdb/issues/95) Issues, Pull requests, Feature requests are warmly welcome on elasticsearch-river-couchdb project repository: https://github.com/elastic/elasticsearch-river-couchdb/ For questions or comments around this plugin, feel free to use elasticsearch mailing list: https://groups.google.com/forum/#!forum/elasticsearch Enjoy, -The Elasticsearch team -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/551a956d.430bb40a.4545.54a4SMTPIN_ADDED_MISSING%40gmr-mx.google.com. For more options, visit https://groups.google.com/d/optout.
Re: Health Status Red(Accidentally started another node)
Thank you for the solution ! A quick clarification though. So I am assuming that on restarting the second node, the cluster will automatically come back to green, right? Also, is it possible that instead of restarting the second node(which was accidental and not really needed), I can force shard allocation of unassigned shard to my original node? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02b1b570-a7e9-42b8-af25-45714612fa6f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Reindex from the existing index
If you Search on the web site for reindex, you will find this: http://www.elastic.co/guide/en/elasticsearch/guide/current/reindex.html David Le 1 avr. 2015 à 06:15, Vladi Feigin vla...@liveperson.com a écrit : Hi, Is there a way to build a new index from the existing index ? Thank you, Vladi -- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9f51ce24-b826-4bcb-8d2c-ab18732e8016%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/A8B19DE8-AE00-4482-BB59-09993806309D%40pilato.fr. For more options, visit https://groups.google.com/d/optout.
Re: Nested object under path [messages] is not of nested type
Hi,
You put mapping of message type while you query on thread type.
Since thread type isn't defined, you get the error.
Masaru
On April 1, 2015 at 01:06:51, Daniel Buckle (danielbuckl...@gmail.com) wrote:
I am having a lot of issues with the nested type mapping in Elasticsearch, I
have ran this to create my index:
curl -XPOST 'http://localhost:9200/thread_and_messages' -d
'{mappings : {
message: {
properties: {
messages: {
type: nested,
include_in_parent: true,
properties: {
message_id: {type: string},
message_text: {type: string},
message_nick: {type: string}
}
}
}
}
}}'
Then this is how I've indexed a document:
curl -XPUT 'http://localhost:9200/thread_and_messages/thread/1' -d
'{
thread_id:2,
thread_name:Windows,
created:Wed Mar 25 2015,
first_nick:Admin,
messages:[
{message_id:5, message_text: Pc with a mouse,
message_nick:Admin},
{message_id:6, message_text:Keyboard, message_nick:Admin},
{message_id:7, message_text:iPhone, message_nick:Admin},
{message_id:8, message_text:Gym, message_nick:Admin}]
}'
This is my query:
curl -XGET 'http://localhost:9200/thread_and_messages/thread/_search' -d
'{query: {
bool: {
must: [
{match: {thread_name: windows}},
{nested: {
path: messages, query: {
bool: {
must: [{
match: {messages.message_text: gym}
}]
}
}
}}
]}
}
}'
I am receiving this error, even though I have clearly mapped messages as a
nested type:
QueryParsingException[[thread_and_messages] [nested]
nested object under path [messages] is not of nested type
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9c1395b9-99aa-47e8-900b-a819323275ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.551b4d21.74b0dc51.3b55%40citra-2.local.
For more options, visit https://groups.google.com/d/optout.
Re: Health Status Red(Accidentally started another node)
Restart the other node, let it join and then bring the cluster back to green. Then either 1) add replicas so each node has the data and remove the old node and set replicas to 0, or 2) disable allocation and move the shards off the other node to the main one manually. On 1 April 2015 at 11:08, Shomo shoumitrados...@gmail.com wrote: Today, I accidentally started another node on a different machine on the same LAN which resulted in my first ES instance adding this new node to the cluster. I killed this second node with SIGKILL and restarted my original ES instance. However, on checking the health status of the various indices, a lot of them show red. The one's that show a red status have no. of replicas set to 0. The ones that are green had the number of replicas set to 1. Is there any way I can cleanly get the health back to Green without deleting and re-indexing. I am using the default setting of having 5 shards per index. Also, for each index that has its' status as red has exactly one shard that is unassigned. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1548aca9-50b4-42be-8b08-6be1a4d99bff%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/1548aca9-50b4-42be-8b08-6be1a4d99bff%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_PrYoYFwysLL5Nr%3Dh4sfT0wRo-y8fcQPiDNNpTpOTLXw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Elastic Search - How to delete old records and improve performance
You could do a few things, from my perspective. (Hope this hits some idea's you like) - upgrade, never hurts :) as long as you read the release notes to make sure nothing your depending on. - Add some filters to your existing queries to exclude unneeded data - you can create filter aliases which lets you set up predefined alias with filtered terms in-case you can't delete or loose data - http://www.elastic.co/guide/en/elasticsearch/reference/1.3/indices-aliases.html - I would rotate your index so all new data would go in to a new one while the archive can wither and die - Maybe rotate it on a daily basis or a point of time that works for your data sets you can still have basic aliases to summarize multiple days depending on your query needs - Use a project called curator to purge old data on a routine basis (cron) - https://github.com/elastic/curator/wiki - Finally if you need to massage the data you can look at this project - https://github.com/taskrabbit/elasticsearch-dumpit does full dumps and also Query Style dumpes. - Dump the whole index - Then dump with a query - Delete and import your query dump to a Date based index that can be purged later by curator - Oh just thought of this one, increase the number of shards and ES nodes, I see you have only 2 - To do this will require you to export and import the data again. (at least for the old data On Tuesday, March 31, 2015 at 4:13:07 PM UTC-4, AALISHE wrote: mkBig ... thanks for the suggestion ... but how do I exclude the things I dont need ? cheers! -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4228dbd3-84a1-41ba-bed9-3bcf4cb70033%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
How to achieve ELK High availability
Hi Exerts, Need your valuable suggestions here . I have ELK on a single windows instance and I want to make it high available . I mean if one machine goes down second will take up the whole load, like clustering. Can you suggest how I can achieve this. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/38b05ef0-56e7-4590-b5e1-aaa64ab68150%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Reindex from the existing index
Hi, Is there a way to build a new index from the existing index ? Thank you, Vladi -- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9f51ce24-b826-4bcb-8d2c-ab18732e8016%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Health Status Red(Accidentally started another node)
It should, only way to know is to try. You can try to force allocation if that node has them there. On 1 April 2015 at 12:39, Shomo shoumitrados...@gmail.com wrote: Thank you for the solution ! A quick clarification though. So I am assuming that on restarting the second node, the cluster will automatically come back to green, right? Also, is it possible that instead of restarting the second node(which was accidental and not really needed), I can force shard allocation of unassigned shard to my original node? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02b1b570-a7e9-42b8-af25-45714612fa6f%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/02b1b570-a7e9-42b8-af25-45714612fa6f%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-EJs2hvkUoiu4B9STgxuYX8%2BVvYdBJ95VFiTku0WeAEw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: ElasticSearch date format
I am speculating here that you should use the ISO 8601 format for dates
since this is quite common and has a timezone which yours lacks
(introducing ambiguity).
The T is a common separator between the date and time parts.
On 31 March 2015 at 14:26, phani.nadimi...@goktree.com wrote:
Hi All,
when i insert date to elastic search in the format of -MM-DD
hh:mm:ss from java api getting date format exception.but when i use
-MM-DDThh:mm:ss between them it is inserted fine.please explain me is T
here elasticsearch standard format?
please explain me functionality of T in date.i used the default mapping
of elasticsearch i didn't customized any thing.
mapping:
created: {
type: date,
format: dateOptionalTime
},
Thanks
phani
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/94b29fca-637d-4d73-84fe-6af4b6300611%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/94b29fca-637d-4d73-84fe-6af4b6300611%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BawJXDAs175DFWH5T5z5%2B-UvJgppKjvqPRBehzMX2R%2Bz7A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: is _id of document affects on scoring?
Thanks for your response.
it is completely confusing for me why elasticsearch was configured for
large data by default instead of development and developers should find
these tiny tips under thousands of documentations.
On Tuesday, 31 March 2015 17:54:00 UTC+4:30, Adrien Grand wrote:
The difference is probably due to the fact that your documents ended on
different shards where doc frequencies are different. You should be able to
make it a bit better by using distributed frequencies.
https://www.elastic.co/blog/understanding-query-then-fetch-vs-dfs-query-then-fetch
On Tue, Mar 31, 2015 at 3:18 PM, Mohammad.R esmailzadeh
esmai...@gmail.com javascript: wrote:
I add two same documents the only different thing is _id of documents (I
restart scenario for each of them and I do not add them sequentially. to be
sure my test is correct)
one of them changes order of result of this query and one of them does
not:
GET index_for_test/business/_search
{
query: {
multi_match: {
query: italian,
type:most_fields,
fields: [ name^10, categories ]
}
}
}
i think it should be bug
my original question was:
https://github.com/elastic/elasticsearch/issues/10341
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5d0eba0f-0809-4a68-b247-0640addb9119%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/5d0eba0f-0809-4a68-b247-0640addb9119%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
Adrien
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b78aa9d8-2e22-40a3-b36b-5817fd2567ef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Try to make es-hadoop run
Debugging Hadoop is ... tricky. Most likely the parameter you are passing (es.version) is incorrect - there's no such Elasticsearch version. 2.1.0.Beta3 is the connector, es-hadoop version. es.version indicates the version of Elasticsearch itself. My advice is to first start with basic steps and defaults as described in the docs [1]. As for the debug part itself, you have to look into your node/cluster logs, in particular the yarn/node ones to see the underlying container exception. Whatever message/exception is thrown by the underlying container, it does not bubble up (it is always wrapped by YARN itself) and thus one has to manually track it down... [1] http://www.elastic.co/guide/en/elasticsearch/hadoop/2.1.Beta/ey-usage.html On 3/31/15 5:50 PM, stéphane Verdy wrote: Hi all, HDFS 2.6.0.2.2.0.0 Apache Hadoop Distributed File System Es-Hadoop : 2.1.0.Beta3 OS : Centos 6 When i start elasticsearch (hadoop jar elasticsearch-yarn-2.1.0.Beta3.jar -start es.version=2.1.0.Beta3), the process end FINISHED/FAILED and the only log i found is : 2015-03-31 13:29:57,792 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(283)) - at java.lang.Thread.run(Thread.java:744) 2015-03-31 13:29:57,793 WARN launcher.ContainerLaunch (ContainerLaunch.java:call(338)) - Container exited with a non-zero exit code 127 No ERROR log only WARN... Can'i set the logger in DEBUG mode (how) ? Are there any other log files ? Thank Stephane -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com mailto:elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/82bea02f-7356-4bab-8895-00b26842352a%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/82bea02f-7356-4bab-8895-00b26842352a%40googlegroups.com?utm_medium=emailutm_source=footer. For more options, visit https://groups.google.com/d/optout. -- Costin -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/551AB786.1040107%40gmail.com. For more options, visit https://groups.google.com/d/optout.
Problems with combination of attributes in Elasticsearch
Hi all, When designing an application I would like to know whether ElasticSearch is a suitable tool for implementing it (and how to do it). Any advice will be much appreciated! My application needs to store (many) documents, each document being represented as a sequence of words. I want also to associate information with each word. For example, suppose I want to associate the word length with each word. So I would have something like this: The house is yellow 3 5 2 6 Now, I would like to perform queries such as for instance give me words with length 2, followed by the word 'yellow'. In a relational database I would store the word forms and lengths as different attributes, for instance: Word Length N the 3 1 house 5 2 is 2 3 yellow 6 4 (where N is the position of the word) and in SQL I would make something like this: SELECT word, N1 as N FROM documents WHERE (word=”yellow” AND N1 in (SELECT N2 as N FROM documents WHERE length=2 AND (N1-N2=1 OR N2-N1=1) ) ) I'm struggling to implement this same functionality into ElasticSearch. I've read online manuals and the reference book, but I'm unable to figure out how to do this with ES. So any advice from your part will be very much appreciated. Take into account: - The database will have many attributes associated with words, and I'll need to query for any combination of them. - Those attributes are pre-computed and loaded into the database offline. Thank you! -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1697f5ad-e00b-4777-a7f9-8239e9c64e91%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Elastic Search - How to delete old records and improve performance
mkBig ... thanks for the suggestion ... but how do I exclude the things I dont need ? cheers! -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/454830bc-a9c5-4803-ae95-8b37f65fedbf%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: is _id of document affects on scoring?
On Tue, Mar 31, 2015 at 4:13 PM, Mohammad.R esmailzadeh esmailza...@gmail.com wrote: it is completely confusing for me why elasticsearch was configured for large data by default instead of development and developers should find these tiny tips under thousands of documentations. I'm sorry you had a bad experience. I don't really have a good answer for you here: the right value for a particular setting depends on your usage of elasticsearch and while we strive at having good defaults, they can't always be right for everyone. In case you missed it, there is a free book about elasticsearch ( http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html) which you can read and dives you into how elasticsearch works and how to use it so that you don't have to crawl the reference API every time that you start using a new API. -- Adrien -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAgPajZ5Qn4_RFnJ1Y6KxfAWg8i2d8C2u54OYe_QgEss4w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
parent-child document relation problem
Hi all, could you help me with little problem regarding parent-child documents relation? I use elasticsearch v1.4.4. Considering JSON, I have objects, each of them contains an array of sub-objects. Sub-objects contain some text fields. I need to maintain full-text-search on these objects and construct snippets. I need highlighting for building snippets. If I use nested objects, highlighting does not deal with them. Therefore, I use Parent-Child relationships. Now I need to retrieve Parent-documents, which children match the query_string. Furthermore, I need to get highlighted fields of matched children and associate each one(each child) with corresponding parent to construct snippets in my application. Is it possible to accomplish my goal in one query? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5e018b6c-8354-4600-822b-8956eb530237%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
A problem: ClusterBlockException : blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
This error occurred when i restart cluster. Debug log: [2015-04-01 11:25:09,743][DEBUG][node ] [es_node_4_1] using home [/home/nat/elasticsearch-1.1.2], config [/home/nat/elasticsearch-1.1.2/config], data [[/home/nat/esdata.d, /natlog1/nat/esdata.d, /natlog4/nat/esdata.d, /natlog5/nat/esdata.d, /natlog6/nat/esdata.d, /natlog7/nat/esdata.d, /natlog8/nat/esdata.d]], logs [/home/nat/elasticsearch-1.1.2/logs], work [/home/nat/elasticsearch-1.1.2/work], plugins [/home/nat/elasticsearch-1.1.2/plugins] [2015-04-01 11:25:09,765][INFO ][plugins ] [es_node_4_1] loaded [], sites [head, bigdesk] [2015-04-01 11:25:09,775][DEBUG][common.compress.lzf ] using [UnsafeChunkDecoder] decoder [2015-04-01 11:25:09,787][DEBUG][env ] [es_node_4_1] using node location [[/home/nat/esdata.d/elasticsearch_log/nodes/0, /natlog1/nat/esdata.d/elasticsearch_log/nodes/0, /natlog4/nat/esdata.d/elasticsearch_log/nodes/0, /natlog5/nat/esdata.d/elasticsearch_log/nodes/0, /natlog6/nat/esdata.d/elasticsearch_log/nodes/0, /natlog7/nat/esdata.d/elasticsearch_log/nodes/0, /natlog8/nat/esdata.d/elasticsearch_log/nodes/0]], local_node_id [0] [2015-04-01 11:25:09,825][INFO ][node ] [es_node_4_2] version[1.1.2], pid[53749], build[e511f7b/2014-05-22T12:27:39Z] [2015-04-01 11:25:09,826][INFO ][node ] [es_node_4_2] initializing ... [2015-04-01 11:25:09,826][DEBUG][node ] [es_node_4_2] using home [/home/nat/elasticsearch-1.1.2], config [/home/nat/elasticsearch-1.1.2/config], data [[/home/nat/esdata.d, /natlog2/nat/esdata.d, /natlog3/nat/esdata.d, /natlog9/nat/esdata.d, /natlog10/nat/esdata.d, /natlog11/nat/esdata.d, /natlog12/nat/esdata.d, /natlog13/nat/esdata.d]], logs [/home/nat/elasticsearch-1.1.2/logs], work [/home/nat/elasticsearch-1.1.2/work], plugins [/home/nat/elasticsearch-1.1.2/plugins] [2015-04-01 11:25:09,862][INFO ][plugins ] [es_node_4_2] loaded [], sites [head, bigdesk] [2015-04-01 11:25:09,879][DEBUG][common.compress.lzf ] using [UnsafeChunkDecoder] decoder [2015-04-01 11:25:09,901][DEBUG][env ] [es_node_4_2] using node location [[/home/nat/esdata.d/elasticsearch_log/nodes/1, /natlog2/nat/esdata.d/elasticsearch_log/nodes/1, /natlog3/nat/esdata.d/elasticsearch_log/nodes/1, /natlog9/nat/esdata.d/elasticsearch_log/nodes/1, /natlog10/nat/esdata.d/elasticsearch_log/nodes/1, /natlog11/nat/esdata.d/elasticsearch_log/nodes/1, /natlog12/nat/esdata.d/elasticsearch_log/nodes/1, /natlog13/nat/esdata.d/elasticsearch_log/nodes/1]], local_node_id [1] [2015-04-01 11:25:11,258][DEBUG][threadpool ] [es_node_4_1] creating thread_pool [generic], type [cached], keep_alive [30s] [2015-04-01 11:25:11,286][DEBUG][threadpool ] [es_node_4_1] creating thread_pool [index], type [fixed], size [32], queue_size [200] [2015-04-01 11:25:11,294][DEBUG][threadpool ] [es_node_4_1] creating thread_pool [bulk], type [fixed], size [20], queue_size [32] [2015-04-01 11:25:11,295][DEBUG][threadpool ] [es_node_4_1] creating thread_pool [get], type [fixed], size [32], queue_size [1k] [2015-04-01 11:25:11,296][DEBUG][threadpool ] [es_node_4_1] creating thread_pool [search], type [fixed], size [96], queue_size [1k] [2015-04-01 11:25:11,297][DEBUG][threadpool ] [es_node_4_1] creating thread_pool [suggest], type [fixed], size [32], queue_size [1k] [2015-04-01 11:25:11,297][DEBUG][threadpool ] [es_node_4_1] creating thread_pool [percolate], type [fixed], size [32], queue_size [1k] [2015-04-01 11:25:11,298][DEBUG][threadpool ] [es_node_4_1] creating thread_pool [management], type [scaling], min [1], size [5], keep_alive [5m] [2015-04-01 11:25:11,300][DEBUG][threadpool ] [es_node_4_1] creating thread_pool [flush], type [scaling], min [1], size [5], keep_alive [5m] [2015-04-01 11:25:11,301][DEBUG][threadpool ] [es_node_4_1] creating thread_pool [merge], type [fixed], size [4], queue_size [32] [2015-04-01 11:25:11,302][DEBUG][threadpool ] [es_node_4_1] creating thread_pool [refresh], type [scaling], min [1], size [10], keep_alive [5m] [2015-04-01 11:25:11,303][DEBUG][threadpool ] [es_node_4_1] creating thread_pool [warmer], type [scaling], min [1], size [5], keep_alive [5m] [2015-04-01 11:25:11,304][DEBUG][threadpool ] [es_node_4_1] creating thread_pool [snapshot], type [scaling], min [1], size [5], keep_alive [5m] [2015-04-01 11:25:11,305][DEBUG][threadpool ] [es_node_4_1] creating thread_pool [optimize], type [fixed], size [1], queue_size [null] [2015-04-01 11:25:11,348][DEBUG][threadpool ] [es_node_4_2] creating thread_pool [generic], type [cached], keep_alive [30s] [2015-04-01 11:25:11,348][DEBUG][transport.netty ] [es_node_4_1] using worker_count[64],
Re: How to achieve ELK High availability
Which part do you want to make HA? ES is pretty simple to do, the rest is dependent on a few things. On 1 April 2015 at 14:06, vikas gopal vikas.ha...@gmail.com wrote: Hi Exerts, Need your valuable suggestions here . I have ELK on a single windows instance and I want to make it high available . I mean if one machine goes down second will take up the whole load, like clustering. Can you suggest how I can achieve this. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/38b05ef0-56e7-4590-b5e1-aaa64ab68150%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/38b05ef0-56e7-4590-b5e1-aaa64ab68150%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_0%2BjcG4pVfOa9cbxeb_-Z%2BBzoRK6miKjc4hyM_zj1eWA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Does english analyzer prevent fields from highlighting?
Hi all, could you help me with little problem regarding language-specific
analyzers and highliting in elasticsearch?
I need search documents by a query string and highlight matched strings.
Here is my mapping:
{
usr: {
properties: {
text0: {
type: string,
analyzer: english
},
text1: {
type: string
}
}
}
}
Note, that for text0 field english analyzer is set, and for text1
field is used standard analyzer by default.
In my index there is one document for now:
hits: [{
_index: tt,
_type: usr,
_id: AUxvIPAv84ayQMZV-3Ll,
_score: 1,
_source: {
text0: highlighted. need to be highlighted.,
text1: highlighted. need to be highlighted.
}
}]
Consider following query:
{
query: {
query_string : {
query : *highlighted*
}
},
highlight : {
fields : {
* : {}
}
}
}
I've expected each field in the document to be highlighted, but
highlighting appeared only in text1 field (where is no analyzer set):
hits: [{
_type: usr,
_source: {
text0: highlighted. need to be highlighted.,
text1: highlighted. need to be highlighted.
},
_score: 0.19178301,
_index: tt,
highlight: {
text1: [
emhighlighted/em. need to be emhighlighted/em.
]
},
_id: AUxvIPAv84ayQMZV-3Ll
}]
Let's consider the following query(I expected highlighted matches
highlight because of analyzer):
{
query: {
query_string : {
query : *highlight*
}
},
highlight : {
fields : {
* : {}
}
}
}
But there was no hist in response at all: (Did the english analyzer even
work here?)
hits: {
hits: [],
total: 0,
max_score: null
}
At last, consider some curl commands (requests and responses):
curl http://localhost:9200/tt/_analyze?field=text0; -d highlighted
{tokens:[{
token:*highlight*,
start_offset:0,
end_offset:11,
type:ALPHANUM,
position:1
}]}
curl http://localhost:9200/tt/_analyze?field=text1; -d highlighted
{tokens:[{
token:*highlighted*,
start_offset:0,
end_offset:11,
type:ALPHANUM,
position:1
}]}
We see, by passing text through the english and standard analyzers, the
result is different.
Finally, the question:
*does english analyzer prevent fields from highlighting? How can I get my
fields highlighted while full-text search?*
P.S. I use elasticsearch v1.4.4 on my local machine with windows 8.1.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/59cea72f-08e3-42b2-ad0e-e5d5ba7762a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Debug Rest Plugin
Hi I created a plugin, I'm able to create the zip, install it in my cluster, etc. My problem comes if I want to debug from Eclipse having access to the current data in my cluster. To do that the easiest way should be make the debug node part of my cluster. When I run the java application in Eclipse it launches a new elasticsearch node but it uses the generic cluster name elasticsearch, as my cluster have a different name the discovery functionality is not activated. Is there a way to specify elasticsearch config parameters (i.e. the parameter you usually add in elasticsearch.yml) when you debug the java app in your ide? Thanks!! -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0dd66199-336b-4d6d-ad25-68e67fa2f30c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: elasticsearch high cpu usage every hourly
From what I can see in your graphs I noticed two things. You seem to have a spike in search requests at that time, a spike in http traffic, and a cache eviction right at the beginning of it. Are you certain you don't have an external user with a cron job that runs at the top of the hour? Perhaps a large scan and scroll query that dumps alot of data? Take a look at your network graphs to see if you have a correlated spike in traffic to your ElasticSearch cluster. I wouldn't expect with a cluster that size that you don't have any users, probably quite a few users. It would not be unreasonable to expect that one such user is doing something beyond what you had intended and causing stress on your system. On Monday, March 30, 2015 at 8:37:54 PM UTC-6, vincent Park wrote: we have 8 clustered nodes and each nodes have 1 replica. total document size is about 4GB and 1,984,173 docs. I was suffering from very high CPU usage 80%~90% every hourly. It is held for 5 min. there is no other process except es on each server. there is no other cron job even at that time. I thought there are something wrong with es process. maybe external attacks or gc problem.. I don't know. It happened every hourly. I don't know what's going on elasticsearch at this time!! somebody help me, tell me what happened in there. please.. $ ./elasticsearch -v Version: 1.4.2, Build: 927caff/2014-12-16T14:11:12Z, JVM: 1.7.0_75 $ java -version java version 1.7.0_75 Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) and I installed plugins - HQ, bigdesk, head, kopf, sense heres bigdesk graphs at cpu peak time: http://elasticsearch-users.115913.n3.nabble.com/file/n4072788/es_cpu_high.png -- View this message in context: http://elasticsearch-users.115913.n3.nabble.com/elasticsearch-high-cpu-usage-every-hourly-tp4072788.html Sent from the ElasticSearch Users mailing list archive at Nabble.com. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/aca13c50-3c3a-4630-a941-4116dc31e55e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Debug Rest Plugin
When I want to debug a plugin, I often run the org.elasticsearch.bootstrap.Bootstrap class. And I pass something like -Des.foreground=true -Des.http.cors.enabled=true -Dhttp.cors.allow-origin=/.*/ -Dhttp.cors.allow-credentials=true -Des.path.conf=/path/to/your/config/dir/ /path/to/your/config/dir/ contains logging.yml and elasticsearch.yml HTH -- David Pilato - Developer | Evangelist elastic.co @dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs Le 31 mars 2015 à 18:10, María Carolina Bessega Lodi cbess...@gmail.com a écrit : Hi I created a plugin, I'm able to create the zip, install it in my cluster, etc. My problem comes if I want to debug from Eclipse having access to the current data in my cluster. To do that the easiest way should be make the debug node part of my cluster. When I run the java application in Eclipse it launches a new elasticsearch node but it uses the generic cluster name elasticsearch, as my cluster have a different name the discovery functionality is not activated. Is there a way to specify elasticsearch config parameters (i.e. the parameter you usually add in elasticsearch.yml) when you debug the java app in your ide? Thanks!! -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com mailto:elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0dd66199-336b-4d6d-ad25-68e67fa2f30c%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/0dd66199-336b-4d6d-ad25-68e67fa2f30c%40googlegroups.com?utm_medium=emailutm_source=footer. For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/98B7E3C1-DE1F-48AB-AA85-2E3CD3C95CA7%40pilato.fr. For more options, visit https://groups.google.com/d/optout.
Nested object under path [messages] is not of nested type
I am having a lot of issues with the nested type mapping in Elasticsearch,
I have ran this to create my index:
curl -XPOST 'http://localhost:9200/thread_and_messages' -d
'{mappings : {
message: {
properties: {
messages: {
type: nested,
include_in_parent: true,
properties: {
message_id: {type: string},
message_text: {type: string},
message_nick: {type: string}
}
}
}
}
}}'
Then this is how I've indexed a document:
curl -XPUT 'http://localhost:9200/thread_and_messages/thread/1' -d
'{
thread_id:2,
thread_name:Windows,
created:Wed Mar 25 2015,
first_nick:Admin,
messages:[
{message_id:5, message_text: Pc with a mouse,
message_nick:Admin},
{message_id:6, message_text:Keyboard, message_nick:Admin},
{message_id:7, message_text:iPhone, message_nick:Admin},
{message_id:8, message_text:Gym, message_nick:Admin}]
}'
This is my query:
curl -XGET 'http://localhost:9200/thread_and_messages/thread/_search' -d
'{query: {
bool: {
must: [
{match: {thread_name: windows}},
{nested: {
path: messages, query: {
bool: {
must: [{
match: {messages.message_text: gym}
}]
}
}
}}
]}
}
}'
I am receiving this error, even though I have clearly mapped messages as a
nested type:
QueryParsingException[[thread_and_messages] [nested]
nested object under path [messages] is not of nested type
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9c1395b9-99aa-47e8-900b-a819323275ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[ANN] Elasticsearch Mapper Attachment plugin 2.5.0 released
Heya, We are pleased to announce the release of the Elasticsearch Mapper Attachment plugin, version 2.5.0. The mapper attachments plugin adds the attachment type to Elasticsearch using Apache Tika.. https://github.com/elastic/elasticsearch-mapper-attachments/ Release Notes - elasticsearch-mapper-attachments - Version 2.5.0 Fix: * [121] - Don't wrap exceptions in `MapperParsingException` (https://github.com/elastic/elasticsearch-mapper-attachments/pull/121) * [118] - parse java.specification.version not java.version, so that it is robust (https://github.com/elastic/elasticsearch-mapper-attachments/pull/118) Update: * [119] - Update to elasticsearch 1.5.0 (https://github.com/elastic/elasticsearch-mapper-attachments/issues/119) * [90] - Tests: upgrade randomizedtesting-runner to 2.1.10 (https://github.com/elastic/elasticsearch-mapper-attachments/issues/90) Issues, Pull requests, Feature requests are warmly welcome on elasticsearch-mapper-attachments project repository: https://github.com/elastic/elasticsearch-mapper-attachments/ For questions or comments around this plugin, feel free to use elasticsearch mailing list: https://groups.google.com/forum/#!forum/elasticsearch Enjoy, -The Elasticsearch team -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/551ac92f.28e4b40a.33ed.6e2eSMTPIN_ADDED_MISSING%40gmr-mx.google.com. For more options, visit https://groups.google.com/d/optout.
Re: ES/Lucene eating up entire memory!
You need to read up a bit on how memory is allocated in Linux. In an ElasticSearch or Database server, this seems to be both, you want that free column to be 0. All available free memory should be used to cache files. In your snapshot you have 35GB of file cache listed under the cached heading. Memory listed under cached is essentially free memory that is temporarily being used to cache files until it is otherwise requested. This is how Linux makes efficient use of your memory, leveraging free memory for file cache, but still having it available when you need it. As such when determining if your box is out of memory you need to sum free and cached. This is precisely the reason that it is recommended that ElasticSearch only be allocated 50% of the memory on the box for heap. In your case where you have databases running, it should be 50% of the memory you have available for ElasticSearch. For that matter you should apply the same basic rule( 50%) to your database unless it has specifically some other file caching mechanism. For instance, you have 50GB of ram, assuming MySQL and ElasticSearch, and you want to equally divide the ram, 25GB to each. ElasticSearch then would be allowed to use 25GB, 12GB should be allocated to heap, the balance left to the OS for file caching on behalf of ElasticSearch. Assuming MySQL, with MyIsam, the same would be done 12GB to MySQL, 12GB to the OS for file system caching of the MyISAM tables. Now if you are using InnoDB things are different but that is way outside the scope of this discussion. So that you have 35GB of files being cached is a very good thing. It means that you have a large amount of your data cached. It means you have ample free memory, well beyond the 12GB a 50/50 split would demand. The 12GB of free you have now probably came from the processes that you killed, I think you meant this was ElasticSearch, though you were not specific. The one concern I see looking at your top, is that you have a large swap, and that some of it has been used. This is a sign that at some point you had memory pressure, the only sign I see from your snapshot. That pressure was not significant, but any swapping will destroy the performance of a database, or ElasticSearch. In many cases people go to the extreme of disabling swap entirely, as performance during swapping will be so poor, that it will be unusable. Further by the time you were to even put a dent in the size of that swap you will have wanted to reboot your box. My approach is to keep a small swap available, so that I can see if the system ever got to a point that it needed it, and to potentially buy a moment of time. If you are experiencing database slowdowns, this screenshot does not illustrate that it is due to memory issues. Rather I would suspect disk IO instead based on this information. On Tuesday, March 31, 2015 at 4:25:40 AM UTC-6, Yogesh wrote: Thanks Uwe. As I mentioned earlier, I did guess that VIRT doesn't indicate RAM consumption. What I am concerned about is the 3rd row which shows memory and indicates that out of the total 50g, 43g is in use. Once this number crosses 45g, my other databases start behaving badly. Problem is, even after I kill all the processes, this doesn't go down. (Attaching snapshot of top after killing all processes). Right now what I do is reboot the system every three days which is the time it takes to gradually fill the memory with something (I have no clue what that is). Though I think the max file descriptors wouldn't be the culprit for this? I haven't changed that yet. On Mon, Mar 30, 2015 at 3:19 AM, Uwe Schindler uwe.h.s...@gmail.com javascript: wrote: You should read: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html Maybe this allows you to figure out what's going on! VIRT means nothing about consumption, you should look at RES. Thanks, Uwe Am Sonntag, 29. März 2015 22:23:00 UTC+2 schrieb Yogesh: Hi, I have a single node ES setup (50GB memory, 500GB disk, 4 cores) and I run the Twitter river on it. I've set the ES_HEAP_SIZE to 5g. However, when I do top, the ES process shows the VIRT memory to be around 34g. That would be I assume the max mapped memory. The %MEM though always hovers around 10% However, within a few days post-reboot, the memory used keeps going up. From 10g to almost 50g (as shown in the third line) because of which my other dbs start behaving badly. Below is the snapshot of top. Despite the fact that VIRT and %MEM still hover around the same 34g and 10% respectively. Please help me understand where is my memory going over time! My one guess is that Lucene is eating it up. How do I remedy it? Thanks-in-advance! https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AN8/aq8-wxm2bBg/s1600/top.png -- You received this message because you are subscribed to a topic in the Google Groups
Re: ElasticSearch date format
http://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html
The T is a standard symbol, not exclusive to any one software project.
ElasticSearch uses Joda to parse dates, and it looks like the default
accepts ISO 8601 which I would expect.
On 31 March 2015 at 15:32, phani.nadimi...@goktree.com wrote:
HI James,
Thanks for quick reply..
so my understanding is while inserting in to elasticsearch we need to
use T as separator other wise it won't insert in to date field from java
API because it expecting standard format of ISO 8601 am I right?
Thanks
phani
On Tuesday, March 31, 2015 at 7:48:42 PM UTC+5:30, James Green wrote:
I am speculating here that you should use the ISO 8601 format for dates
since this is quite common and has a timezone which yours lacks
(introducing ambiguity).
The T is a common separator between the date and time parts.
On 31 March 2015 at 14:26, phani.n...@goktree.com wrote:
Hi All,
when i insert date to elastic search in the format of -MM-DD
hh:mm:ss from java api getting date format exception.but when i use
-MM-DDThh:mm:ss between them it is inserted fine.please explain me is T
here elasticsearch standard format?
please explain me functionality of T in date.i used the default
mapping of elasticsearch i didn't customized any thing.
mapping:
created: {
type: date,
format: dateOptionalTime
},
Thanks
phani
--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/94b29fca-637d-4d73-84fe-6af4b6300611%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/94b29fca-637d-4d73-84fe-6af4b6300611%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0a7fd7e8-b244-42e9-8427-752474250b07%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0a7fd7e8-b244-42e9-8427-752474250b07%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BawqvJ_DOZUR2u7C%3DU7AvFFr%3DhyBoBTMRwCbdow7kZxOsg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: Wildcard search on raw not_analyzed field
I need help determining how wildcard matching for non_analyzed fields works.
I have a field `message` with this mapping defined:
message : {
type : string,
norms : {
enabled : false
},
fields : {
raw : {
type : string,
index : not_analyzed,
doc_values : true,
ignore_above : 256
}
}
}
I've indexed a document that has this data: {message: Failed to connect}
I'm executing a Query String Query, and here are the results:
message.raw:Failed*
No matches
message.raw:failed*
No matches
message.raw:?ailed*
Match found!
Why don't the first two match?
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f55a7497-3bbe-4d32-bf27-89fc3a4533bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Is there any blog which shows new frameworks related to elasticsearch?
Hi all, Does elasticsearch itself have a blog which basically combines and shows extensions or tools made by developers from all over the world? Why I am asking is bacause we have also implemented an extension called Pes which helps in building inline query DSLs with less effort. Perhaps, other developers want to contribute to this project on GitHub in order to improve it. https://github.com/kodcu/pes -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5bc9-5ce5-4038-bf1d-ba58fd68a7c8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[ANN] JDBC plugin with feeder mode as an alternative to the deprecated Elasticsearch River API
Hi, if you use the JDBC river plugin and you are concerned about the deprecation of the river API, I wrote a step-by-step guide how to start the JDBC plugin in a feeder mode. The feeder mode is a standalone JVM which connects to an ES cluster using Java TransportClient under the hood. You can read the instructions here: https://github.com/jprante/elasticsearch-river-jdbc/wiki/JDBC-plugin-feeder-mode-as-an-alternative-to-the-deprecated-Elasticsearch-River-API Best regards, Jörg -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFj2tHMqkhO8SYcOJ6MAt8Lrp0w1KBpza4FT0uqooRAMw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: elastisticsearch_dsl python to create pivot tables
tested it. works as expected. Thanks again for your help Am Dienstag, 31. März 2015 00:04:39 UTC+2 schrieb Mike: Thanks Honza. You made my day (my night rather, it is midnight here in Brussels). I quickly tested the code and it gives the same results as the manually chained expression. I will test with various metrics tomorrow, I will then mark the question as “completed”. Thanks a lot and have a nice day. P.S:This opens the possibilty to perform any “pivot” table in ES . The challenge will be to parse the resulting json resuts (see http://stackoverflow.com/questions/29280480/), but I hope to find a way. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6e22c9c6-b8d8-4d55-b0d7-57d7b17f45be%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Wildcard search on raw not_analyzed field
Update, it seems that the problem is lowercase_expanded_terms defaults to
true.
Setting it to false in my query returned results for the first two queries.
On Monday, January 26, 2015 at 5:04:44 PM UTC-6, BradVido wrote:
I have a dynamic template that sets up not_analyzed raw mappings for all
my string fields.
When I perform a query search like this:
field.raw=KnownValue
it works (has hits).
When I do this:
field.raw=Known?alue
it doesn't return any data (? wildcard doesn't work).
However,
field.raw=*
does return data.
What am I missing? Does it have to do with case-sensitivity?
Here is the dynamic template:
{
string_fields: {
mapping: {
index: analyzed,
type: string,
fields: {
raw: {
index: not_analyzed,
ignore_above: 256,
doc_values: true,
type: string
}
}
},
match: *,
match_mapping_type: string
}
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8b081a6d-ad56-497f-851c-abd34b7db916%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Running Elasticsearch error
Hello, When I run ./bin/elasticsearch like it says in the http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash tutorial I get the following error? failed to connect to master [[Annie Ghazikhanian] [127.0.0.1:9300]], retrying... Caused by: java.net.ConnectException: Connection refused: /127.0.0.1:9300 Would you happen to know how I can fix this error? I believe I may be accidentally connecting to someone else's cluster. Thank you! -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52a946d5-24a5-4b7d-b8d7-f48678556dce%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: what are the research papers that ES relies on?
Thats what I was looking for (murmur3) I really wondered what they used and I was going to ask about murmur3 as weel. But as I see things, are going pretty awesome. Thanks 31 Mart 2015 Salı 00:42:45 UTC+3 tarihinde Aaron Mefford yazdı: I understand that if you do not have sufficient storage space, then you cannot manage a replica on every node. However, you are not limited to the size of a usual hdd. You can have a file system that spans many hdds. I am not suggesting this, but if you have a situation where you need to distribute all of your data, then you can. Also as we have little info on your use case, and the most typical seems to be log ingestion, in that scenario you can have that hot index, the most recent treated differently than the others. You could have the number of replicas on your most recent index spread data across the entire cluster, but then as a new index comes online reduce the number of replicas. You could also reindex historical data into fewer shards, improving performance, reducing addtl maintenance tasks. The reason I think you need to spend a bit more time reading is that the algorithm is very easy to find: http://www.elastic.co/guide/en/elasticsearch/guide/master/routing-value.html It is a very simple algorithm and standard approach to the issue of sharding: shard = hash(routing) % number_of_primary_shards The routing value by default is the document id, though you can specify your own routing value. The specifics of which hash are not as important except in very odd cases. A bit more research shows this from the source: https://github.com/elastic/elasticsearch/commit/9ea25df64927172787f2ffa1049f9c7804a91053#diff-d1fcc8637b3800bf7da881b93e1de983 Current implementations seem to use the DJB2 hash which is good but does have some cases such as 33 shards where it behaves poorly. In version 2.0 it appears they are moving to murmur3 which is a more consistent hash across a greater set of use cases. Note that with the default of 5 shards, DJB2 performs ideally. On Monday, March 30, 2015 at 10:04:08 AM UTC-6, MrBu wrote: Aaron, thanks for the reply. You cant distribute all of the documents if the size of it is more than a usual hdd. Also that was an example I gave. I am just figuring out the magical ways that ES uses rather than lucene has its own. 30 Mart 2015 Pazartesi 18:55:49 UTC+3 tarihinde Aaron Mefford yazdı: Automagic routing happens already on hashing the document id. It sounds like you may have a situation where your document id is creating a hot spot. This being the case what you want is not automagic routing but more control over the routing or a better document id. There is the ability to code your own routing and create a more even distribution, for your given keyset, but I think you would be better served by a better document key, this isnt mongo or hbase where the document key rules the world. The other possible reason you are hot-spotting is index creation. In a log ingestion scenario, the most recent index is almost always the hottest index. That is where all indexing is occurring, that is where all queries start. If you have tweaked the 5 shard norm and are only creating 1 shard that shard will be hot in this scenario. Your comment on routing a shard to another shard does not make any sense. You need to read a bit more on what the shards are and how they work. That said if you have multiple replicas of a shard, then those shards will automatically be distributed across all of your nodes. In fact if the number of replicas is the same as the number of nodes in the cluster, you should automatically have all data on all nodes, and any node will be able to query local data, and no node will be hot because of query volume. However indexing is still routed to the master shard. Like was mentioned previously, the code is open, however it sounds like you are looking to go deep water diving before learning to swim. On Monday, March 30, 2015 at 8:57:51 AM UTC-6, MrBu wrote: Jörg, Thanks for the input. I have read many tutorials, guides (official one too). Just I want to re-route in more automagic way. Like routing evenly to the shard and duplicating mostly used shard to other shards maybe. 30 Mart 2015 Pazartesi 10:33:19 UTC+3 tarihinde Jörg Prante yazdı: Elasticsearch is open source, so reading (and using and modifying) the algorithms is possible. There is also a lot of introductory material available online, and I recommend Elasticsearch - The definitive guide if you want paperwork. If you create an index, ES creates shards for this index (by default 5), and different nodes receive one of such shards, so indexing and search is automatically distributed over the participating nodes. ES keeps a map of shards in the cluster state, so every node is able to route a query or an index command. You don't need to
Storing fields in _source and nested values in a strict mapping
Hi,
I seem to be having an issue with getting the _source value to populate -
I'm not sure why as I thought this happened on default - but it keeps
coming back empty.
I also got the following error while trying to save the second value in a
nested array (end_call_info). Do I need an _index field and if so what
values does it need?
{error:RemoteTransportException[[es004][inet[/172.16.2.202:9300]][indices:data/write/index]];
nested: StrictDynamicMappingException[mapping set to strict, dynamic
introduction of [_index] within [cdr_data] is not allowed]; ,status:400}
I made different changes to the mapping from:
1) changing _source includes and enabled,
2) Changing store values for fields
The value is always the same:
Example :
{
- _index: cdr,
- _type: cdr_data,
- _id: vs-1426792279.84561,
- _version: 1,
- _score: 1,
- _source: { }
}
Mapping is a follows:
{
dynamic: strict,
_source:{
enabled:true,
includes: [meta.*, doc.*]
},
properties: {
uniqueid: {
type: string,
index: not_analyzed,
store: true
},
context: {
type: string,
index: not_analyzed,
store: true
},
start_call_info: {
type : nested,
properties: {
disposition: {
type: string,
index: not_analyzed
}
}
},
end_call_info: {
type : nested,
properties: {
uniqueid: {
type: string,
index: not_analyzed,
store: true
},
calldate: {
type: date,
format: date_hour_minute_second,
store: true
},
end_calldate: {
type: date,
format: date_hour_minute_second,
store: true
},
clid: {
type: string,
index: not_analyzed,
store: true
},
src: {
type: string,
index: not_analyzed,
store: true
},
dst: {
type: string,
index: not_analyzed,
store: true
},
dcontext: {
type: string,
index: not_analyzed,
store: true
},
channel: {
type: string,
index: not_analyzed,
store: true
},
dstchannel: {
type: string,
index: not_analyzed,
store: true
},
lastapp: {
type: string,
index: not_analyzed,
store: true
},
lastdata: {
type: string,
index: not_analyzed,
store: true
},
duration: {
type: long,
store: true
},
billsec: {
type: long,
store: true
},
call_direction: {
type: string,
index: not_analyzed,
store:no
},
disposition: {
type: string,
index: not_analyzed,
store: no
},
amaflags: {
type: long,
index: not_analyzed,
store: true
},
userfield: {
type: string,
index: not_analyzed,
store: true
},
Can I turn off kibana search highlight?
The search highlight is somehow very useful. But is there a setting or a configuration that allow me to turn the search highlight off? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a585ed41-01e7-45d9-953e-dc634b2cd0d0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: what are the research papers that ES relies on?
Murmur3 appears to be coming in 2.0. Currently it looks like it is using DJB2. On Tue, Mar 31, 2015 at 11:53 AM, MrBu metin.aky...@gmail.com wrote: Thats what I was looking for (murmur3) I really wondered what they used and I was going to ask about murmur3 as weel. But as I see things, are going pretty awesome. Thanks 31 Mart 2015 Salı 00:42:45 UTC+3 tarihinde Aaron Mefford yazdı: I understand that if you do not have sufficient storage space, then you cannot manage a replica on every node. However, you are not limited to the size of a usual hdd. You can have a file system that spans many hdds. I am not suggesting this, but if you have a situation where you need to distribute all of your data, then you can. Also as we have little info on your use case, and the most typical seems to be log ingestion, in that scenario you can have that hot index, the most recent treated differently than the others. You could have the number of replicas on your most recent index spread data across the entire cluster, but then as a new index comes online reduce the number of replicas. You could also reindex historical data into fewer shards, improving performance, reducing addtl maintenance tasks. The reason I think you need to spend a bit more time reading is that the algorithm is very easy to find: http://www.elastic.co/guide/en/elasticsearch/guide/master/ routing-value.html It is a very simple algorithm and standard approach to the issue of sharding: shard = hash(routing) % number_of_primary_shards The routing value by default is the document id, though you can specify your own routing value. The specifics of which hash are not as important except in very odd cases. A bit more research shows this from the source: https://github.com/elastic/elasticsearch/commit/ 9ea25df64927172787f2ffa1049f9c7804a91053#diff- d1fcc8637b3800bf7da881b93e1de983 Current implementations seem to use the DJB2 hash which is good but does have some cases such as 33 shards where it behaves poorly. In version 2.0 it appears they are moving to murmur3 which is a more consistent hash across a greater set of use cases. Note that with the default of 5 shards, DJB2 performs ideally. On Monday, March 30, 2015 at 10:04:08 AM UTC-6, MrBu wrote: Aaron, thanks for the reply. You cant distribute all of the documents if the size of it is more than a usual hdd. Also that was an example I gave. I am just figuring out the magical ways that ES uses rather than lucene has its own. 30 Mart 2015 Pazartesi 18:55:49 UTC+3 tarihinde Aaron Mefford yazdı: Automagic routing happens already on hashing the document id. It sounds like you may have a situation where your document id is creating a hot spot. This being the case what you want is not automagic routing but more control over the routing or a better document id. There is the ability to code your own routing and create a more even distribution, for your given keyset, but I think you would be better served by a better document key, this isnt mongo or hbase where the document key rules the world. The other possible reason you are hot-spotting is index creation. In a log ingestion scenario, the most recent index is almost always the hottest index. That is where all indexing is occurring, that is where all queries start. If you have tweaked the 5 shard norm and are only creating 1 shard that shard will be hot in this scenario. Your comment on routing a shard to another shard does not make any sense. You need to read a bit more on what the shards are and how they work. That said if you have multiple replicas of a shard, then those shards will automatically be distributed across all of your nodes. In fact if the number of replicas is the same as the number of nodes in the cluster, you should automatically have all data on all nodes, and any node will be able to query local data, and no node will be hot because of query volume. However indexing is still routed to the master shard. Like was mentioned previously, the code is open, however it sounds like you are looking to go deep water diving before learning to swim. On Monday, March 30, 2015 at 8:57:51 AM UTC-6, MrBu wrote: Jörg, Thanks for the input. I have read many tutorials, guides (official one too). Just I want to re-route in more automagic way. Like routing evenly to the shard and duplicating mostly used shard to other shards maybe. 30 Mart 2015 Pazartesi 10:33:19 UTC+3 tarihinde Jörg Prante yazdı: Elasticsearch is open source, so reading (and using and modifying) the algorithms is possible. There is also a lot of introductory material available online, and I recommend Elasticsearch - The definitive guide if you want paperwork. If you create an index, ES creates shards for this index (by default 5), and different nodes receive one of such shards, so indexing and search is automatically distributed over the participating nodes. ES keeps a map of
Re: Debug Rest Plugin
Thank you David! It worked :-) El martes, 31 de marzo de 2015, 12:13:23 (UTC-4), David Pilato escribió: When I want to debug a plugin, I often run the org.elasticsearch.bootstrap.Bootstrap class. And I pass something like -Des.foreground=true -Des.http.cors.enabled=true -Dhttp.cors.allow-origin=/.*/ -Dhttp.cors.allow-credentials=true -Des.path.conf=/path/to/your/config/dir/ /path/to/your/config/dir/ contains logging.yml and elasticsearch.yml HTH -- *David Pilato* - Developer | Evangelist *elastic.co http://elastic.co* @dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs Le 31 mars 2015 à 18:10, María Carolina Bessega Lodi cbes...@gmail.com javascript: a écrit : Hi I created a plugin, I'm able to create the zip, install it in my cluster, etc. My problem comes if I want to debug from Eclipse having access to the current data in my cluster. To do that the easiest way should be make the debug node part of my cluster. When I run the java application in Eclipse it launches a new elasticsearch node but it uses the generic cluster name elasticsearch, as my cluster have a different name the discovery functionality is not activated. Is there a way to specify elasticsearch config parameters (i.e. the parameter you usually add in elasticsearch.yml) when you debug the java app in your ide? Thanks!! -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0dd66199-336b-4d6d-ad25-68e67fa2f30c%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/0dd66199-336b-4d6d-ad25-68e67fa2f30c%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bb8687b3-4d83-47e4-93b3-505b0074eb98%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: unable to get script_fields to work
I'm having the same problem. Were you able to resolve this? Thanks! On Thursday, February 26, 2015 at 10:38:26 AM UTC-8, Gerald DeConto wrote: fyi: my config files (elasticsearch.yml and logging.yml) are in /etc/elasticsearch. the script_fields documentation indicated that I put the test.groovy in config/scripts folder, so I assumed my location is correct as I am unable to find a config folder in any elasticsearch folder -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1b71e8f1-7cb7-41d2-8587-b6a5f88e5ebb%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Is there any blog which shows new frameworks related to elasticsearch?
We don't have a blog but we do include things in our weekly summary, so I will pass this onto our community team. Thanks for sharing it though, and good luck! On 1 April 2015 at 06:21, Hakan Özler ozler.ha...@gmail.com wrote: Hi all, Does elasticsearch itself have a blog which basically combines and shows extensions or tools made by developers from all over the world? Why I am asking is bacause we have also implemented an extension called Pes which helps in building inline query DSLs with less effort. Perhaps, other developers want to contribute to this project on GitHub in order to improve it. https://github.com/kodcu/pes -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5bc9-5ce5-4038-bf1d-ba58fd68a7c8%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/5bc9-5ce5-4038-bf1d-ba58fd68a7c8%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X805HPT0cmEB_jK8hEPTm%2BAsDvOWW6TPXn%2BfQMKnJy%3DJA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Groovy ElsticSearch script
I'm having a similar problem. Were you able to get this to work? Thanks! On Friday, February 13, 2015 at 4:18:06 AM UTC-8, Sergio Soto Núñez wrote: Hi guys, I'm making a simple script to add information to documents. My first aproximation to solve my problem is based in a local script. I need to parse it and before add simple field to the documents. My problem is making import groovy.json.JsonSlurper def jsonFile = new File(./custom_data.json); def json = new JsonSlurper().parseText(jsonFile.text); def ident=id1; json[ident]; ElasticSearch show the next error when i save the script: java.lang.SecurityException: Importing [groovy.json.JsonSlurper] is not allowed at org.codehaus.groovy.control.customizers.SecureASTCustomizer.assertImportIsAllowed(SecureASTCustomizer.java:609) at org.codehaus.groovy.control.customizers.SecureASTCustomizer.call(SecureASTCustomizer.java:531) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1047) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:583) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:561) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:538) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:286) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:259) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:245) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:203) at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:119) at org.elasticsearch.script.ScriptService$ScriptChangesListener.onFileInit(ScriptService.java:548) at org.elasticsearch.script.ScriptService$ScriptChangesListener.onFileChanged(ScriptService.java:581) at org.elasticsearch.watcher.FileWatcher$FileObserver.onFileChanged(FileWatcher.java:261) at org.elasticsearch.watcher.FileWatcher$FileObserver.checkAndNotify(FileWatcher.java:112) at org.elasticsearch.watcher.FileWatcher$FileObserver.updateChildren(FileWatcher.java:197) at org.elasticsearch.watcher.FileWatcher$FileObserver.checkAndNotify(FileWatcher.java:98) at org.elasticsearch.watcher.FileWatcher.doCheckAndNotify(FileWatcher.java:52) at org.elasticsearch.watcher.AbstractResourceWatcher.checkAndNotify(AbstractResourceWatcher.java:43) at org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor.run(ResourceWatcherService.java:180) at org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:489) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 1 error ] at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124) at org.elasticsearch.script.ScriptService$ScriptChangesListener.onFileInit(ScriptService.java:548) at org.elasticsearch.script.ScriptService$ScriptChangesListener.onFileChanged(ScriptService.java:581) at org.elasticsearch.watcher.FileWatcher$FileObserver.onFileChanged(FileWatcher.java:261) at org.elasticsearch.watcher.FileWatcher$FileObserver.checkAndNotify(FileWatcher.java:112) at org.elasticsearch.watcher.FileWatcher$FileObserver.updateChildren(FileWatcher.java:197) at org.elasticsearch.watcher.FileWatcher$FileObserver.checkAndNotify(FileWatcher.java:98) at org.elasticsearch.watcher.FileWatcher.doCheckAndNotify(FileWatcher.java:52) at org.elasticsearch.watcher.AbstractResourceWatcher.checkAndNotify(AbstractResourceWatcher.java:43) at org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor.run(ResourceWatcherService.java:180) at org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:489) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) Here all the non empty lines in elasticsearch.yaml: cat /etc/elasticsearch/elasticsearch.yml |grep
Re: ElasticSearch date format
HI James,
Thanks for quick reply..
so my understanding is while inserting in to elasticsearch we need to
use T as separator other wise it won't insert in to date field from java
API because it expecting standard format of ISO 8601 am I right?
Thanks
phani
On Tuesday, March 31, 2015 at 7:48:42 PM UTC+5:30, James Green wrote:
I am speculating here that you should use the ISO 8601 format for dates
since this is quite common and has a timezone which yours lacks
(introducing ambiguity).
The T is a common separator between the date and time parts.
On 31 March 2015 at 14:26, phani.n...@goktree.com javascript: wrote:
Hi All,
when i insert date to elastic search in the format of -MM-DD
hh:mm:ss from java api getting date format exception.but when i use
-MM-DDThh:mm:ss between them it is inserted fine.please explain me is T
here elasticsearch standard format?
please explain me functionality of T in date.i used the default mapping
of elasticsearch i didn't customized any thing.
mapping:
created: {
type: date,
format: dateOptionalTime
},
Thanks
phani
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/94b29fca-637d-4d73-84fe-6af4b6300611%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/94b29fca-637d-4d73-84fe-6af4b6300611%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0a7fd7e8-b244-42e9-8427-752474250b07%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Wrong load distribution
Hi all, I meet a load distribution problem today and I browsed the Internet to find out someone having the same problem but unfortunately, no one seems to have. Here is my cluster configuration: - 4 machines with 16 Go Ram (9600 Mo allocated to JVM) - 8 core CPUs - Special cluster config: awareness of the vm host for shard primary and replica allocation (to avoid having both a primary and a replica on the same hardware). I tried a (huge) insert today, and it leads to one of the 4 nodes having a load average extremely higher than the three others (1.5 to 2 for the 3 relax servers vs. 12 for the high loaded one). I was thinking that Elasticsearch was designed to avoid it. Am I wrong here? Any tips is welcome :) Regards, Loïc -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d1aa9ed2-cdeb-4e07-aa7f-b760fb5f728e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Is there any blog which shows new frameworks related to elasticsearch?
Thank you :) On Tuesday, March 31, 2015 at 11:40:47 PM UTC+3, Mark Walkom wrote: We don't have a blog but we do include things in our weekly summary, so I will pass this onto our community team. Thanks for sharing it though, and good luck! On 1 April 2015 at 06:21, Hakan Özler ozler...@gmail.com javascript: wrote: Hi all, Does elasticsearch itself have a blog which basically combines and shows extensions or tools made by developers from all over the world? Why I am asking is bacause we have also implemented an extension called Pes which helps in building inline query DSLs with less effort. Perhaps, other developers want to contribute to this project on GitHub in order to improve it. https://github.com/kodcu/pes -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5bc9-5ce5-4038-bf1d-ba58fd68a7c8%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/5bc9-5ce5-4038-bf1d-ba58fd68a7c8%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d37d7a75-a1be-4708-b5ea-dab7f4936b66%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
is _id of document affects on scoring?
I add two same documents the only different thing is _id of documents (I
restart scenario for each of them and I do not add them sequentially. to be
sure my test is correct)
one of them changes order of result of this query and one of them does not:
GET index_for_test/business/_search
{
query: {
multi_match: {
query: italian,
type:most_fields,
fields: [ name^10, categories ]
}
}
}
i think it should be bug
my original question was:
https://github.com/elastic/elasticsearch/issues/10341
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5d0eba0f-0809-4a68-b247-0640addb9119%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: is _id of document affects on scoring?
The difference is probably due to the fact that your documents ended on
different shards where doc frequencies are different. You should be able to
make it a bit better by using distributed frequencies.
https://www.elastic.co/blog/understanding-query-then-fetch-vs-dfs-query-then-fetch
On Tue, Mar 31, 2015 at 3:18 PM, Mohammad.R esmailzadeh
esmailza...@gmail.com wrote:
I add two same documents the only different thing is _id of documents (I
restart scenario for each of them and I do not add them sequentially. to be
sure my test is correct)
one of them changes order of result of this query and one of them does
not:
GET index_for_test/business/_search
{
query: {
multi_match: {
query: italian,
type:most_fields,
fields: [ name^10, categories ]
}
}
}
i think it should be bug
my original question was:
https://github.com/elastic/elasticsearch/issues/10341
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5d0eba0f-0809-4a68-b247-0640addb9119%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/5d0eba0f-0809-4a68-b247-0640addb9119%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
Adrien
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAg-z%3DDqNptG14Ekp_czAYLxCZXYkgzK%3DokV3X9f6d%2BvSA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: How to merge elasticsearch 1.4.2 data with already running elasticsearch 1.4.4?
One option would be to take snapshots of your data in your 1.4.2 cluster and restore it in your 1.4.4 cluster. http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html On Tue, Mar 31, 2015 at 1:33 PM, Priya G g.shanmugapriy...@gmail.com wrote: I am having some data in elasticsearch 1.4.2.i am also having some data in elasticsearch 1.4.4. Now i want to take data stored in elasticsearch 1.4.2 and i have to store that data in elasticsearch1.4.4. Can anyone tell me how to do? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e7d87a86-03a0-4f34-a97c-8b28dc6e5ac3%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/e7d87a86-03a0-4f34-a97c-8b28dc6e5ac3%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Adrien -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAjqXAHyAkTmL6xs7i%2B75fCVNbyPY_8ZPBjxEeYGNJm2dw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Kibana showing wrong sales per month
Hi Schmorgs ,
Thanks for reply. using daily netsales also I tested, those also going
wrong and I checked my query it is fine and per year I am checking my
monthly netsales and one more point here the date field values we are
inserting as CST time.is it be a problem when we mapped to kibana.ES is
representing the values in UTC i heard.is there any change in kibana we
change time zone to CST?
date mapping is:
created: {
type: date,
format: dateOptionalTime
},
Thanks,
phani
On Friday, March 27, 2015 at 7:24:14 PM UTC+5:30, Schmorgs wrote:
Timestamping is a pretty critical feature for Kibana so I am 99.999%
(recurring) that this is not an Elasticsearch issue.
Have you checked the data that's being returned to confirm that the
timestamp field in the data matches the timerange in Kibana?
i.e. is your query correct
On Wednesday, 25 March 2015 14:14:01 UTC, phani.n...@goktree.com wrote:
Hi All,
i have the following scenario in kibana
I am using kibana version 3.1.2.
I am getting my monthly sales wrong in kibana dashboards. let say i
have all the data for Jan on elastic search now I created one dashboard
show net sales it is displaying some part of sales in december and some
part of sales in Jan due to that I am unable to track sale per month .
Data populating to elasticsearch is fine but coming to sales getting
numbers wrong but in past year sales displayed fine from kibana. please let
me know cause for this change?
Thanks Regards
phani
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8273ac72-22b5-437c-a5d7-c780f8b7263a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Elastic Search - How to delete old records and improve performance
Can you create a new index and exclude what you dont need. we did this recently because of some other mapping reason. take a snapshot (with newwer version) before doing so. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c473d836-50dd-4706-b7ab-da29cacd2612%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Question on upsert
NVM, I found the problem.
On Tue, Mar 31, 2015 at 4:10 PM, Bae, Jae Hyeon metac...@gmail.com wrote:
Hi
I am using upsert bulk api with ES 1.4.1. The problem is, I get the
following errors so frequently.
Error messages is
error - ElasticsearchIllegalArgumentException[failed to execute
script]; nested: GroovyScriptExecutionException[NullPointerException[Cannot
execute null+null]];
and the upsert payload is
{
params: {
count: 1
},
script: ctx._source.logcount += count,
upsert: {
ts: 1427842500662,
logcount: 1,
EVENT_TYPE: NF_ERRORS
}
}
I stripped out a few fields to reduce the space.
When I retrieve the document from ES, I found something weird, the
document is showing upsert payload itself.
{
_index: nf_errors_log20150331,
_type: default,
_id:
23797375spider-i-c5a68429com.netflix.cloudservice.resources.JarVersions.JarVersionsController115,
_version: 662,
found: true,
_source: {
params: {
count: 1
},
script: ctx._source.logcount += count,
upsert: {
ts: 1427842522664,
logcount: 1,
EVENT_TYPE: NF_ERRORS
}
}
}
Do you have any idea what I am doing wrong?
Thank you
Best, Jae
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKe7ALffcsLjQQa4bmazYRMDux2jH6CqrQ2shnO40RQysyF13Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: mlockall
Can you show us the relevant config sections from the nodes that don't have mlockall set? On 31 March 2015 at 17:40, Cheten Dev cheten@carwale.com wrote: Hi, here is the info Distributor ID: Ubuntu Description:Ubuntu 14.04.1 LTS Release:14.04 Codename: trusty I have tried to set mlockall = true on two server . i am able to successfully do it on one but not able to do it on other server On Tue, Mar 31, 2015 at 11:56 AM, Mark Walkom markwal...@gmail.com wrote: Which linux? On 31 March 2015 at 16:16, Cheten Dev cheten@carwale.com wrote: Hi, I am on linux server with elasticsearch 1.5.0 On Tue, Mar 31, 2015 at 8:07 AM, Mark Walkom markwal...@gmail.com wrote: A bit more info would be useful. What version of ES, what OS? On 30 March 2015 at 22:24, Chetan Dev cheten@carwale.com wrote: Hi, i have tried everything written on the document to set mlackall = true but nothing seems working here what am i missing here ? Thanks -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5748d9fb-120a-4521-8caa-62b780ea0594%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/5748d9fb-120a-4521-8caa-62b780ea0594%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/R41hW2QaL0w/unsubscribe . To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-92m9Qw4j4T8gwUV6%2BumkvbAOENTTsiupMurpGf8NZSw%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-92m9Qw4j4T8gwUV6%2BumkvbAOENTTsiupMurpGf8NZSw%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAG_NmL9T4dZeHUYpN0_5gwbfppL7%2BOJyNc6t3SQUpSe9N5oN2Q%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAG_NmL9T4dZeHUYpN0_5gwbfppL7%2BOJyNc6t3SQUpSe9N5oN2Q%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/R41hW2QaL0w/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_5X8ubgUVPd6ZV%2Bh_EP1m72Noht4DOvMCysUieWrQp9Q%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_5X8ubgUVPd6ZV%2Bh_EP1m72Noht4DOvMCysUieWrQp9Q%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAG_NmL-ADaAv%2BUx76DUVmVc7q0CJrT%2BV5VOJnUCYtfa-7CcW2w%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAG_NmL-ADaAv%2BUx76DUVmVc7q0CJrT%2BV5VOJnUCYtfa-7CcW2w%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Gfe-ZuNppxwF-ydLOQwNnDsOR0xf3A9BN5sGkkMXT_Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Health Status Red(Accidentally started another node)
Today, I accidentally started another node on a different machine on the same LAN which resulted in my first ES instance adding this new node to the cluster. I killed this second node with SIGKILL and restarted my original ES instance. However, on checking the health status of the various indices, a lot of them show red. The one's that show a red status have no. of replicas set to 0. The ones that are green had the number of replicas set to 1. Is there any way I can cleanly get the health back to Green without deleting and re-indexing. I am using the default setting of having 5 shards per index. Also, for each index that has its' status as red has exactly one shard that is unassigned. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1548aca9-50b4-42be-8b08-6be1a4d99bff%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Does english analyzer prevent fields from highlighting?
Using inline highlighters doesn't help highlighting. No. For the most
part you should stay away from inline analyzers and use a mapping instead.
On Tue, Mar 31, 2015 at 12:02 PM, Viacheslav Shalamov sslavian...@gmail.com
wrote:
Hi all, could you help me with little problem regarding language-specific
analyzers and highliting in elasticsearch?
I need search documents by a query string and highlight matched strings.
Here is my mapping:
{
usr: {
properties: {
text0: {
type: string,
analyzer: english
},
text1: {
type: string
}
}
}
}
Note, that for text0 field english analyzer is set, and for text1
field is used standard analyzer by default.
In my index there is one document for now:
hits: [{
_index: tt,
_type: usr,
_id: AUxvIPAv84ayQMZV-3Ll,
_score: 1,
_source: {
text0: highlighted. need to be highlighted.,
text1: highlighted. need to be highlighted.
}
}]
Consider following query:
{
query: {
query_string : {
query : *highlighted*
}
},
highlight : {
fields : {
* : {}
}
}
}
I've expected each field in the document to be highlighted, but
highlighting appeared only in text1 field (where is no analyzer set):
hits: [{
_type: usr,
_source: {
text0: highlighted. need to be highlighted.,
text1: highlighted. need to be highlighted.
},
_score: 0.19178301,
_index: tt,
highlight: {
text1: [
emhighlighted/em. need to be emhighlighted/em.
]
},
_id: AUxvIPAv84ayQMZV-3Ll
}]
Let's consider the following query(I expected highlighted matches
highlight because of analyzer):
{
query: {
query_string : {
query : *highlight*
}
},
highlight : {
fields : {
* : {}
}
}
}
But there was no hist in response at all: (Did the english analyzer even
work here?)
hits: {
hits: [],
total: 0,
max_score: null
}
At last, consider some curl commands (requests and responses):
curl http://localhost:9200/tt/_analyze?field=text0; -d highlighted
{tokens:[{
token:*highlight*,
start_offset:0,
end_offset:11,
type:ALPHANUM,
position:1
}]}
curl http://localhost:9200/tt/_analyze?field=text1; -d highlighted
{tokens:[{
token:*highlighted*,
start_offset:0,
end_offset:11,
type:ALPHANUM,
position:1
}]}
We see, by passing text through the english and standard analyzers, the
result is different.
Finally, the question:
*does english analyzer prevent fields from highlighting? How can I get my
fields highlighted while full-text search?*
P.S. I use elasticsearch v1.4.4 on my local machine with windows 8.1.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/59cea72f-08e3-42b2-ad0e-e5d5ba7762a8%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/59cea72f-08e3-42b2-ad0e-e5d5ba7762a8%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3m3mR2OMu6nbQG-9fVh04YOL_qTjqyn-%3DKqNPAdoeHuA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: Wrong load distribution
Do you have your shards equally distributed over the 4 nodes? Or do you use the default of 5 shards? Jörg On Tue, Mar 31, 2015 at 5:28 PM, Loïc Wenkin loic.wen...@gmail.com wrote: Hi all, I meet a load distribution problem today and I browsed the Internet to find out someone having the same problem but unfortunately, no one seems to have. Here is my cluster configuration: - 4 machines with 16 Go Ram (9600 Mo allocated to JVM) - 8 core CPUs - Special cluster config: awareness of the vm host for shard primary and replica allocation (to avoid having both a primary and a replica on the same hardware). I tried a (huge) insert today, and it leads to one of the 4 nodes having a load average extremely higher than the three others (1.5 to 2 for the 3 relax servers vs. 12 for the high loaded one). I was thinking that Elasticsearch was designed to avoid it. Am I wrong here? Any tips is welcome :) Regards, Loïc -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d1aa9ed2-cdeb-4e07-aa7f-b760fb5f728e%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/d1aa9ed2-cdeb-4e07-aa7f-b760fb5f728e%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH-iy-%3D5NPb0YoYuydwd-MXR%2BuLQ7p_4m9mNVrNLyEPyg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Question on upsert
Hi
I am using upsert bulk api with ES 1.4.1. The problem is, I get the
following errors so frequently.
Error messages is
error - ElasticsearchIllegalArgumentException[failed to execute
script]; nested: GroovyScriptExecutionException[NullPointerException[Cannot
execute null+null]];
and the upsert payload is
{
params: {
count: 1
},
script: ctx._source.logcount += count,
upsert: {
ts: 1427842500662,
logcount: 1,
EVENT_TYPE: NF_ERRORS
}
}
I stripped out a few fields to reduce the space.
When I retrieve the document from ES, I found something weird, the document
is showing upsert payload itself.
{
_index: nf_errors_log20150331,
_type: default,
_id:
23797375spider-i-c5a68429com.netflix.cloudservice.resources.JarVersions.JarVersionsController115,
_version: 662,
found: true,
_source: {
params: {
count: 1
},
script: ctx._source.logcount += count,
upsert: {
ts: 1427842522664,
logcount: 1,
EVENT_TYPE: NF_ERRORS
}
}
}
Do you have any idea what I am doing wrong?
Thank you
Best, Jae
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKe7ALeeagTNAzeVKaBzknvtHDEoBijSw71xdcsXvpuKCWiaVQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
