Logstash agent and rsync

I am trying to use logstash to index a set of log files from our webservers. The log files that I can access are stored on a network attached storage system which are copied over from the webservers using rsync. Logstash agent ends up throwing this error: {:timestamp=>"2015-02-18T13:10:45.190

Re: Elasticsearch as Logwatch

I believe you can tell logwatch to output its reports as a file, which could then be ingested with logstash. Alternatively, logstash has an imap input that you could use to get emails into Elasticsearch. -- The information transmitted in this email is intended only for the person(s) or entity to

Re: Force search on a local node?

Itamar - perfect, thanks very much! On Sat, Feb 7, 2015 at 3:03 PM, Itamar Syn-Hershko wrote: > > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-preference.html > > -- > > Itamar Syn-Hershko > http://code972.com | @synhershko

Upgrade from 1.2.1 to 1.4.2 and indices/shards corrupted

Hello, We upgraded from 1.2.1 to 1.4.2 and after the restart, some shards were in UNASSIGNED state because they was corrupted. Here is a log: https://gist.github.com/anonymous/63a0e3f70a72bab37052 and "Failed to start shard, message [IndexShardGatewayRecoveryException[[events_x][2] failed t

Kibana 4 Template/RISON URL's

t a stable endpoint to the updated URL, injecting the parameters into the RISON URL Option one is a bit gross, and option two has no integration into Kibana 4 itself currently :[ What is everyone else thinking? -Greg -- You received this message because you are subscribed to the Goo

Re: Elasticsearch as Logwatch

Just to throw it out there, is there a reason you wouldn't take the daily results from logwatch, and pump those into elasticsearch? If dealing with the hundreds of emails is the issue, then that could let you make a query to show (for example) which users had the most login failures across all

Re: powerful cluster is not able to handle 1.5Tb of data, how to optimize?

I run 1.3TB of active indices on a single node (64 GB ram with 12GB heap size, and 15 small disks in a raid 5), with most of my messages quite small, which makes it looks similar to your case in volume, although I have a significantly lower (about 5K) indexing rate. I suspect that the single di

Re: I search same thing, but once can get and once can not get???

This is a symptom that could happen with bad GC events, or with split brain. Can you look at the GC logging output to see how long the stop the world pauses you're seeing are? You can also run a query like " curl -XGET 'http://localhost:9200/_cluster/state/master_node?local=true' " on each of t

Re: Constant High (~99%) CPU on 1 of 5 Nodes in Cluster

>From the Marvel image, it looks like the heap utilization isn't dropping periodically as it does on the other nodes. Can you verify that GC is behaving nicely while this occurs? -- The information transmitted in this email is intended only for the person(s) or entity to which it is addressed

JDK version 1.7.0_65?

? The push for upgrading is coming from this security update: https://lists.debian.org/debian-security-announce/2014/msg00169.html Thanks -Greg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop

Aggregation using the results of an aggregation?

"sales": { "sum": { "field": "Sale" } }, "onhand": { "avg": { "field": "Onhand" } }, "*stock

Re: enabling scripting for installed scripts only

ecute locally stored scripts. See the example in > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#modules-scripting > > > --Alex > > > On Fri, Jun 27, 2014 at 8:44 PM, greg j > > wrote: > >> Hi, >> >>

enabling scripting for installed scripts only

Hi, We'd like to enable custom scoring using a script that we'll install under config/scripts, so that we can invoke it as part of a function_score query, like "query": { "function_score": { "query" : { ... }, "functions": [ { "script_score": { "script": "my-script" //

Re: Cannot Increase Write TPS in Elasticsearch by adding more nodes

I haven't seen it asked yet; what is feeding data into your elasticsearch? Depending on what you're doing to get it there, a large document size could easily bottleneck some feeding mechanisms. It's also noteable that some "green" spinning disks top out in the realm of 72MB/s. It might be useful

Re: Increasing thread pool / queue size.

You might also keep an eye on what your disk utilization is like when the search queue is filling up; CPU isn't the only possible bottleneck here. -- The information transmitted in this email is intended only for the person(s) or entity to which it is addressed and may contain confidential and

Re: ElasticSearch not seen in JPS

Fixed, permission issue in /tmp Regards, Greg On Thursday, May 15, 2014 5:41:41 PM UTC+2, Greg Bui wrote: > > Hello, > > I encounter a weird problem, jps is not returning elasticsearch whereas it > is running and (seems) working fine. I remember that was the case before. > js

ElasticSearch not seen in JPS

"build_snapshot" : false, "lucene_version" : "4.7" }, "tagline" : "You Know, for Search" } Thanks a lot in advance, Greg -- You received this message because you are subscribed to the Google Groups "elasticsearch" gr

Re: Splunk vs. Elastic search performance?

I'm running elasticsearch much smaller than this, but with a PowerEdge R900 with 2 X7350 CPUs, and 64 GB of RAM (24GB heap for elasticsearch) I'm able to sustain something like 80GB per day (1/16 your volume). Some of the latest Intel CPUs are about 4 times as powerful as the X7350, so extrapol

Need some input/advice on scripting approach

pt in Java for performance reasons? I have no experience with MVEL, and I know Java already - is Java faster than MVEL for number crunching? Thanks in advance for any advice you might have! Greg -- You received this message because you are subscribed to the Google Groups "elasticsearch&

Delete documents after split brain

an delete the documents? Thanks Greg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion

Re: Is it possible to combine filters and nested filters?

Hey Binh, I tried your example and it does in fact work. I realized my query was just wrong...really stupid mistake. Thanks for your help! On Wednesday, March 26, 2014 5:22:44 PM UTC-4, Binh Ly wrote: > > Hmmm, not sure. I just tried this and it works for me (ES 1.1): > > 1. PUT http://localho

Re: Is it possible to combine filters and nested filters?

Hey Binh, The field is indexed the same way I am searching on it, and should be an exact match. The term filter matches when I remove the company_id filter, and the company_id filter works when I remove the categories.name nested filter. However, when combined, it does not return results. On

Is it possible to combine filters and nested filters?

I have a document that contains nested documents. I know it is possible to combine multiple filters on fields in the root document, but I haven't figured out how to combine those filters with nested filters. Take this query for example: { "from": 0, "size": 20, "query": { "filtered"