Re: ElasticSearch Website down for quite some days?
Try http://elastic.co Elasticsearch (the company) changed their name to Elastic this week. The site redirects for me, but maybe there's something amiss in your browser or proxy. On Friday, March 13, 2015 at 4:16:36 PM UTC-6, liu wei wrote: > > Starting from beginning of this week, i have trouble access that. Anyone > have similar issues? > http://www.elasticsearch.org/ > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3b94e2b6-aff8-4d35-bc30-95f8c6077579%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: scripted fields and type conversion with Kibana4B3
Well, I figured out that I can use doc['field_name'].value.toString(), but that returns the integer version of the IP address as a string rather than the dotted decimal version. I see some discussion of this problem in the ES Github issues, but I can't tell if there's a solution. On Wednesday, December 17, 2014 7:56:31 PM UTC-7, Jay Swan wrote: > > I just started playing with scripted fields in Kibana4 Beta3. > > I'm looking for guidance on how to do type conversions with the new > scripted fields feature. Specifically, I have some ES fields that I've > typed as "ip" to (obviously) store IPv4 addresses. I'd line to convert > these to strings when creating my scripted field. Since 1) I didn't know > anything about Groovy until yesterday, and 2) Groovy doesn't appear to have > a native IP type, I'm at a loss as to how to approach this. Anybody know? > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3147a62a-9efb-4a17-af97-4b6c02acc082%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
scripted fields and type conversion with Kibana4B3
I just started playing with scripted fields in Kibana4 Beta3. I'm looking for guidance on how to do type conversions with the new scripted fields feature. Specifically, I have some ES fields that I've typed as "ip" to (obviously) store IPv4 addresses. I'd line to convert these to strings when creating my scripted field. Since 1) I didn't know anything about Groovy until yesterday, and 2) Groovy doesn't appear to have a native IP type, I'm at a loss as to how to approach this. Anybody know? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6f6ff411-95f9-4003-bc9a-d355f3a3165c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Elasticsearch percolator and Logstash ... How to ?
Logstash doesn't support the percolator API today.
On Friday, December 5, 2014 8:02:39 AM UTC-7, Thorsten Nickel wrote:
>
> Dear all,
>
> this is my first post to this group, since any more googling does not
> result in any more results, this looks like my last hope.
>
> Regarding my problem, I have an ELK Stack up and running. My logstasher
> sits on system logs, grok's them as I want them to, and puts these
> informations
> via the elasticsearch output plugin into ES. Using Kibana Frontend, I can
> see my Loglines flowing in, and everything is good.
>
> Now for my scenario, I would like to have some form of basic 'alerting'
> put into this chain. For this, I put up a percolator query into ES.
> If I use CURL to ask with this percolator, I get my result like this :
>
> curl -XGET '
> 127.0.0.1:9200/logstash-2014.12.05/MONITOR/AUoaylhicoeQR6lWzrbU/_percolate?pretty
> '
> {
> "took" : 38,
> "_shards" : {
> "total" : 5,
> "successful" : 5,
> "failed" : 0
> },
> "total" : 1,
> "matches" : [ {
> "_index" : "logstash-2014.12.05",
> "_id" : "monitor_red"
> } ]
> }
>
> The point is, I kinda need to do this percolator query, when I put my Doc
> into ES using logstash so I can see, if it triggers the percolator query or
> not. Or another point would be, if I have like
> thousands of docs in ES, how would I percolate through all of them to see,
> which ones match the monitor_red query?
>
> As far as I can see, I need to query with the percolator one doc at a
> time looks really cumbersome to me.
>
> I hope that I explained my problem good enough.
>
> Kind regards,
> T. Nickel
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5de53375-4078-4f98-bbfc-4b3fe2c0f485%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Differences between _source in Kibana and in elasticsearch
I would guess that you need to refresh your field list in the Settings > Indices > Index pattern section of Kibana4; this is a new thing in Kibana4 that's very different from v3. Drove me crazy trying to figure it out until I filed an issue. See Rashid's answer to my Github issue here: https://github.com/elasticsearch/kibana/issues/1995 It would be nice to see this happen at least semi-automagically in the future; it's going to bite a lot of people during migration. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fa95a091-752b-40de-b5f4-99dc3cc0c5aa%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: ELK stack and monitoring/alerting
As Mark says, there's nothing built in. Approaches that I've read about for bolting alerting onto ELK: 1) Use Logstash to output to some other alerting pipeline, such as: email, Nagios, Riemann. 2) Write a cron job / scheduled task to run Elasticsearch queries periodically and take action based on the results. This is not part of Logstash or Kibana; you need to write it yourself. I believe the MozDef project has some code to do this: https://github.com/jeffbryner/MozDef. 3) Write your own indexer that makes use of Elasticsearch percolators. Percolators allow you to match indexed queries against new indexed documents, which is kind of like alerting. Again, you'd need to write it yourself. On Friday, November 7, 2014 9:48:41 AM UTC-7, Wish wrote: > > I am new to ELK stack. I guess, I understand ELK can be used for log > management. You can view the details on dash board using kibana etc. > > however, one question, can we have an alerting system as an extension of > ELK stack ? > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9a5f17eb-0e2b-40ef-b668-45c5598accf6%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Kibana server-size integration with R, Perl, and other tools
I think you could automate some of this with the Elasticsearch scripting module: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html Unfortunately none of that is accessible through Kibana yet, but we can hold out hope for Kibana 4.0. On Thursday, September 25, 2014 9:57:44 AM UTC-6, Brian wrote: > > Ash, > > JSON is a natural for Kibana's Javascript to read and therefore emit as > CSV. So what I really was asking is Kibana going to become a serious > conteder and allow user-written commands to be inserted into the pipeline > between data query/response and charting. After my few weeks with R, I have > gotten it to far exceed GNUPlot for plotting (even with the base plotting > functions; I haven't yet dived into ggplot2 package), and to also far > exceeds Kibana. For example, setting up a custom dashboard is tedious, and > it's not easily customizable. > > Now, I am not suggesting that the ELK stack turn into Splunk directly. But > since it wants to become a serious contender, I an am strongly recommending > that the ELK team take the next step and allow a user-written command to be > run against the Kibana output and its charting. And I recommend that the > output be CSV because that's what R supports so naturally. And with R, I > can build out custom analysis scripts that are flexible (and not hard-coded > like Kibana dashboards). > > For example, I have an R script that gives me the most-commonly used > functions that the Splunk timechart command offers. And with all of its > customizability: Selecting the fields to use as the analysis, the by field > (for example, plotting response time by host name), the statistics (mean, > max, 95th percentile, and so on), even splitting the colors so that the > plot instantly shows the distribution of load across 10 hosts that reside > within two data centers. > > This is an excellent (and free) book that shows what Splunk can do by way > of clear examples: > > http://www.splunk.com/goto/book > > Again, I don't suggest that Kibana duplicate this. But I strongly suggest > that Kibana gives me a way to insert my own commands into the processing so > that I can implement the specific functions that our group requires, and > can do it without my gorpy Perl script and copy-paste command mumbo-jumbo, > and instead in a much more friendly and accessible way that even the PMs > can run from their Windows laptops without touching the command line. > > And as my part of the bargain, I will use Perl, R, or whatever else is at > my disposal to create custom commands that can run on the Kibana host and > perform all of the analysis that our group needs. > > Brian > > On Wednesday, September 24, 2014 4:34:43 PM UTC-4, Ashit Kumar wrote: >> >> Brian, >> >> I like the direction you are going down and am trying to do that myself. >> However, being a perl fledgling, I am still battling Dumper etc. I would >> appreciate it if you could share your code to convert and ES query to CSV. >> I want to use aggregations and print/report/graph results. Kibana is very >> pretty and does the basics well, but I want to know who used web mail and >> order it by volume of data sent by hour of day and either graph / tabulate >> / csv out the result. I just cant see how to do that with Kibana. >> >> Thanks >> >> Ash >> >> -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c1c9cebf-2c4e-4783-a8c0-734c0e064c32%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Elastic + Kibana Server Specs Recommendation
For Elasticsearch, try m3.xlarge and set ES_HEAP_SIZE to 7 or 8GB. You may
also want to have more than one node in your cluster.
You might also want to split Logstash off onto a separate instance. It is
CPU intensive but not particularly RAM intensive. Set the -w {n} flag in
the startup script to allow Logstash to run multiple threads across
multiple cores. You might start with a m3.large for this and use -w 2 and
see how it goes.
On Wednesday, August 13, 2014 9:38:10 AM UTC-6, AK wrote:
>
> Hi,
>
> I recently launched ELK and I'm receiving about 3,000,000 - 8,000,000 docs
> per day (~ 5GB)
> I'm running on AWS on a small server, and after a week of data collection
> the system becomes very very slow, mainly when I am looking for data older
> than 2 days.
> Do you have a recommendation for servers in points such as cpu, memory and
> iops and elstic settings like shards.
>
> Thanks
> AK
>
>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/90398be0-4804-44d7-9f8e-e033daa7050b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: New version of Kibana in the works?
When I took the Core Elasticsearch class last week, someone from Elasticsearch said that Kibana 4.0 is in the works but is at least a couple of months away from release. There wasn't any real detail on features, other than to say that it is a complete rewrite that tries to expose as much native Elasticsearch query functionality as possible. I was also told that most security-related features will be released in a commercial "security module" for Elasticsearch that will have a non-trivial cost associated with it. I don't know if this includes role-based access control or not. On Tuesday, August 12, 2014 10:44:55 AM UTC-6, Antonio Augusto Santos wrote: > > This one is for the devs, and Rashid in special: there is any new version > of Kibana in the works? > I'm asking this because I'm about to start a project in my company for log > management, and there are some requisites to it (user separation, event > correlation, histogram to compare two values, and so on). > > So, any changes of these functionalities landing on Kibana 4.0? ;) > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ca643d8f-2b03-4e4d-930c-5d9a0a6a70eb%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Simple setup question
I don't know about this particular problem, but you could try the UDP input
instead:
input {
udp {
port => your_portnumber_here (514 is the syslog default)
type => your_typename_here
}
}
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f29db21c-3c49-4f1f-b8de-f78ff79b9507%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
