Re: Soliciting input: ElasticSearch receiver
And that will be in a new module ;-) Gary On Sat, Jan 27, 2018 at 6:47 PM, Matt Sicker wrote: > On the sending side, we'd want to be able to configure index name, mapping > type name, override mapping settings, bulk request (i.e., batch) size, > refresh interval overrides, something similar to the column mappings thing > in the Cassandra and JDBC plugins, authentication, could be missing some > ideas (which I'm sure I'll remember next week when I'm deep in ES again). > > On the receiving side, ability to specify an arbitrary query would be > great. A minimal query feature could be to just specify the index name and > do a match_all query on it. Add in a polling interval and JSON parsing > (potentially with mappings from JSON output to whatever internal LogEvent > type class is in use) along with authentication and that'd cover a lot of > the basics. More advanced features are in Kibana < > https://www.elastic.co/products/kibana>, so we could always take some > ideas > from there as well. > > For me, if I'm ever using ES for log data, I use it mostly for interactive > queries, not for polling. For continual log ingestion, I'd go with Kafka or > Flume depending on the infrastructure in place. > > On 27 January 2018 at 18:58, Remko Popma wrote: > > > Sorry I won’t be able to help you with that; no experience with > > ElasticSearch. > > > > Remko > > > > (Shameless plug) Every java main() method deserves http://picocli.info > > > > > On Jan 28, 2018, at 9:48, Scott Deboy wrote: > > > > > > I'm looking at adding an ES receiver and was curious what folks would > > > like to see when it comes to configuration options/capabilities, other > > > than the ability to continually retrieve new events on some polling > > > interval, which I'll make sure to add. > > > > > > Scott > > > > > > -- > Matt Sicker >
Re: Soliciting input: ElasticSearch receiver
On the sending side, we'd want to be able to configure index name, mapping type name, override mapping settings, bulk request (i.e., batch) size, refresh interval overrides, something similar to the column mappings thing in the Cassandra and JDBC plugins, authentication, could be missing some ideas (which I'm sure I'll remember next week when I'm deep in ES again). On the receiving side, ability to specify an arbitrary query would be great. A minimal query feature could be to just specify the index name and do a match_all query on it. Add in a polling interval and JSON parsing (potentially with mappings from JSON output to whatever internal LogEvent type class is in use) along with authentication and that'd cover a lot of the basics. More advanced features are in Kibana < https://www.elastic.co/products/kibana>, so we could always take some ideas from there as well. For me, if I'm ever using ES for log data, I use it mostly for interactive queries, not for polling. For continual log ingestion, I'd go with Kafka or Flume depending on the infrastructure in place. On 27 January 2018 at 18:58, Remko Popma wrote: > Sorry I won’t be able to help you with that; no experience with > ElasticSearch. > > Remko > > (Shameless plug) Every java main() method deserves http://picocli.info > > > On Jan 28, 2018, at 9:48, Scott Deboy wrote: > > > > I'm looking at adding an ES receiver and was curious what folks would > > like to see when it comes to configuration options/capabilities, other > > than the ability to continually retrieve new events on some polling > > interval, which I'll make sure to add. > > > > Scott > -- Matt Sicker
Re: Soliciting input: ElasticSearch receiver
Sorry I won’t be able to help you with that; no experience with ElasticSearch. Remko (Shameless plug) Every java main() method deserves http://picocli.info > On Jan 28, 2018, at 9:48, Scott Deboy wrote: > > I'm looking at adding an ES receiver and was curious what folks would > like to see when it comes to configuration options/capabilities, other > than the ability to continually retrieve new events on some polling > interval, which I'll make sure to add. > > Scott
Soliciting input: ElasticSearch receiver
I'm looking at adding an ES receiver and was curious what folks would like to see when it comes to configuration options/capabilities, other than the ability to continually retrieve new events on some polling interval, which I'll make sure to add. Scott