On 04/07/2015 02:34 PM, Sandy Walsh wrote: > >> Tooling in general seems to be moving towards richer event data as well. >> The logging tools (Loggly/Logstash/PaperTrail/zillions of others) are >> intended to take your unstructured logs and turn them into events, so >> why not have Heat output structured events that we can present to the >> user with Ceilometer rather than sending log lines (through syslog or >> otherwise) and using tooling to reassemble them into events later. > > The trend in the monitoring space seems to be: > > 1. Alarms are issued from Metrics as Events. > (events can issue alarms too, but conventional alarming is metric based) > 2. Multiple events are analyzed to produce Metrics (stream processing) > 3. Go to Step 1 >
Indeed. I sort of envisioned heat sending out events that are then consumed both as metrics and by the user (where appropriate). In StackTach I can see that being implemented as /--> resource events ----> other tools Heat --> Winchester ---> notifications stream ------> user \--> metrics stream --> alerts --/ >> TL;DR: I think what we really want is a place to send and react to >> *events*. Logs are a way to do that, of course, but the Ceilometer way >> sounds pretty attractive. > > The difference is structured vs. unstructured data. Elasticsearch-based > solutions tend to ignore structure by looking at keywords. Some solutions, > like TopLog, infer a structure by gleaning regexs from logs. > > Events start as structured data. More so, we're looking at establishing > AVRO-based schema definitions on top of these events (slow progress). Yeah, I'd really like to have a schema for Heat events so we can have a single event stream and repackage events for different consumption goals (metrics, notifications, programmatic interaction, etc). > If anything we should consider changing the logging library to use structured > messages. Specifically: > > log("The %(foo)s did %(thing)s" % {'foo':'x', 'thing':'action'}) > it would be > log({'message':"The %(foo)s did %(thing)s", 'foo':'x', 'thing':'action'}) > > Which can still be formatted for conventional logs, but also sent as a > notification or as a higher-level structure to things like ES, TopLog, etc. > The driver can decide. > >> * CloudWatch has you send unstructured log messages, then build filters >> to parse them into quantifiable events, then set alarms on those metrics. > > Having to build filters is a relatively error-prone approach compared to the > methods described above. I wasn't saying *we* should do the unstructured message + regex filters strategy, I was just pointing out the CW solution for folks who hadn't used it. >>> [snip] > > The Fujitsu team have already added logging support to Monasca (with an > elasticsearch backend) and HP is currently adding StackTach.v3 support for > notification->event conversion as well as our Winchester event stream > processing engine. Also, this is based on Kafka vs. RabbitMQ, which has better > scaling characteristics for this kind of data. Oooh, I'll have a look into that, Kafka as an event bus sounds like a good fit. I have the same concern Angus voiced earlier about Zaqar though. What's the deployment of StackTach.v3 across OpenStack installations? Is it mostly deployed for Helion/Rackspace, or are smaller deployers using it as well? > >> This could be extended to richer JSON events that include the stack, >> resources affected in the update, stats like "num-deleted-resources" or >> "num-replaced-resources", autoscaling actions, and info about stack errors. > > Some of these sound more like a metrics than notifications. We should be > careful not to misuse the two. I think they're events, and have facets that are quantifiable as metrics (num-replaced-resources on an update action) and that should be user-visible (update is complete, or autoscaling actions taken). >> Is there a way for users as-is to view those raw notifications, not just >> the indexed k/v pairs? > > In StackTach.v3 we ship the raw notifications to HDFS for archiving, but > expose the reduced event via the API. The message-id links the two. > > Lots more here: http://www.stacktach.com Thanks! I'll have to read up. -- Ryan Brown / Software Engineer, Openstack / Red Hat, Inc. __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev