Re: [Openstack-operators] looking for feedback on proposals to improve logging
Excerpts from Nematollah Bidokhti's message of 2017-04-27 22:30:34 +: > Hi, > > I have been working on the concept of fault management blueprint to increase > cloud resiliency. As part of this proposal, info such as logs, KPIs, health > checks and so on are critical since we are engaging in deep data analysis and > machine learning. > > To ease the data analysis process there must be consistent logs. This comes > with having IDs and severity properties. > > Cloud resiliency in general requires fast fault detection, isolation and > recovery. In addition, there are applications such as NFV that is sensitive > to fast fault detection and recovery. One approach is to have meaningful logs > where by parsing the data we can make real time fault management decisions. > > Similar to interrupts, I would like us to have logging hierarchy which can > help an automated fault management system to take accurate and appropriate > actions. The format of the logs is important since it will ease the ML > analysis later in the process. I'm not sure what you mean by "logging hierarchy". Do you mean the severity levels that we have (like INFO, WARNING, and ERROR), or something else? Are you using the JSON formatter to make the logs easier to parse for the automated processing you're doing? If so, did you find that complex to configure? If not, were you aware that was possible and if you were, what caused you to decide not to use it? Does it not match your needs? Doug > > Thanks, > Nemat > > -Original Message- > From: Doug Hellmann [mailto:d...@doughellmann.com] > Sent: Wednesday, April 26, 2017 7:28 AM > To: openstack-operators > Subject: [Openstack-operators] looking for feedback on proposals to improve > logging > > I am looking for some feedback on two new proposals to add IDs to log > messages. Please see the thread on openstack-dev, and comment there or on the > specs referenced there. > > http://lists.openstack.org/pipermail/openstack-dev/2017-April/115958.html > > Thanks! > Doug > > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] looking for feedback on proposals to improve logging
There is an interesting proposal developing as part of https://review.openstack.org/460112 which it would be great to get more feedback on from other operators. My view is that OpenStack would benefit greatly from a troubleshooting guide. I remember my days with AIX (long ago) when a guide had the classic IBM information on “if you get this, this is why, you need to check this and here is how to fix it”. Developing a framework for every message to have an ID would be a significant change and would require major investment of effort. The proposal outlined in that we use the Python Exception framework to have the Exception Name as the basis for trouble shooting and googling for errors. While there is no guarantee of uniqueness, this gives a basis for developing the appropriate guides. Feel free to provide your input on the review, Tim On 28/04/17 00:30, "Nematollah Bidokhti" wrote: Hi, I have been working on the concept of fault management blueprint to increase cloud resiliency. As part of this proposal, info such as logs, KPIs, health checks and so on are critical since we are engaging in deep data analysis and machine learning. >> >> To ease the data analysis process there must be consistent logs. This comes with having IDs and severity properties. >> >> Cloud resiliency in general requires fast fault detection, isolation and recovery. In addition, there are applications such as NFV that is sensitive to fast fault detection and recovery. One approach is to have meaningful logs where by parsing the data we can make real time fault management decisions. >> >> Similar to interrupts, I would like us to have logging hierarchy which can help an automated fault management system to take accurate and appropriate actions. The format of the logs is important since it will ease the ML analysis later in the process. >> >> Thanks, >> Nemat -Original Message- From: Doug Hellmann [mailto:d...@doughellmann.com] Sent: Wednesday, April 26, 2017 7:28 AM To: openstack-operators Subject: [Openstack-operators] looking for feedback on proposals to improve logging I am looking for some feedback on two new proposals to add IDs to log messages. Please see the thread on openstack-dev, and comment there or on the specs referenced there. http://lists.openstack.org/pipermail/openstack-dev/2017-April/115958.html Thanks! Doug ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] looking for feedback on proposals to improve logging
Hi, I have been working on the concept of fault management blueprint to increase cloud resiliency. As part of this proposal, info such as logs, KPIs, health checks and so on are critical since we are engaging in deep data analysis and machine learning. >> >> To ease the data analysis process there must be consistent logs. This comes >> with having IDs and severity properties. >> >> Cloud resiliency in general requires fast fault detection, isolation and >> recovery. In addition, there are applications such as NFV that is sensitive >> to fast fault detection and recovery. One approach is to have meaningful >> logs where by parsing the data we can make real time fault management >> decisions. >> >> Similar to interrupts, I would like us to have logging hierarchy which can >> help an automated fault management system to take accurate and appropriate >> actions. The format of the logs is important since it will ease the ML >> analysis later in the process. >> >> Thanks, >> Nemat -Original Message- From: Doug Hellmann [mailto:d...@doughellmann.com] Sent: Wednesday, April 26, 2017 7:28 AM To: openstack-operators Subject: [Openstack-operators] looking for feedback on proposals to improve logging I am looking for some feedback on two new proposals to add IDs to log messages. Please see the thread on openstack-dev, and comment there or on the specs referenced there. http://lists.openstack.org/pipermail/openstack-dev/2017-April/115958.html Thanks! Doug ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] looking for feedback on proposals to improve logging
I am looking for some feedback on two new proposals to add IDs to log messages. Please see the thread on openstack-dev, and comment there or on the specs referenced there. http://lists.openstack.org/pipermail/openstack-dev/2017-April/115958.html Thanks! Doug ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
