Hi Frank, 'thinking out loud' - ideally I'd like to see this failure (and it reasons) surfaced to the admins outside of the logs. Off the top of my head - a (helpful) detailed event/alert notification.
More helpful info in the message (ie not 'host 24') *should* be pretty simple. I'll have a look and ask one of our developers to help me with the actual code, once I've tracked it down. Kind regards, Paul Angus Regards, Paul Angus paul.an...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -----Original Message----- From: Frank Louwers [mailto:fr...@openminds.be] Sent: 21 April 2016 08:14 To: users@cloudstack.apache.org Subject: Re: CloudStack Logging Hi Paul, I’d love to see improvement in that area! Especially for my operational admins, the biggest issue we face is this: - Someone wants to deploy a new VM, and it fails. - Currently, it’s very hard to figure out exactly why. The best they get is “Capacity Planner failure”. - They come to me :) I think these failures are quite an easy use-case to improve logging on. If a VM can’t be started because of a “capacity” issue, it would we good to log (in a non-debug, non-info log): - Which hosts were considered for CPU/Mem (based on zone, tags etc) (list them by name pref, not by “host 24” which means nothing to them as that number isn’t anywhere in the CS UI) - Which of those considered hosts have the capacity needed (and for those that are excluded, weather they are excluded for RAM or CPU reasons) - Exact same for Storage Pools This would solve over 95% of my “Frank, I can’t find out why this VM won’t deploy” logging issues. Regards, Frank > On 20 Apr 2016, at 22:05, Paul Angus <paul.an...@shapeblue.com> wrote: > > Hi, > > I don't think that it's a question of importance. INFO vs DEBUG should be > telling you different things. ERROR and WARN are also quite different. > In general I'm targeting the cloud operational admins, the people who need to > know the health of their cloud and deal with issues as they're constantly > reading the log. > > However, I'm not proposing to add or remove any messages, just revisit the > categorisation. If the 'operator' has their logging on debug they'd actually > see the same messages. The idea is to make turning the logging down to INFO > feasible. > > You'd turn the logging back up to pick up code issues rather than just > operational issues. > > Some VERY simplified definitions might be: > > DEBUG: inner workings of CloudStack that only a developer can 'understand' > INFO: actions that CloudStack is performing or information analysis (host 5 > is full so not going to use it). > WARN: this isn't good > ERROR: something just didn't work. > > > Grabbing a couple of examples > > these should be debug - I can't 'do' anything with this information. > > INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-4119d5bc) Begin cleanup expired > async-jobs INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-4119d5bc) End cleanup expired async-jobs > INFO [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-2:ctx-3209f65c) > Scan hung worker VM to recycle INFO [c.c.h.v.r.VmwareResource] > (DirectAgentCronJob-113:ctx-56d9f9f2) Scan hung worker VM to recycle > > Whereas this should be WARN. I wouldn't want to lose this by switching > off DEBUG > > DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-91226be7) > Detected management node left, id:2, nodeIP:10.2.0.6 DEBUG > [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-0bccd078) Detected > management node left, id:2, nodeIP:10.2.0.6 > > > > > Kind regards, > > Paul Angus > > Regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > -----Original Message----- > From: Chaz PC [mailto:dreeems4e...@hotmail.com] > Sent: 20 April 2016 11:32 > To: users@cloudstack.apache.org > Subject: RE: CloudStack Logging > > Hello, > I have a question what is the criteria that you are going to follow to decide > the log record importance. Who is the user that you will design the logging > process for. Is it for finding issues with the installation or is it for > security breaches or auditing. > These questions are good to put in mind when designing the logging process. > > > Sent from my Samsung Galaxy smartphone.<div> </div><div> > </div><!-- originalMessage --><div>-------- Original message > --------</div><div>From: Paul Angus <paul.an...@shapeblue.com> > </div><div>Date: 20/04/2016 1:01 PM (GMT+04:00) </div><div>To: > d...@cloudstack.apache.org, users@cloudstack.apache.org > </div><div>Subject: RE: CloudStack Logging </div><div> </div> Hi > Simon, > > My gut says that it's probably not worth going back before 4.3. There have > been large changes since then but I think that it's better to get as much > data as possible, limiting to 4.6+ might not give a particularly large > install base. > > > Kind regards, > > Paul Angus > > Regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > -----Original Message----- > From: Simon Weller [mailto:swel...@ena.com] > Sent: 19 April 2016 21:51 > To: users@cloudstack.apache.org; d...@cloudstack.apache.org > Subject: Re: CloudStack Logging > > Paul, > > Are you wanting for focus on logs on releases later than a particular version > (e.g. 4.6)? > > - Si > ________________________________________ > From: Paul Angus <paul.an...@shapeblue.com> > Sent: Tuesday, April 19, 2016 10:55 AM > To: users@cloudstack.apache.org; d...@cloudstack.apache.org > Subject: RE: CloudStack Logging > > No problem. > > Good question - Any and all logs. Often, it's the mundane stuff which > obscures the messages that you really need to see! > > > Kind regards, > > Paul Angus > > Regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > -----Original Message----- > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On > Behalf Of Will Stevens > Sent: 19 April 2016 16:53 > To: d...@cloudstack.apache.org > Cc: users@cloudstack.apache.org > Subject: Re: CloudStack Logging > > Thanks Paul. Are you looking for only logs with errors or are you just > looking for just any management server logs? > > *Will STEVENS* > Lead Developer > > *CloudOps* *| *Cloud Solutions Experts > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw > @CloudOps_ > > On Tue, Apr 19, 2016 at 11:45 AM, Paul Angus > <paul.an...@shapeblue.com> > wrote: > >> Thanks Will, >> >> As far as I know anything sensitive is removed or obfuscated (vcenter >> password displayed as h**********). IP addresses and hostnames are shown. >> Users may want to use a tool like SED to replace IP addresses or >> internal domain names >> >> http://forums.qrz.com/index.php?threads/remove-ip-addresses-from-log- >> f >> iles.204196/ >> >> >> Kind regards, >> >> Paul Angus >> >> Regards, >> >> Paul Angus >> >> paul.an...@shapeblue.com >> www.shapeblue.com >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue >> >> -----Original Message----- >> From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On >> Behalf Of Will Stevens >> Sent: 19 April 2016 16:40 >> To: d...@cloudstack.apache.org >> Cc: users@cloudstack.apache.org >> Subject: Re: CloudStack Logging >> >> I like this initiative Paul. I just wanted to check with you. Do >> you know if there is any confidential information like credentials or >> the like that is logged in the management server log? I thought we >> had done a push at one point to make sure no sensitive data was >> logged, but I don't remember for sure. I think knowing the answer to >> this will help people feel more comfortable uploading logs. >> >> Thanks for the valuable initiative... >> >> *Will STEVENS* >> Lead Developer >> >> *CloudOps* *| *Cloud Solutions Experts >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw >> @CloudOps_ >> >> On Tue, Apr 19, 2016 at 11:34 AM, Paul Angus >> <paul.an...@shapeblue.com> >> wrote: >> >>> Hi All, >>> >>> I'm running an initiative to improve CloudStack logging. I'm going >>> to be analysing logs and re-categorising the DEBUG, INFO, WARN and >>> ERROR where appropriate such that we can reduce the default logging >>> level to INFO without losing important troubleshooting information, >>> while at the same time making the management-server log far more >>> readable for operational admins. >>> >>> To achieve this I need logs to work with. >>> >>> Please could anyone interested upload logs to: >>> https://shapeblue.brickftp.com >>> >>> The logs will be treated in the strictest confidence and public >>> download of the logs will not be available. >>> I'm especially interested in logs which you found particularly >>> difficult to read or specific messages which you think are >>> misscategorised >>> >>> If you wish, put comments at the top of the logs or send a message >>> directly or via the mailing lists pointing out specific issues. >>> Please compress the logs! >>> >>> Kind regards, >>> >>> Paul Angus >>> >>> >>> Regards, >>> >>> Paul Angus >>> >>> paul.an...@shapeblue.com >>> www.shapeblue.com >>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue >>> >>
