Re: getting rid of all-machines.log
... > > I may be misremembering, but at the time that was the preferred > approach. But > > then someone said Go's inbuilt syslog APIs were broke, so the compromise > was to > > use rsyslog forwarding. > > > > Does anyone else recall why it may have been said that Go's syslog APIs > are broken? > > The reconnect logic is broken in all the version's of the syslog api. > The general consensus is that package is a mistake and should not be > used. > > I believe there is also an issue where we couldn't format the logs the way we wanted to. (The prefix/timestamp are added by the package and cannot be configured). John =:-> -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: getting rid of all-machines.log
On Wed, Aug 27, 2014 at 3:04 PM, Ian Booth wrote: > > > On 27/08/14 14:56, John Meinel wrote: >> My personal vote would be: >> a) Use something that can write directly to multiple syslog receivers over >> a TLS encrypted connection from inside the jujud binary (e.g. don't use >> rsyslog to read the log files and forward them on to the state servers, but >> just write directly) > > I may be misremembering, but at the time that was the preferred approach. But > then someone said Go's inbuilt syslog APIs were broke, so the compromise was > to > use rsyslog forwarding. > > Does anyone else recall why it may have been said that Go's syslog APIs are > broken? The reconnect logic is broken in all the version's of the syslog api. The general consensus is that package is a mistake and should not be used. > >> I'd actually like to continue keeping a log file on local disk, as it can >> help diagnose when the problem is that you are failing to connect to >> somewhere else. > > Big +1 IMHO. I've always been a fan of logging to a text file readable in vi, > emacs, whatever. Lowest common denominator is often the only option for > debugging. I have no issue with logging elsewhere also, but can we please keep > the log file as well. > > > -- > Juju-dev mailing list > Juju-dev@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: getting rid of all-machines.log
On 27/08/14 14:56, John Meinel wrote: > My personal vote would be: > a) Use something that can write directly to multiple syslog receivers over > a TLS encrypted connection from inside the jujud binary (e.g. don't use > rsyslog to read the log files and forward them on to the state servers, but > just write directly) I may be misremembering, but at the time that was the preferred approach. But then someone said Go's inbuilt syslog APIs were broke, so the compromise was to use rsyslog forwarding. Does anyone else recall why it may have been said that Go's syslog APIs are broken? > I'd actually like to continue keeping a log file on local disk, as it can > help diagnose when the problem is that you are failing to connect to > somewhere else. Big +1 IMHO. I've always been a fan of logging to a text file readable in vi, emacs, whatever. Lowest common denominator is often the only option for debugging. I have no issue with logging elsewhere also, but can we please keep the log file as well. -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: getting rid of all-machines.log
My personal vote would be: a) Use something that can write directly to multiple syslog receivers over a TLS encrypted connection from inside the jujud binary (e.g. don't use rsyslog to read the log files and forward them on to the state servers, but just write directly) I'd actually like to continue keeping a log file on local disk, as it can help diagnose when the problem is that you are failing to connect to somewhere else. b) In the short term, continue using rsyslog as the aggregator on state server machines. c) Long term (not this cycle), look towards logging into an actual database (be it another collection in mongo, or something more specifically designed for handling logs like logstash, fluentd, scribe, flume, etc). Ideally we'd provide something that people could easily integrate into whatever system they wanted (and for that, all-machines.log is probably the simplest thing to then tail and pull into another system.) John =:-> On Wed, Aug 27, 2014 at 7:19 AM, Mark Ramm-Christensen (Canonical.com) < mark.ramm-christen...@canonical.com> wrote: > Thanks Horacio, it's very helpful to get this all in one place. > Adding my thoughts in line. > > On Tue, Aug 26, 2014 at 3:35 PM, Horacio Duran > wrote: > > > Here is a summary I made to help myself while following this discussion: > > > > Nate original suggestion: > > * Remove all-machines.log: Claiming it takes a lot of space and it is > not a > > multi platform solution > > > > Tim, John, Aaaron, etc: > > * all-machines.log is required for debug-log > > * makes it big and it would be nice to rotate it. > > We do not have the option of removing debug-log, and therefore there > must be a "central" repository of log data. This need not live in a > text file on the state servers called all-machines.log but it must > exist. > > -- > Juju-dev mailing list > Juju-dev@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev > -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: getting rid of all-machines.log
Thanks Horacio, it's very helpful to get this all in one place. Adding my thoughts in line. On Tue, Aug 26, 2014 at 3:35 PM, Horacio Duran wrote: > Here is a summary I made to help myself while following this discussion: > > Nate original suggestion: > * Remove all-machines.log: Claiming it takes a lot of space and it is not a > multi platform solution > > Tim, John, Aaaron, etc: > * all-machines.log is required for debug-log > * makes it big and it would be nice to rotate it. We do not have the option of removing debug-log, and therefore there must be a "central" repository of log data. This need not live in a text file on the state servers called all-machines.log but it must exist. -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: getting rid of all-machines.log
>> you'll have to be more specific, there's been a shotgun of statements in >> this thread, touching on logstash, aggregation removal, rsyslog removal, >> log rotation, deferring to stderr/stdout, 12factor apps, working with ha >> state servers, etc. > > I was referring to Nate's lumberjack package (PR seams to be gone) and > the syslog streaming change. Lumberjack works on windows as well. Just so it is easily accessable from this thread the URL for the aformentioned lumberjack is here: https://godoc.org/gopkg.in/natefinch/lumberjack.v2 -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: getting rid of all-machines.log
Hi Horatio, I don't see a way to resolve the very disparate set of opinions you've highlighted below. It's also not clear from your email who is responsible for making a decision. I suggest reframing the discussion as user stories. ie * As a Juju user with a Windows workstation I want to use juju debug-log (this should work today) * As a Juju user who has deployed a mixed environment (to the best of my knowledge there is no requirement for the state servers to run on Windows, this seems contrary to Canonical's goal of Free Software) containing a windows workload charm I want to view the logs from that charm. Dave On Wed, Aug 27, 2014 at 5:35 AM, Horacio Duran wrote: > Hey, In an effort to move forward with juju's windows integration I have > summarized what seems to be the core points of this discussion to the best > of my ability (please excuse me if I missed or misunderstood something). > The two core points of discussion on this thread are: > * should we remove all-machines.log: which has been voted against, at least > for the moment, since it is used for debug-log. > * how do we support logging in windows: The strongest suggestions here are a > syslog package by gabriel and logging into MongoDB by Gustavo. > > We do require some decision on the front of windows logging to have a > complete windows support. Ideally we need senior citizens of juju dev > community to weight into this in order to get a clear path to follow. > > Here is a summary I made to help myself while following this discussion: > > Nate original suggestion: > * Remove all-machines.log: Claiming it takes a lot of space and it is not a > multi platform solution > > Tim, John, Aaaron, etc: > * all-machines.log is required for debug-log > * makes it big and it would be nice to rotate it. > > Nate, gabriel: > * keep all-machines.log > * use a go-only solution (syslog package with ports from gabriel for > windows) > John > * agrees. > > Nate, gabriel: > * remove rsyslog from al OSes in favor of one solution that fits all OSes > * Replace with go only solution. > > Dave: > * Dont mind about the logs, make it just output and let external tools > handle logging and rotation. > * all-machines.log might be a bit bloated and it could contain less data > that is more useful. > (Here is the reference to 12factor that will later be attibuted to nate) > Ian: > * Agrees with dave, yet we should provide a rolling mechanism. > > Gabriel: > * Windows does not support capturing stdout as a logging mechanism, it > requires to explicitly log into the event log. > * Thinks that using rsyslog to stream logs from agents to state server is > too much overhead on external tools. > * Proposes replacing external rsyslog with in app solution for the case of > streaming logs. > * Alternative solution, he does not recommend it, to create (and bundle with > jujud.exe) a wrapper for windows only. > > Gustavo: > * Present a possible alternative by using a MongoDB "capped collection" > which will suit our use cases but does not recommend it because of the idea > needs maturing on some details. > > Matt: > * We should provide the option to log to stdout or syslog. > > Kapil: > * Supports Gustavo's idea of logging in a structured form into Mongo as it > makes sense to dump structured data with structure instead of serializing it > to be de-serialized later. > * We can send also messages to syslog and let OPS people collec them > themselves. > > Gabriel (summarizing) > * I will be looking into event log for local windows logging. This will > probably require writing a package. > * the syslog change will solve in the sort term, the aggregation issue from > Windows nodes (when something better comes along, I will personally send a > case of beer or ice-cream...or both, to whomever removes syslog as a > dependency) > * lumberjack works *now* for local logging on both Windows and Ubuntu. It > simply removes 2 dependencies (for logging) with just a few lines of code... > > -- > Horacio > > > -- > Juju-dev mailing list > Juju-dev@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev > -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: getting rid of all-machines.log
Hey, In an effort to move forward with juju's windows integration I have summarized what seems to be the core points of this discussion to the best of my ability (please excuse me if I missed or misunderstood something). The two core points of discussion on this thread are: * should we remove all-machines.log: which has been voted against, at least for the moment, since it is used for debug-log. * how do we support logging in windows: The strongest suggestions here are a syslog package by gabriel and logging into MongoDB by Gustavo. We do require some decision on the front of windows logging to have a complete windows support. Ideally we need senior citizens of juju dev community to weight into this in order to get a clear path to follow. Here is a summary I made to help myself while following this discussion: Nate original suggestion: * Remove all-machines.log: Claiming it takes a lot of space and it is not a multi platform solution Tim, John, Aaaron, etc: * all-machines.log is required for debug-log * makes it big and it would be nice to rotate it. Nate, gabriel: * keep all-machines.log * use a go-only solution (syslog package with ports from gabriel for windows) John * agrees. Nate, gabriel: * remove rsyslog from al OSes in favor of one solution that fits all OSes * Replace with go only solution. Dave: * Dont mind about the logs, make it just output and let external tools handle logging and rotation. * all-machines.log might be a bit bloated and it could contain less data that is more useful. (Here is the reference to 12factor that will later be attibuted to nate) Ian: * Agrees with dave, yet we should provide a rolling mechanism. Gabriel: * Windows does not support capturing stdout as a logging mechanism, it requires to explicitly log into the event log. * Thinks that using rsyslog to stream logs from agents to state server is too much overhead on external tools. * Proposes replacing external rsyslog with in app solution for the case of streaming logs. * Alternative solution, he does not recommend it, to create (and bundle with jujud.exe) a wrapper for windows only. Gustavo: * Present a possible alternative by using a MongoDB "capped collection" which will suit our use cases but does not recommend it because of the idea needs maturing on some details. Matt: * We should provide the option to log to stdout or syslog. Kapil: * Supports Gustavo's idea of logging in a structured form into Mongo as it makes sense to dump structured data with structure instead of serializing it to be de-serialized later. * We can send also messages to syslog and let OPS people collec them themselves. Gabriel (summarizing) * I will be looking into event log for local windows logging. This will probably require writing a package. * the syslog change will solve in the sort term, the aggregation issue from Windows nodes (when something better comes along, I will personally send a case of beer or ice-cream...or both, to whomever removes syslog as a dependency) * lumberjack works *now* for local logging on both Windows and Ubuntu. It simply removes 2 dependencies (for logging) with just a few lines of code... -- Horacio -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Triggering Server Restarts
On Tue, Aug 26, 2014 at 3:27 AM, Andrew Wilkins wrote: > It doesn't have to be the tomb itself; it can be an interface which is > implemented by a type wrapping a tomb. My point was that it seems > unnecessary to add channels and signalling when we already have that with > Runner+Tomb. Yeah, agreed: a minimal implementation where Server just exposed an interface that got passed into FacadeFactory would be fine, for a given value of fine. If we're touching those args, though, let's make it a params struct so we have less pain to deal with next time it changes ;). Cheers William -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
