getting rid of all-machines.log

[Nate Finch]

all-machines.log seems both redundant and a ticking time bomb of disk space
usage.  Do we really need it?  Can we drop it and maybe later schedule some
time to use something like logstash that is both more featureful and is
cross platform compatible (unlike rsyslog)?
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Tim Penhey]

On 06/08/14 16:11, Nate Finch wrote:
> all-machines.log seems both redundant and a ticking time bomb of disk
> space usage.  Do we really need it?  Can we drop it and maybe later
> schedule some time to use something like logstash that is both more
> featureful and is cross platform compatible (unlike rsyslog)?

not yet...

debug-log uses all-machines.log, we cannot get rid of it at this stage.

We can't drop it until a replacement is in place.

Tim


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[John Meinel]

all-machines.log is where we aggregate the messages from all
machines/units/etc.
It is likely to get big, which is why we want log rotate, but if you want
to be able to "juju debug-log" and actually get the feed of everything that
is going on, that needs to be *somewhere*. And yes, we'd like to switch to
something like log stash instead, but until we get there we do still need
it.
John
=:->



On Wed, Aug 6, 2014 at 4:13 PM, Tim Penhey  wrote:

> On 06/08/14 16:11, Nate Finch wrote:
> > all-machines.log seems both redundant and a ticking time bomb of disk
> > space usage.  Do we really need it?  Can we drop it and maybe later
> > schedule some time to use something like logstash that is both more
> > featureful and is cross platform compatible (unlike rsyslog)?
>
> not yet...
>
> debug-log uses all-machines.log, we cannot get rid of it at this stage.
>
> We can't drop it until a replacement is in place.
>
> Tim
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Nate Finch]

Fair enough, I forgot about debug-log.  Do we have an idea of how we'll
aggregate logs from Windows machines?  rsyslog won't run there.   We might
be able to do a go-only solution by using the syslog package gabriel
made a port of it that will run on Windows and supports TLS:
https://github.com/gabriel-samfira/syslogThen we'd just have to give
loggo a writer that writes to both the log file and syslog.


On Wed, Aug 6, 2014 at 10:24 AM, John Meinel  wrote:

> all-machines.log is where we aggregate the messages from all
> machines/units/etc.
> It is likely to get big, which is why we want log rotate, but if you want
> to be able to "juju debug-log" and actually get the feed of everything that
> is going on, that needs to be *somewhere*. And yes, we'd like to switch to
> something like log stash instead, but until we get there we do still need
> it.
> John
> =:->
>
>
>
> On Wed, Aug 6, 2014 at 4:13 PM, Tim Penhey 
> wrote:
>
>> On 06/08/14 16:11, Nate Finch wrote:
>> > all-machines.log seems both redundant and a ticking time bomb of disk
>> > space usage.  Do we really need it?  Can we drop it and maybe later
>> > schedule some time to use something like logstash that is both more
>> > featureful and is cross platform compatible (unlike rsyslog)?
>>
>> not yet...
>>
>> debug-log uses all-machines.log, we cannot get rid of it at this stage.
>>
>> We can't drop it until a replacement is in place.
>>
>> Tim
>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[John Meinel]

That's probably a reasonable starting point. With the caveat that we know
we don't really want to just use rsyslog forever. Just didn't get scheduled
for this cycle.

John
=:->



On Wed, Aug 6, 2014 at 4:42 PM, Nate Finch  wrote:

> Fair enough, I forgot about debug-log.  Do we have an idea of how we'll
> aggregate logs from Windows machines?  rsyslog won't run there.   We might
> be able to do a go-only solution by using the syslog package gabriel
> made a port of it that will run on Windows and supports TLS:
> https://github.com/gabriel-samfira/syslogThen we'd just have to give
> loggo a writer that writes to both the log file and syslog.
>
>
> On Wed, Aug 6, 2014 at 10:24 AM, John Meinel 
> wrote:
>
>> all-machines.log is where we aggregate the messages from all
>> machines/units/etc.
>> It is likely to get big, which is why we want log rotate, but if you want
>> to be able to "juju debug-log" and actually get the feed of everything that
>> is going on, that needs to be *somewhere*. And yes, we'd like to switch to
>> something like log stash instead, but until we get there we do still need
>> it.
>> John
>> =:->
>>
>>
>>
>> On Wed, Aug 6, 2014 at 4:13 PM, Tim Penhey 
>> wrote:
>>
>>> On 06/08/14 16:11, Nate Finch wrote:
>>> > all-machines.log seems both redundant and a ticking time bomb of disk
>>> > space usage.  Do we really need it?  Can we drop it and maybe later
>>> > schedule some time to use something like logstash that is both more
>>> > featureful and is cross platform compatible (unlike rsyslog)?
>>>
>>> not yet...
>>>
>>> debug-log uses all-machines.log, we cannot get rid of it at this stage.
>>>
>>> We can't drop it until a replacement is in place.
>>>
>>> Tim
>>>
>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>
>>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Aaron Bentley]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 14-08-06 10:11 AM, Nate Finch wrote:
> all-machines.log seems both redundant and a ticking time bomb of
> disk space usage.  Do we really need it?  Can we drop it and maybe
> later schedule some time to use something like logstash that is
> both more featureful and is cross platform compatible (unlike
> rsyslog)?

all-machines.log is one of the files juju CI captures for failure
investigation.  If switching to something else, please ensure it is as
informative and all-machines.log and that you coordinate the switch
with us.

Aaron
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEcBAEBAgAGBQJT4ntlAAoJEK84cMOcf+9hJsUIALsJSTPtPo0JmDqqiUb81DsO
/OVNobQLiW47yTLlp8aCPXJTH6tswK+vgMhSFbYPqcD2pVY4zZGgwSRapZbOi1x9
SFnNB50onMTej+QKOiOYbLlvy9MzZGNnV8e7KAFnkQ6rZ9r8twChf1IuBl57nWJV
oqmWnbVdVmkjvHCCrEmK2xNfG4Y18pP5sLYLbqN3GinNz+LhNLzuDczYjN2gHZBG
GptDTWui6P+yxdkL0UTKs2JvisQWHYD2zBG2A8dwhdK864bVA1HJig5ryBhbP+AX
RkY9bkE0NP4D6R+SecLMlROrAdZasJh5KGHKq5c6VCy6ZvnI2d0DtqZ/epAW9xU=
=py/M
-END PGP SIGNATURE-

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[jaquilina]

there is an rsyslog for windows but obviously it requires a license

http://www.rsyslog.com/windows-agent/

On 2014-08-06 16:42, Nate Finch wrote:

Fair enough, I forgot about debug-log.  Do we have an idea of how
well aggregate logs from Windows machines?  rsyslog wont run there.
  We might be able to do a go-only solution by using the syslog
package gabriel made a port of it that will run on Windows and
supports TLS: https://github.com/gabriel-samfira/syslog [4]    Then
wed just have to give loggo a writer that writes to both the log file
and syslog.

On Wed, Aug 6, 2014 at 10:24 AM, John Meinel  wrote:


all-machines.log is where we aggregate the messages from all
machines/units/etc.
It is likely to get big, which is why we want log rotate, but if you
want to be able to "juju debug-log" and actually get the feed of
everything that is going on, that needs to be *somewhere*. And yes,
wed like to switch to something like log stash instead, but until we
get there we do still need it.
John
=:->

On Wed, Aug 6, 2014 at 4:13 PM, Tim Penhey  wrote:


On 06/08/14 16:11, Nate Finch wrote:
> all-machines.log seems both redundant and a ticking time bomb
of disk
> space usage.  Do we really need it?  Can we drop it and maybe
later
> schedule some time to use something like logstash that is both
more
> featureful and is cross platform compatible (unlike rsyslog)?

not yet...

debug-log uses all-machines.log, we cannot get rid of it at this
stage.

We cant drop it until a replacement is in place.

Tim

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com [1]
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju-dev [2]




Links:
--
[1] mailto:Juju-dev@lists.ubuntu.com
[2] https://lists.ubuntu.com/mailman/listinfo/juju-dev
[3] mailto:tim.pen...@canonical.com
[4] https://github.com/gabriel-samfira/syslog
[5] mailto:j...@arbash-meinel.com


--
Regards,
Jonathan Aquilina
Founder Eagle Eye T

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Gabriel Samfira]

The standard Go syslog package works great (with slight changes). As a 
bonus, it would be great to remove the dependency on rsyslog from all 
non bootstrap nodes if all we need it to do is tail the juju log and 
send it to an agregator :).

As a native windows alternative, If we really do need to log to 
something else then a file on a windows machine, we may as well go with 
the built in event log.

Gabriel

On 08.08.2014 12:41, jaquilina wrote:
> there is an rsyslog for windows but obviously it requires a license
>
> http://www.rsyslog.com/windows-agent/
>
> On 2014-08-06 16:42, Nate Finch wrote:
>> Fair enough, I forgot about debug-log.  Do we have an idea of how
>> well aggregate logs from Windows machines?  rsyslog wont run there.
>>   We might be able to do a go-only solution by using the syslog
>> package gabriel made a port of it that will run on Windows and
>> supports TLS: https://github.com/gabriel-samfira/syslog [4]  Then
>> wed just have to give loggo a writer that writes to both the log file
>> and syslog.
>>
>> On Wed, Aug 6, 2014 at 10:24 AM, John Meinel > [5]> wrote:
>>
>>> all-machines.log is where we aggregate the messages from all
>>> machines/units/etc.
>>> It is likely to get big, which is why we want log rotate, but if you
>>> want to be able to "juju debug-log" and actually get the feed of
>>> everything that is going on, that needs to be *somewhere*. And yes,
>>> wed like to switch to something like log stash instead, but until we
>>> get there we do still need it.
>>> John
>>> =:->
>>>
>>> On Wed, Aug 6, 2014 at 4:13 PM, Tim Penhey >> [3]> wrote:
>>>
 On 06/08/14 16:11, Nate Finch wrote:
 > all-machines.log seems both redundant and a ticking time bomb
 of disk
 > space usage.  Do we really need it?  Can we drop it and maybe
 later
 > schedule some time to use something like logstash that is both
 more
 > featureful and is cross platform compatible (unlike rsyslog)?

 not yet...

 debug-log uses all-machines.log, we cannot get rid of it at this
 stage.

 We cant drop it until a replacement is in place.

 Tim

 -- 
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com [1]
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev [2]
>>
>>
>>
>> Links:
>> --
>> [1] mailto:Juju-dev@lists.ubuntu.com
>> [2] https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> [3] mailto:tim.pen...@canonical.com
>> [4] https://github.com/gabriel-samfira/syslog
>> [5] mailto:j...@arbash-meinel.com
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Nate Finch]

I'm all for taking rsyslog out of the equation even on linux... what's nice
about it is that, because we'd be writing log output separately to both the
remote syslog and to the local file log, we wouldn't need to worry about
log rotation of the local log screwing up what gets sent to the remote
syslog.  It's trivial to have loggo write to two or more targets (the file
plus one for each state server).




On Fri, Aug 8, 2014 at 8:58 AM, Gabriel Samfira <
gsamf...@cloudbasesolutions.com> wrote:

> The standard Go syslog package works great (with slight changes). As a
> bonus, it would be great to remove the dependency on rsyslog from all
> non bootstrap nodes if all we need it to do is tail the juju log and
> send it to an agregator :).
>
> As a native windows alternative, If we really do need to log to
> something else then a file on a windows machine, we may as well go with
> the built in event log.
>
> Gabriel
>
> On 08.08.2014 12:41, jaquilina wrote:
> > there is an rsyslog for windows but obviously it requires a license
> >
> > http://www.rsyslog.com/windows-agent/
> >
> > On 2014-08-06 16:42, Nate Finch wrote:
> >> Fair enough, I forgot about debug-log.  Do we have an idea of how
> >> well aggregate logs from Windows machines?  rsyslog wont run there.
> >>   We might be able to do a go-only solution by using the syslog
> >> package gabriel made a port of it that will run on Windows and
> >> supports TLS: https://github.com/gabriel-samfira/syslog [4]  Then
> >> wed just have to give loggo a writer that writes to both the log file
> >> and syslog.
> >>
> >> On Wed, Aug 6, 2014 at 10:24 AM, John Meinel  >> [5]> wrote:
> >>
> >>> all-machines.log is where we aggregate the messages from all
> >>> machines/units/etc.
> >>> It is likely to get big, which is why we want log rotate, but if you
> >>> want to be able to "juju debug-log" and actually get the feed of
> >>> everything that is going on, that needs to be *somewhere*. And yes,
> >>> wed like to switch to something like log stash instead, but until we
> >>> get there we do still need it.
> >>> John
> >>> =:->
> >>>
> >>> On Wed, Aug 6, 2014 at 4:13 PM, Tim Penhey  >>> [3]> wrote:
> >>>
>  On 06/08/14 16:11, Nate Finch wrote:
>  > all-machines.log seems both redundant and a ticking time bomb
>  of disk
>  > space usage.  Do we really need it?  Can we drop it and maybe
>  later
>  > schedule some time to use something like logstash that is both
>  more
>  > featureful and is cross platform compatible (unlike rsyslog)?
> 
>  not yet...
> 
>  debug-log uses all-machines.log, we cannot get rid of it at this
>  stage.
> 
>  We cant drop it until a replacement is in place.
> 
>  Tim
> 
>  --
>  Juju-dev mailing list
>  Juju-dev@lists.ubuntu.com [1]
>  Modify settings or unsubscribe at:
>  https://lists.ubuntu.com/mailman/listinfo/juju-dev [2]
> >>
> >>
> >>
> >> Links:
> >> --
> >> [1] mailto:Juju-dev@lists.ubuntu.com
> >> [2] https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >> [3] mailto:tim.pen...@canonical.com
> >> [4] https://github.com/gabriel-samfira/syslog
> >> [5] mailto:j...@arbash-meinel.com
> >
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[David Britton]

On Fri, Aug 08, 2014 at 12:03:21PM -0400, Nate Finch wrote:
[...]
> remote syslog and to the local file log, we wouldn't need to worry about
> log rotation of the local log screwing up what gets sent to the remote

Do the standard rsyslog log rotation mechanisms not function well?

On Windows, what about the event log (which has remote
viewing/aggregation capabilities built in)?

-- 
David Britton 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Nate Finch]

So, rsyslog rotation works fine on Linux, but we can't do that on Windows.
 If we have to do something different for Windows, I'd rather just do one
thing which is cross platform compatible for all our OSes, and not have to
support a different configuration for each OS.  Doing it all in-application
also insulates us from external dependencies... if some future or past
Ubuntu series (or CentOS) has a different version of rsyslog, it could
behave differently / require a different configuration, etc.



On Fri, Aug 8, 2014 at 12:40 PM, David Britton 
wrote:

> On Fri, Aug 08, 2014 at 12:03:21PM -0400, Nate Finch wrote:
> [...]
> > remote syslog and to the local file log, we wouldn't need to worry about
> > log rotation of the local log screwing up what gets sent to the remote
>
> Do the standard rsyslog log rotation mechanisms not function well?
>
> On Windows, what about the event log (which has remote
> viewing/aggregation capabilities built in)?
>
> --
> David Britton 
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[David Cheney]

Ian asked me to post my thoughts here.

I am not in favour of applications rolling their own logs, I believe
that applications should not know anything about their log output,
they should just dump it all to stdout and another process should take
care of shuttling the data, logging it, culling it, whatever.

This is summarised here, http://12factor.net/logs

I think our current system with rsyslog is working fine and there is
no reason to remove it.

The problems with all-machines.log being to large is independent of
log rolling or any of these other arguments. We simply log too much.
There would be no request for log rolling it all-machines.log
contained only useful data.

Dave

On Sat, Aug 9, 2014 at 2:58 AM, Nate Finch  wrote:
> So, rsyslog rotation works fine on Linux, but we can't do that on Windows.
> If we have to do something different for Windows, I'd rather just do one
> thing which is cross platform compatible for all our OSes, and not have to
> support a different configuration for each OS.  Doing it all in-application
> also insulates us from external dependencies... if some future or past
> Ubuntu series (or CentOS) has a different version of rsyslog, it could
> behave differently / require a different configuration, etc.
>
>
>
> On Fri, Aug 8, 2014 at 12:40 PM, David Britton 
> wrote:
>>
>> On Fri, Aug 08, 2014 at 12:03:21PM -0400, Nate Finch wrote:
>> [...]
>> > remote syslog and to the local file log, we wouldn't need to worry about
>> > log rotation of the local log screwing up what gets sent to the remote
>>
>> Do the standard rsyslog log rotation mechanisms not function well?
>>
>> On Windows, what about the event log (which has remote
>> viewing/aggregation capabilities built in)?
>>
>> --
>> David Britton 
>
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Ian Booth]

Just to back up Dave's arguments - all sys admins I know would be a big -1 on
Juju doing it's own log rolling. It's a recipe for lost log files, missing data
etc. It's a mixing of responsibilities that should be handled separately.

Just on the volume point Dave raised - we do log a lot but that's also an
orthogonal issue. Even if we logged less we'd still need a rolling mechanism.

On 14/08/14 13:13, David Cheney wrote:
> Ian asked me to post my thoughts here.
> 
> I am not in favour of applications rolling their own logs, I believe
> that applications should not know anything about their log output,
> they should just dump it all to stdout and another process should take
> care of shuttling the data, logging it, culling it, whatever.
> 
> This is summarised here, http://12factor.net/logs
> 
> I think our current system with rsyslog is working fine and there is
> no reason to remove it.
> 
> The problems with all-machines.log being to large is independent of
> log rolling or any of these other arguments. We simply log too much.
> There would be no request for log rolling it all-machines.log
> contained only useful data.
> 
> Dave
> 
> On Sat, Aug 9, 2014 at 2:58 AM, Nate Finch  wrote:
>> So, rsyslog rotation works fine on Linux, but we can't do that on Windows.
>> If we have to do something different for Windows, I'd rather just do one
>> thing which is cross platform compatible for all our OSes, and not have to
>> support a different configuration for each OS.  Doing it all in-application
>> also insulates us from external dependencies... if some future or past
>> Ubuntu series (or CentOS) has a different version of rsyslog, it could
>> behave differently / require a different configuration, etc.
>>
>>
>>
>> On Fri, Aug 8, 2014 at 12:40 PM, David Britton 
>> wrote:
>>>
>>> On Fri, Aug 08, 2014 at 12:03:21PM -0400, Nate Finch wrote:
>>> [...]
 remote syslog and to the local file log, we wouldn't need to worry about
 log rotation of the local log screwing up what gets sent to the remote
>>>
>>> Do the standard rsyslog log rotation mechanisms not function well?
>>>
>>> On Windows, what about the event log (which has remote
>>> viewing/aggregation capabilities built in)?
>>>
>>> --
>>> David Britton 
>>
>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
> 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Gabriel Samfira]

Please forgive the lengthy response :). The following are just a few 
thoughts regarding this subject.

On 14.08.2014 06:13, David Cheney wrote:
> Ian asked me to post my thoughts here.
>
> I am not in favour of applications rolling their own logs, I believe
> that applications should not know anything about their log output,
> they should just dump it all to stdout and another process should take
> care of shuttling the data, logging it, culling it, whatever.
> This is summarised here, http://12factor.net/logs
This philosophy works under the assumption that all applications that 
run as services, will only do so on linux/unix systems. Services on 
Windows have no way to capture stdout unless we use a wrapper to start 
the application as a service, capture its stdout and later on, do 
something with it. Windows services mostly stream to local event log. 
Given that we want juju to run both on linux and windows, there needs to 
be a common denominator.

The ability for juju to write its own logfile *works* across most, if 
not all platforms (as it does for other apps such as nginx, apache, 
openstack components, etc). If we want to have log handling outside of 
juju, we need to implement a separate configuration on separate 
platforms (adding more moving parts). Doing this in juju removes 
external dependencies, and gives us a unified way of handling logs 
across platforms.

I can understand how this philosophy is ok for applications that are 
meant to run on a PaaS where the application needs to adapt to its 
environment, and where the PaaS prefers to not allow the developer to 
make mistakes when it comes to saving the log of the application, but we 
are not running juju on a PaaS. Juju will only be run on a IaaS or on 
bare metal, where you are adapting the environment to your needs. What 
is good for a PaaS running apps mostly on Linux, is not necessarily good 
when you are targeting bare metal or IaaS using multiple platforms. We 
have to keep in mind that not all platforms are created equal. Some have 
limitations that others do not.
>
> I think our current system with rsyslog is working fine and there is
> no reason to remove it.
There is no rsyslog for windows (well, not free at least). Also, 
installing a 3rd party service that needs to be updated separately from 
the standard windows workflow opens up the system to vulnerabilities if 
not done right.

Windows aside, on non-bootstrap nodes we currently only use rsyslog to 
tail the logs generated by juju (and written by an append redirect from 
upstart), and then stream those logs to the a state machine. So you have 
3 components that have to work together just for the end result of 
having a log streamed to a state machine. IMO there are 2 moving parts 
too many.

Most languages have a syslog package that can stream directly to syslog. 
IMHO, the rsyslog forwarder is one external dependency that should be 
easily replaced. The less moving parts, the better. This also allows 
juju to stream logs from all platforms directly to syslog (its just a 
TCP/UDP connection in the end...).

This is just my 2 cents. Its up to you guys in the end. As far as 
windows is concerned, as long as I have clear guidelines defined, I can 
adapt the PRs.

An option would be to start writing a wrapper for Windows that will be 
bundled along side jujud.exe...but it would be a platform specific 
thing. This is something I would rather not do if I have a choice.

Cheers,
Gabriel
>
> The problems with all-machines.log being to large is independent of
> log rolling or any of these other arguments. We simply log too much.
> There would be no request for log rolling it all-machines.log
> contained only useful data.
>
> Dave
>
> On Sat, Aug 9, 2014 at 2:58 AM, Nate Finch  wrote:
>> So, rsyslog rotation works fine on Linux, but we can't do that on Windows.
>> If we have to do something different for Windows, I'd rather just do one
>> thing which is cross platform compatible for all our OSes, and not have to
>> support a different configuration for each OS.  Doing it all in-application
>> also insulates us from external dependencies... if some future or past
>> Ubuntu series (or CentOS) has a different version of rsyslog, it could
>> behave differently / require a different configuration, etc.
>>
>>
>>
>> On Fri, Aug 8, 2014 at 12:40 PM, David Britton 
>> wrote:
>>> On Fri, Aug 08, 2014 at 12:03:21PM -0400, Nate Finch wrote:
>>> [...]
 remote syslog and to the local file log, we wouldn't need to worry about
 log rotation of the local log screwing up what gets sent to the remote
>>> Do the standard rsyslog log rotation mechanisms not function well?
>>>
>>> On Windows, what about the event log (which has remote
>>> viewing/aggregation capabilities built in)?
>>>
>>> --
>>> David Britton 
>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>

-- 
Juju-dev mailing list
Ju

Re: getting rid of all-machines.log

[Gabriel Samfira]

On 14.08.2014 06:20, Ian Booth wrote:
> Just to back up Dave's arguments - all sys admins I know would be a big -1 on
> Juju doing it's own log rolling. It's a recipe for lost log files, missing 
> data
> etc. It's a mixing of responsibilities that should be handled separately.
I am unsure how juju rotating its own logs can lead to loss of data. Any 
input on this would be appreciated.
>
> Just on the volume point Dave raised - we do log a lot but that's also an
> orthogonal issue. Even if we logged less we'd still need a rolling mechanism.
>
> On 14/08/14 13:13, David Cheney wrote:
>> Ian asked me to post my thoughts here.
>>
>> I am not in favour of applications rolling their own logs, I believe
>> that applications should not know anything about their log output,
>> they should just dump it all to stdout and another process should take
>> care of shuttling the data, logging it, culling it, whatever.
>>
>> This is summarised here, http://12factor.net/logs
>>
>> I think our current system with rsyslog is working fine and there is
>> no reason to remove it.
>>
>> The problems with all-machines.log being to large is independent of
>> log rolling or any of these other arguments. We simply log too much.
>> There would be no request for log rolling it all-machines.log
>> contained only useful data.
>>
>> Dave
>>
>> On Sat, Aug 9, 2014 at 2:58 AM, Nate Finch  wrote:
>>> So, rsyslog rotation works fine on Linux, but we can't do that on Windows.
>>> If we have to do something different for Windows, I'd rather just do one
>>> thing which is cross platform compatible for all our OSes, and not have to
>>> support a different configuration for each OS.  Doing it all in-application
>>> also insulates us from external dependencies... if some future or past
>>> Ubuntu series (or CentOS) has a different version of rsyslog, it could
>>> behave differently / require a different configuration, etc.
>>>
>>>
>>>
>>> On Fri, Aug 8, 2014 at 12:40 PM, David Britton 
>>> wrote:
 On Fri, Aug 08, 2014 at 12:03:21PM -0400, Nate Finch wrote:
 [...]
> remote syslog and to the local file log, we wouldn't need to worry about
> log rotation of the local log screwing up what gets sent to the remote
 Do the standard rsyslog log rotation mechanisms not function well?

 On Windows, what about the event log (which has remote
 viewing/aggregation capabilities built in)?

 --
 David Britton 
>>>
>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Gustavo Niemeyer]

As a side note, MongoDB offers a "capped collection" mechanism with
the semantics that you can insert rows at will, and it rolls around
automatically by replacing oldest entries with the newest ones. This
tends to be a very convenient way to do structured logging, both on
the writing and on the reading side. Using that as a general rsyslog
might be unwise, as there are a number of details to get right and the
volume may get wild depending on the application, but at least for the
juju implementation itself it might be quite convenient. Imagine being
able to ask "please tell me the output of the last run of the
db-relation-joined hook on unit db/3".

On Wed, Aug 6, 2014 at 11:24 AM, John Meinel  wrote:
> all-machines.log is where we aggregate the messages from all
> machines/units/etc.
> It is likely to get big, which is why we want log rotate, but if you want to
> be able to "juju debug-log" and actually get the feed of everything that is
> going on, that needs to be *somewhere*. And yes, we'd like to switch to
> something like log stash instead, but until we get there we do still need
> it.
> John
> =:->
>
>
>
> On Wed, Aug 6, 2014 at 4:13 PM, Tim Penhey  wrote:
>>
>> On 06/08/14 16:11, Nate Finch wrote:
>> > all-machines.log seems both redundant and a ticking time bomb of disk
>> > space usage.  Do we really need it?  Can we drop it and maybe later
>> > schedule some time to use something like logstash that is both more
>> > featureful and is cross platform compatible (unlike rsyslog)?
>>
>> not yet...
>>
>> debug-log uses all-machines.log, we cannot get rid of it at this stage.
>>
>> We can't drop it until a replacement is in place.
>>
>> Tim
>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>

-- 
gustavo @ http://niemeyer.net

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Matt Rae]

Many operations teams already have a standard log collecting systems. I
think it would be best to be flexible enough to work in environments with
existing systems.

Standard ways are logging to syslog so any syslog implementation can be
used, or logging to stdout so a supervisor like djb daemontools can collect
and rotate logs.


On Thu, Aug 14, 2014 at 3:00 AM, Gabriel Samfira <
gsamf...@cloudbasesolutions.com> wrote:

> On 14.08.2014 06:20, Ian Booth wrote:
> > Just to back up Dave's arguments - all sys admins I know would be a big
> -1 on
> > Juju doing it's own log rolling. It's a recipe for lost log files,
> missing data
> > etc. It's a mixing of responsibilities that should be handled separately.
> I am unsure how juju rotating its own logs can lead to loss of data. Any
> input on this would be appreciated.
> >
> > Just on the volume point Dave raised - we do log a lot but that's also an
> > orthogonal issue. Even if we logged less we'd still need a rolling
> mechanism.
> >
> > On 14/08/14 13:13, David Cheney wrote:
> >> Ian asked me to post my thoughts here.
> >>
> >> I am not in favour of applications rolling their own logs, I believe
> >> that applications should not know anything about their log output,
> >> they should just dump it all to stdout and another process should take
> >> care of shuttling the data, logging it, culling it, whatever.
> >>
> >> This is summarised here, http://12factor.net/logs
> >>
> >> I think our current system with rsyslog is working fine and there is
> >> no reason to remove it.
> >>
> >> The problems with all-machines.log being to large is independent of
> >> log rolling or any of these other arguments. We simply log too much.
> >> There would be no request for log rolling it all-machines.log
> >> contained only useful data.
> >>
> >> Dave
> >>
> >> On Sat, Aug 9, 2014 at 2:58 AM, Nate Finch 
> wrote:
> >>> So, rsyslog rotation works fine on Linux, but we can't do that on
> Windows.
> >>> If we have to do something different for Windows, I'd rather just do
> one
> >>> thing which is cross platform compatible for all our OSes, and not
> have to
> >>> support a different configuration for each OS.  Doing it all
> in-application
> >>> also insulates us from external dependencies... if some future or past
> >>> Ubuntu series (or CentOS) has a different version of rsyslog, it could
> >>> behave differently / require a different configuration, etc.
> >>>
> >>>
> >>>
> >>> On Fri, Aug 8, 2014 at 12:40 PM, David Britton <
> david.brit...@canonical.com>
> >>> wrote:
>  On Fri, Aug 08, 2014 at 12:03:21PM -0400, Nate Finch wrote:
>  [...]
> > remote syslog and to the local file log, we wouldn't need to worry
> about
> > log rotation of the local log screwing up what gets sent to the
> remote
>  Do the standard rsyslog log rotation mechanisms not function well?
> 
>  On Windows, what about the event log (which has remote
>  viewing/aggregation capabilities built in)?
> 
>  --
>  David Britton 
> >>>
> >>>
> >>> --
> >>> Juju-dev mailing list
> >>> Juju-dev@lists.ubuntu.com
> >>> Modify settings or unsubscribe at:
> >>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >>>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Nate Finch]

The front page of 12factor.net says "offering maximum portability between
execution environments" ... that's exactly what I'm going for.

We're going to support Windows. Windows does not have rsyslog or logrotate.
 We *have* a solution which is cross platform.

My main concern is that if we use rsyslog on linux and something else on
Windows, now we're supporting and maintaining two things, when they could
both be the same.  Why support two things when you can support just one?

I'm sure writing to stdout and redirecting to a logfile that gets watched
by rsyslog is the linux devop standard.  But it has also proved annoyingly
complex for us to configure and maintain. We've wasted several developer
*weeks* of time getting it to work right.

Why would we continue to maintain the rsyslog stuff when it has been such
trouble for us?  Just because it's the standard, doesn't mean it's the
right tool for this specific job.

I talked with Ian about this a little this morning, and my suggestion was
that we give users a way to turn off our in-app log rotation and let them
set it up themselves external to the app (with rsyslog or whatever they
like) I just don't want to have to configure it or support it.

-Nate






On Thu, Aug 14, 2014 at 9:04 AM, Gustavo Niemeyer <
gustavo.nieme...@canonical.com> wrote:

> As a side note, MongoDB offers a "capped collection" mechanism with
> the semantics that you can insert rows at will, and it rolls around
> automatically by replacing oldest entries with the newest ones. This
> tends to be a very convenient way to do structured logging, both on
> the writing and on the reading side. Using that as a general rsyslog
> might be unwise, as there are a number of details to get right and the
> volume may get wild depending on the application, but at least for the
> juju implementation itself it might be quite convenient. Imagine being
> able to ask "please tell me the output of the last run of the
> db-relation-joined hook on unit db/3".
>
> On Wed, Aug 6, 2014 at 11:24 AM, John Meinel 
> wrote:
> > all-machines.log is where we aggregate the messages from all
> > machines/units/etc.
> > It is likely to get big, which is why we want log rotate, but if you
> want to
> > be able to "juju debug-log" and actually get the feed of everything that
> is
> > going on, that needs to be *somewhere*. And yes, we'd like to switch to
> > something like log stash instead, but until we get there we do still need
> > it.
> > John
> > =:->
> >
> >
> >
> > On Wed, Aug 6, 2014 at 4:13 PM, Tim Penhey 
> wrote:
> >>
> >> On 06/08/14 16:11, Nate Finch wrote:
> >> > all-machines.log seems both redundant and a ticking time bomb of disk
> >> > space usage.  Do we really need it?  Can we drop it and maybe later
> >> > schedule some time to use something like logstash that is both more
> >> > featureful and is cross platform compatible (unlike rsyslog)?
> >>
> >> not yet...
> >>
> >> debug-log uses all-machines.log, we cannot get rid of it at this stage.
> >>
> >> We can't drop it until a replacement is in place.
> >>
> >> Tim
> >>
> >>
> >> --
> >> Juju-dev mailing list
> >> Juju-dev@lists.ubuntu.com
> >> Modify settings or unsubscribe at:
> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >
> >
> >
> > --
> > Juju-dev mailing list
> > Juju-dev@lists.ubuntu.com
> > Modify settings or unsubscribe at:
> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >
>
> --
> gustavo @ http://niemeyer.net
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Matt Rae]

On Thu, Aug 14, 2014 at 8:23 AM, Nate Finch 
wrote:

> The front page of 12factor.net says "offering maximum portability between
> execution environments" ... that's exactly what I'm going for.
>
> We're going to support Windows. Windows does not have rsyslog or
> logrotate.  We *have* a solution which is cross platform.
>
> My main concern is that if we use rsyslog on linux and something else on
> Windows, now we're supporting and maintaining two things, when they could
> both be the same.  Why support two things when you can support just one?
>
>
I'm sure writing to stdout and redirecting to a logfile that gets watched
> by rsyslog is the linux devop standard.  But it has also proved annoyingly
> complex for us to configure and maintain. We've wasted several developer
> *weeks* of time getting it to work right.
>
> Why would we continue to maintain the rsyslog stuff when it has been such
> trouble for us?  Just because it's the standard, doesn't mean it's the
> right tool for this specific job.
>
>
I talked with Ian about this a little this morning, and my suggestion was
> that we give users a way to turn off our in-app log rotation and let them
> set it up themselves external to the app (with rsyslog or whatever they
> like) I just don't want to have to configure it or support it.
>

Yeah this sounds like a good suggestion. Maybe we can solve the problem in
a cross platform way but still have options to log to stdout/syslog in case
a user wants to integrate with an existing logging system themselves. A lot
of applications have options of logging to stdout or to syslog depending on
which you prefer.

Principle of least astonishment and taco bell programming [2] are a couple
ideas that might be good to keep in mind when deciding how we do it.

[1] http://en.wikipedia.org/wiki/Principle_of_least_astonishment
[2] http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html


>
>
> -Nate
>
>
>
>
>
>
> On Thu, Aug 14, 2014 at 9:04 AM, Gustavo Niemeyer <
> gustavo.nieme...@canonical.com> wrote:
>
>> As a side note, MongoDB offers a "capped collection" mechanism with
>> the semantics that you can insert rows at will, and it rolls around
>> automatically by replacing oldest entries with the newest ones. This
>> tends to be a very convenient way to do structured logging, both on
>> the writing and on the reading side. Using that as a general rsyslog
>> might be unwise, as there are a number of details to get right and the
>> volume may get wild depending on the application, but at least for the
>> juju implementation itself it might be quite convenient. Imagine being
>> able to ask "please tell me the output of the last run of the
>> db-relation-joined hook on unit db/3".
>>
>> On Wed, Aug 6, 2014 at 11:24 AM, John Meinel 
>> wrote:
>> > all-machines.log is where we aggregate the messages from all
>> > machines/units/etc.
>> > It is likely to get big, which is why we want log rotate, but if you
>> want to
>> > be able to "juju debug-log" and actually get the feed of everything
>> that is
>> > going on, that needs to be *somewhere*. And yes, we'd like to switch to
>> > something like log stash instead, but until we get there we do still
>> need
>> > it.
>> > John
>> > =:->
>> >
>> >
>> >
>> > On Wed, Aug 6, 2014 at 4:13 PM, Tim Penhey 
>> wrote:
>> >>
>> >> On 06/08/14 16:11, Nate Finch wrote:
>> >> > all-machines.log seems both redundant and a ticking time bomb of disk
>> >> > space usage.  Do we really need it?  Can we drop it and maybe later
>> >> > schedule some time to use something like logstash that is both more
>> >> > featureful and is cross platform compatible (unlike rsyslog)?
>> >>
>> >> not yet...
>> >>
>> >> debug-log uses all-machines.log, we cannot get rid of it at this stage.
>> >>
>> >> We can't drop it until a replacement is in place.
>> >>
>> >> Tim
>> >>
>> >>
>> >> --
>> >> Juju-dev mailing list
>> >> Juju-dev@lists.ubuntu.com
>> >> Modify settings or unsubscribe at:
>> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> >
>> >
>> >
>> > --
>> > Juju-dev mailing list
>> > Juju-dev@lists.ubuntu.com
>> > Modify settings or unsubscribe at:
>> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> >
>>
>> --
>> gustavo @ http://niemeyer.net
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Gustavo Niemeyer]

On Thu, Aug 14, 2014 at 12:23 PM, Nate Finch  wrote:
> The front page of 12factor.net says "offering maximum portability between
> execution environments" ... that's exactly what I'm going for.

This can be taken as an excuse to do just about anything.

> We're going to support Windows. Windows does not have rsyslog or logrotate.
> We have a solution which is cross platform.
>
> My main concern is that if we use rsyslog on linux and something else on
> Windows, now we're supporting and maintaining two things, when they could
> both be the same.  Why support two things when you can support just one?

Just to be clear, you really mean "why support two existing and well
known things when I can implement a third thing", right?


gustavo @ http://niemeyer.net

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Nate Finch]

On Thu, Aug 14, 2014 at 12:24 PM, Gustavo Niemeyer <
gustavo.nieme...@canonical.com> wrote:

>
> > Why support two things when you can support just one?
>
> Just to be clear, you really mean "why support two existing and well
> known things when I can implement a third thing", right?
>

Yes, that is exactly what I mean.
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Gustavo Niemeyer]

On Thu, Aug 14, 2014 at 1:35 PM, Nate Finch  wrote:
> On Thu, Aug 14, 2014 at 12:24 PM, Gustavo Niemeyer
>  wrote:
>>
>> > Why support two things when you can support just one?
>>
>> Just to be clear, you really mean "why support two existing and well
>> known things when I can implement a third thing", right?
>
> Yes, that is exactly what I mean.

Okay, on that basis and without any better rationale than "12factor
says I can do anything" I'd be tempted to request further analysis on
the problem if the decision was on my hands. There are more
interesting problems to solve than redoing what already exists.


gustavo @ http://niemeyer.net

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Nate Finch]

I didn't bring up 12 factor, it's irrelevant to my argument.

I'm trying to make our product simpler and easier to maintain.  That is
all.  If there's another cross-platform solution that we can use, I'd be
happy to consider it.  We have to change the code to support Windows.  I'd
rather the diff be +50 -150  than +75 -0.  I don't know how to state it any
simpler than that.


On Thu, Aug 14, 2014 at 1:35 PM, Gustavo Niemeyer <
gustavo.nieme...@canonical.com> wrote:

> On Thu, Aug 14, 2014 at 1:35 PM, Nate Finch 
> wrote:
> > On Thu, Aug 14, 2014 at 12:24 PM, Gustavo Niemeyer
> >  wrote:
> >>
> >> > Why support two things when you can support just one?
> >>
> >> Just to be clear, you really mean "why support two existing and well
> >> known things when I can implement a third thing", right?
> >
> > Yes, that is exactly what I mean.
>
> Okay, on that basis and without any better rationale than "12factor
> says I can do anything" I'd be tempted to request further analysis on
> the problem if the decision was on my hands. There are more
> interesting problems to solve than redoing what already exists.
>
>
> gustavo @ http://niemeyer.net
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Gustavo Niemeyer]

On Thu, Aug 14, 2014 at 3:14 PM, Nate Finch  wrote:
> I didn't bring up 12 factor, it's irrelevant to my argument.

Is there someone else sending messages under your name?

On Thu, Aug 14, 2014 at 12:23 PM, Nate Finch  wrote:
> The front page of 12factor.net says "offering maximum portability between
> execution environments" ... that's exactly what I'm going for.

> I'm trying to make our product simpler and easier to maintain.  That is all.
> If there's another cross-platform solution that we can use, I'd be happy to
> consider it.  We have to change the code to support Windows.  I'd rather the
> diff be +50 -150  than +75 -0.  I don't know how to state it any simpler
> than that.

How about simply allowing people to select their own rsyslog target?


gustavo @ http://niemeyer.net

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Kapil Thangavelu]

On Thu, Aug 14, 2014 at 2:14 PM, Nate Finch 
wrote:

> I didn't bring up 12 factor, it's irrelevant to my argument.
>
> I'm trying to make our product simpler and easier to maintain.  That is
> all.  If there's another cross-platform solution that we can use, I'd be
> happy to consider it.  We have to change the code to support Windows.  I'd
> rather the diff be +50 -150  than +75 -0.  I don't know how to state it
> any simpler than that.
>
>
the abrogation of responsibility which is what ic you adocating for in this
thread,  also makes our product quite a lot less usable imo... Our product
is a distributed system with emergent behavior. Having a debug log is one
of the most useful things you can have to observe the system and back in py
days was one of the most used features and it was just a simple dump to the
db with querying. Its unfortunate that ability to use it usefully didn't
land to core till recently and did so in broken fashion (still requiring
internal tag names for filtering).. or lots more people would be using it.
Gustavo's suggestion of storing the structured log data  in mongo sounds
really good to me. Yes, features are work and require code but that sort of
implementation is also cross platform portable. The current implementation
and proposed alternatives I find somewhat ridicolous in that we basically
dump structured data into an unstructured format only to reparse it every
time we look at it (or ingest into logstash) given that we already have the
structured data. Asking people to setup one of those distributed log
aggregation systems systems and configure them is a huge task, and anyone
suggesting punting that to an end user or charm developer has never setup
one up themselves i suspect. ie. an analogy imo
http://xahlee.info/comp/i/fault-tolerance_NoSQL.png As for the operations
follks who do have them.. we can continue sending messages send to local
syslog and let them collect per their preference.

-k




>
> On Thu, Aug 14, 2014 at 1:35 PM, Gustavo Niemeyer <
> gustavo.nieme...@canonical.com> wrote:
>
>> On Thu, Aug 14, 2014 at 1:35 PM, Nate Finch 
>> wrote:
>> > On Thu, Aug 14, 2014 at 12:24 PM, Gustavo Niemeyer
>> >  wrote:
>> >>
>> >> > Why support two things when you can support just one?
>> >>
>> >> Just to be clear, you really mean "why support two existing and well
>> >> known things when I can implement a third thing", right?
>> >
>> > Yes, that is exactly what I mean.
>>
>> Okay, on that basis and without any better rationale than "12factor
>> says I can do anything" I'd be tempted to request further analysis on
>> the problem if the decision was on my hands. There are more
>> interesting problems to solve than redoing what already exists.
>>
>>
>> gustavo @ http://niemeyer.net
>>
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Kapil Thangavelu]

if we stopped redirecting the log on linux in our upstart jobs, log start
would rotate the per agent/job log files for us.



On Thu, Aug 14, 2014 at 11:49 AM, Matt Rae  wrote:

>
> On Thu, Aug 14, 2014 at 8:23 AM, Nate Finch 
> wrote:
>
>> The front page of 12factor.net says "offering maximum portability
>> between execution environments" ... that's exactly what I'm going for.
>>
>> We're going to support Windows. Windows does not have rsyslog or
>> logrotate.  We *have* a solution which is cross platform.
>>
>> My main concern is that if we use rsyslog on linux and something else on
>> Windows, now we're supporting and maintaining two things, when they could
>> both be the same.  Why support two things when you can support just one?
>>
>>
> I'm sure writing to stdout and redirecting to a logfile that gets watched
>> by rsyslog is the linux devop standard.  But it has also proved annoyingly
>> complex for us to configure and maintain. We've wasted several developer
>> *weeks* of time getting it to work right.
>>
>> Why would we continue to maintain the rsyslog stuff when it has been such
>> trouble for us?  Just because it's the standard, doesn't mean it's the
>> right tool for this specific job.
>>
>>
> I talked with Ian about this a little this morning, and my suggestion was
>> that we give users a way to turn off our in-app log rotation and let them
>> set it up themselves external to the app (with rsyslog or whatever they
>> like) I just don't want to have to configure it or support it.
>>
>
> Yeah this sounds like a good suggestion. Maybe we can solve the problem in
> a cross platform way but still have options to log to stdout/syslog in case
> a user wants to integrate with an existing logging system themselves. A lot
> of applications have options of logging to stdout or to syslog depending on
> which you prefer.
>
> Principle of least astonishment and taco bell programming [2] are a couple
> ideas that might be good to keep in mind when deciding how we do it.
>
> [1] http://en.wikipedia.org/wiki/Principle_of_least_astonishment
> [2] http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html
>
>
>>
>>
>> -Nate
>>
>>
>>
>>
>>
>>
>> On Thu, Aug 14, 2014 at 9:04 AM, Gustavo Niemeyer <
>> gustavo.nieme...@canonical.com> wrote:
>>
>>> As a side note, MongoDB offers a "capped collection" mechanism with
>>> the semantics that you can insert rows at will, and it rolls around
>>> automatically by replacing oldest entries with the newest ones. This
>>> tends to be a very convenient way to do structured logging, both on
>>> the writing and on the reading side. Using that as a general rsyslog
>>> might be unwise, as there are a number of details to get right and the
>>> volume may get wild depending on the application, but at least for the
>>> juju implementation itself it might be quite convenient. Imagine being
>>> able to ask "please tell me the output of the last run of the
>>> db-relation-joined hook on unit db/3".
>>>
>>> On Wed, Aug 6, 2014 at 11:24 AM, John Meinel 
>>> wrote:
>>> > all-machines.log is where we aggregate the messages from all
>>> > machines/units/etc.
>>> > It is likely to get big, which is why we want log rotate, but if you
>>> want to
>>> > be able to "juju debug-log" and actually get the feed of everything
>>> that is
>>> > going on, that needs to be *somewhere*. And yes, we'd like to switch to
>>> > something like log stash instead, but until we get there we do still
>>> need
>>> > it.
>>> > John
>>> > =:->
>>> >
>>> >
>>> >
>>> > On Wed, Aug 6, 2014 at 4:13 PM, Tim Penhey 
>>> wrote:
>>> >>
>>> >> On 06/08/14 16:11, Nate Finch wrote:
>>> >> > all-machines.log seems both redundant and a ticking time bomb of
>>> disk
>>> >> > space usage.  Do we really need it?  Can we drop it and maybe later
>>> >> > schedule some time to use something like logstash that is both more
>>> >> > featureful and is cross platform compatible (unlike rsyslog)?
>>> >>
>>> >> not yet...
>>> >>
>>> >> debug-log uses all-machines.log, we cannot get rid of it at this
>>> stage.
>>> >>
>>> >> We can't drop it until a replacement is in place.
>>> >>
>>> >> Tim
>>> >>
>>> >>
>>> >> --
>>> >> Juju-dev mailing list
>>> >> Juju-dev@lists.ubuntu.com
>>> >> Modify settings or unsubscribe at:
>>> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>> >
>>> >
>>> >
>>> > --
>>> > Juju-dev mailing list
>>> > Juju-dev@lists.ubuntu.com
>>> > Modify settings or unsubscribe at:
>>> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>> >
>>>
>>> --
>>> gustavo @ http://niemeyer.net
>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> http

Re: getting rid of all-machines.log

[Gabriel Samfira]

I think this thread has become a bit lengthy, and we have started to loose 
perspective on what we are actually trying to accomplish. Gustavo's idea to 
save the logs to is awesome and it works across platforms, allows immense 
flexibility and would give people a powerful tool. We should deffinately aspire 
to get that done sooner rather then later. However, at this point in time its 
only an idea, without a clear blueprint.

What Nate is proposing *already exists*, its tangible, proposed as a PR and 
improves the way juju handles logs. The only thing I see missing, that might 
ease people's minds is a --log-file option (that works with --debug) to 
actually enforce the usage of a log file. If we omit that option, then juju 
should just log to stdout/stderr. So we get to keep what we have, but also 
solve a huge PITA on Windows or any other platform that have limitations in 
this respect, with a minimal change...

Please keep in mind that its better to move forward no matter how small the 
steps, then to just stand still while we figure out the perfect logging system. 
I would much rather have windows support today, then 2 months from now when 
someone actually gets around to implement a *new* logging system.

This should not be a discussion about which logging system is _best_. This 
should be a discussion about which logging system is _better_ and available 
*now*. Otherwise we risk of getting caught up in details and loose sight of our 
actual goal.

Just my 2 cents.

Regards,
Gabriel

On 14.08.2014 23:47, Kapil Thangavelu wrote:



On Thu, Aug 14, 2014 at 2:14 PM, Nate Finch 
mailto:nate.fi...@canonical.com>> wrote:
I didn't bring up 12 factor, it's irrelevant to my argument.

I'm trying to make our product simpler and easier to maintain.  That is all.  
If there's another cross-platform solution that we can use, I'd be happy to 
consider it.  We have to change the code to support Windows.  I'd rather the 
diff be +50 -150  than +75 -0.  I don't know how to state it any simpler than 
that.


the abrogation of responsibility which is what ic you adocating for in this 
thread,  also makes our product quite a lot less usable imo... Our product is a 
distributed system with emergent behavior. Having a debug log is one of the 
most useful things you can have to observe the system and back in py days was 
one of the most used features and it was just a simple dump to the db with 
querying. Its unfortunate that ability to use it usefully didn't land to core 
till recently and did so in broken fashion (still requiring internal tag names 
for filtering).. or lots more people would be using it. Gustavo's suggestion of 
storing the structured log data  in mongo sounds really good to me. Yes, 
features are work and require code but that sort of implementation is also 
cross platform portable. The current implementation and proposed alternatives I 
find somewhat ridicolous in that we basically dump structured data into an 
unstructured format only to reparse it every time we look at it (or ingest into 
logstash) given that we already have the structured data. Asking people to 
setup one of those distributed log aggregation systems systems and configure 
them is a huge task, and anyone suggesting punting that to an end user or charm 
developer has never setup one up themselves i suspect. ie. an analogy imo 
http://xahlee.info/comp/i/fault-tolerance_NoSQL.png As for the operations 
follks who do have them.. we can continue sending messages send to local syslog 
and let them collect per their preference.

-k




On Thu, Aug 14, 2014 at 1:35 PM, Gustavo Niemeyer 
mailto:gustavo.nieme...@canonical.com>> wrote:
On Thu, Aug 14, 2014 at 1:35 PM, Nate Finch 
mailto:nate.fi...@canonical.com>> wrote:
> On Thu, Aug 14, 2014 at 12:24 PM, Gustavo Niemeyer
> mailto:gustavo.nieme...@canonical.com>> wrote:
>>
>> > Why support two things when you can support just one?
>>
>> Just to be clear, you really mean "why support two existing and well
>> known things when I can implement a third thing", right?
>
> Yes, that is exactly what I mean.

Okay, on that basis and without any better rationale than "12factor
says I can do anything" I'd be tempted to request further analysis on
the problem if the decision was on my hands. There are more
interesting problems to solve than redoing what already exists.


gustavo @ http://niemeyer.net


--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev





-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Kapil Thangavelu]

On Fri, Aug 15, 2014 at 7:36 AM, Gabriel Samfira <
gsamf...@cloudbasesolutions.com> wrote:

>  I think this thread has become a bit lengthy, and we have started to
> loose perspective on what we are actually trying to accomplish.
>

agreed. afaics that's how do we support logging on windows.


> Gustavo's idea to save the logs to is awesome and it works across
> platforms, allows immense flexibility and would give people a powerful
> tool. We should deffinately aspire to get that done sooner rather then
> later. However, at this point in time its only an idea, without a clear
> blueprint.
>

agreed.

>
> What Nate is proposing *already exists*, its tangible, proposed as a PR
> and improves the way juju handles logs.
>

you'll have to be more specific, there's been a shotgun of statements in
this thread, touching on logstash, aggregation removal, rsyslog removal,
log rotation, deferring to stderr/stdout, 12factor apps, working with ha
state servers, etc.



> The only thing I see missing, that might ease people's minds is a
> --log-file option (that works with --debug) to actually enforce the usage
> of a log file. If we omit that option, then juju should just log to
> stdout/stderr. So we get to keep what we have, but also solve a huge PITA
> on Windows or any other platform that have limitations in this respect,
> with a minimal change...
>

afaics your referencing your branch (
https://github.com/gabriel-samfira/syslog) which will directly send logs to
a remote aggregating syslog server on windows nodes. with regard to the
default service behavior on windows, where does stdout/stderr go for a
service? isn't the expected behavior for a windows service to use the event
log facility? ie. to be inline with expected windows behavior and extant
juju semantics, shouldn't we have multiple handlers on the log facility,
one to rsyslog, and one to event log on windows.

>
> Please keep in mind that its better to move forward no matter how small
> the steps, then to just stand still while we figure out the perfect logging
> system. I would much rather have windows support today, then 2 months from
> now when someone actually gets around to implement a *new* logging system.
>
> This should not be a discussion about which logging system is _best_. This
> should be a discussion about which logging system is _better_ and available
> *now*. Otherwise we risk of getting caught up in details and loose sight of
> our actual goal.
>
> Just my 2 cents.
>
> Regards,
> Gabriel
>
>
Thanks,

Kapil



>
> On 14.08.2014 23:47, Kapil Thangavelu wrote:
>
>
>
>
> On Thu, Aug 14, 2014 at 2:14 PM, Nate Finch 
> wrote:
>
>> I didn't bring up 12 factor, it's irrelevant to my argument.
>>
>>  I'm trying to make our product simpler and easier to maintain.  That is
>> all.  If there's another cross-platform solution that we can use, I'd be
>> happy to consider it.  We have to change the code to support Windows.  I'd
>> rather the diff be +50 -150  than +75 -0.  I don't know how to state it
>> any simpler than that.
>>
>>
>  the abrogation of responsibility which is what ic you adocating for in
> this thread,  also makes our product quite a lot less usable imo... Our
> product is a distributed system with emergent behavior. Having a debug log
> is one of the most useful things you can have to observe the system and
> back in py days was one of the most used features and it was just a simple
> dump to the db with querying. Its unfortunate that ability to use it
> usefully didn't land to core till recently and did so in broken fashion
> (still requiring internal tag names for filtering).. or lots more people
> would be using it. Gustavo's suggestion of storing the structured log data
> in mongo sounds really good to me. Yes, features are work and require code
> but that sort of implementation is also cross platform portable. The
> current implementation and proposed alternatives I find somewhat ridicolous
> in that we basically dump structured data into an unstructured format only
> to reparse it every time we look at it (or ingest into logstash) given that
> we already have the structured data. Asking people to setup one of those
> distributed log aggregation systems systems and configure them is a huge
> task, and anyone suggesting punting that to an end user or charm developer
> has never setup one up themselves i suspect. ie. an analogy imo
> http://xahlee.info/comp/i/fault-tolerance_NoSQL.png As for the operations
> follks who do have them.. we can continue sending messages send to local
> syslog and let them collect per their preference.
>
>  -k
>
>
>
>
>>
>> On Thu, Aug 14, 2014 at 1:35 PM, Gustavo Niemeyer <
>> gustavo.nieme...@canonical.com> wrote:
>>
>>>  On Thu, Aug 14, 2014 at 1:35 PM, Nate Finch 
>>> wrote:
>>> > On Thu, Aug 14, 2014 at 12:24 PM, Gustavo Niemeyer
>>> >  wrote:
>>> >>
>>> >> > Why support two things when you can support just one?
>>> >>
>>> >> Just to be clear, you really mean "why support two existing and well
>>> 

Re: getting rid of all-machines.log

[Gabriel Samfira]

On 15.08.2014 16:35, Kapil Thangavelu wrote:



On Fri, Aug 15, 2014 at 7:36 AM, Gabriel Samfira 
mailto:gsamf...@cloudbasesolutions.com>> wrote:
I think this thread has become a bit lengthy, and we have started to loose 
perspective on what we are actually trying to accomplish.

agreed. afaics that's how do we support logging on windows.

Gustavo's idea to save the logs to is awesome and it works across platforms, 
allows immense flexibility and would give people a powerful tool. We should 
deffinately aspire to get that done sooner rather then later. However, at this 
point in time its only an idea, without a clear blueprint.

agreed.

What Nate is proposing *already exists*, its tangible, proposed as a PR and 
improves the way juju handles logs.

you'll have to be more specific, there's been a shotgun of statements in this 
thread, touching on logstash, aggregation removal, rsyslog removal, log 
rotation, deferring to stderr/stdout, 12factor apps, working with ha state 
servers, etc.

I was referring to Nate's lumberjack package (PR seams to be gone) and the 
syslog streaming change. Lumberjack works on windows as well. While the rest 
have been mentioned, they are just ideas that may or may not be implemented in 
the future. I don't think Nate started this thread with the intent of 
prioritising this *now*. From what I can tell it was just a thought that was 
meant to start a discussion to implement something better in the *future*.

So far Gustavo's idea seams like the best choice with the most benefits in the 
long term. I would take that from this whole thread, and mark the case of log 
aggregation *closed* (of course pending a blueprint).




The only thing I see missing, that might ease people's minds is a --log-file 
option (that works with --debug) to actually enforce the usage of a log file. 
If we omit that option, then juju should just log to stdout/stderr. So we get 
to keep what we have, but also solve a huge PITA on Windows or any other 
platform that have limitations in this respect, with a minimal change...

afaics your referencing your branch (https://github.com/gabriel-samfira/syslog) 
which will directly send logs to a remote aggregating syslog server on windows 
nodes. with regard to the default service behavior on windows, where does 
stdout/stderr go for a service? isn't the expected behavior for a windows 
service to use the event log facility? ie. to be inline with expected windows 
behavior and extant juju semantics, shouldn't we have multiple handlers on the 
log facility, one to rsyslog, and one to event log on windows.

While the initial scope of this thread was to see if we really do need to save 
an all-machines.log via syslog (and apparently the CI needs it) instead of 
using a better tool for this particular job (like logstash...not not 
necessarily that), it has gone off into a bunch of tangents, one of which is 
using lumberjack to write the logs and rotate them instead of having upstart 
write them via redirect, and then logrotate rotate and archive them(2 external 
and platform specific dependencies as opposed to one cross platform 
implementation).

Windows services have no way of capturing stdout/stderr in the same way upstart 
does (solved by using Nate's lumberjack). It is simply lost, or the service 
will fail because it cannot write to stdout/stderr (there is no console 
present).

The syslog change you mentioned is just half of the solution, and will allow 
the windows version to log remotely and be in-line with the ubuntu version 
without needing to install 3rd party applications. That was my initial goal. 
The fact that it removes rsyslog as a dependency from non-bootstrap nodes on 
Ubuntu as well, is in my opinion a bonus :). You get to remove an external 
dependency which you no longer have to configure separately, and we finally do 
ssl certificate checking of the rsyslog server on the state machine (right now 
the way the forwarders are configured, will ignore the SSL errors).

As for logging to event log, that is certainly a good option, and I will look 
into it.  If it solves the local logging issue, I am happy with it and will 
soon propose a PR. If there is a package to do this that implements an 
io.Writer, it should be trivial.

So to summarize:

* I will be looking into event log for local windows logging. This will 
probably require writing a package.
* the syslog change will solve in the sort term, the aggregation issue from 
Windows nodes (when something better comes along, I will personally send a case 
of beer or ice-cream...or both, to whomever removes syslog as a dependency)
* lumberjack works *now* for local logging on both Windows and Ubuntu. It 
simply removes 2 dependencies (for logging) with just a few lines of code...

As I have mentioned, I am willing to adapt to any decision you guys make (I am 
just an outsider), as long as that decision does not imply 2 more months and 3 
extra packages just to get a perfect logging system. I belie

Re: getting rid of all-machines.log

[Horacio Duran]

Hey, In an effort to move forward with juju's windows integration I have
summarized what seems to be the core points of this discussion to the best
of my ability (please excuse me if I missed or misunderstood something).
The two core points of discussion on this thread are:
* should we remove all-machines.log: which has been voted against, at least
for the moment, since it is used for debug-log.
* how do we support logging in windows: The strongest suggestions here are
a syslog package by gabriel and logging into MongoDB by Gustavo.

We do require some decision on the front of windows logging to have a
complete windows support. Ideally we need senior citizens of juju dev
community to weight into this in order to get a clear path to follow.

Here is a summary I made to help myself while following this discussion:

​Nate original suggestion:
* Remove all-machines.log: Claiming it takes a lot of space and it is not a
multi platform solution

Tim, John, Aaaron, etc:
* all-machines.log is required for debug-log
* makes it big and it would be nice to rotate it.

Nate, gabriel:
* keep all-machines.log
* use a go-only solution (syslog package with ports from gabriel for
windows)
John
* agrees.

Nate, gabriel:
* remove rsyslog from al OSes in favor of one solution that fits all OSes
* Replace with go only solution.

Dave:
* Dont mind about the logs, make it just output and let external tools
handle logging and rotation.
* all-machines.log might be a bit bloated and it could contain less data
that is more useful.
(Here is the reference to 12factor that will later be attibuted to nate)
Ian:
* Agrees with dave, yet we should provide a rolling mechanism.

Gabriel:
* Windows does not support capturing stdout as a logging mechanism, it
requires to explicitly log into the event log.
* Thinks that using rsyslog to stream logs from agents to state server is
too much overhead on external tools.
* Proposes replacing external rsyslog with in app solution for the case of
streaming logs.
* Alternative solution, he does not recommend it, to create (and bundle
with jujud.exe) a wrapper for windows only.

Gustavo:
* Present a possible alternative by using a MongoDB "capped collection"
which will suit our use cases but does not recommend it because of the idea
needs maturing on some details.

Matt:
* We should provide the option to log to stdout or syslog.

Kapil:
* Supports Gustavo's idea of logging in a structured form into Mongo as it
makes sense to dump structured data with structure instead of serializing
it to be de-serialized later.
* We can send also messages to syslog and let OPS people collec them
themselves.

Gabriel (summarizing)
* I will be looking into event log for local windows logging. This will
probably require writing a package.
* the syslog change will solve in the sort term, the aggregation issue from
Windows nodes (when something better comes along, I will personally send a
case of beer or ice-cream...or both, to whomever removes syslog as a
dependency)
* lumberjack works *now* for local logging on both Windows and Ubuntu. It
simply removes 2 dependencies (for logging) with just a few lines of code...

--
Horacio
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[David Cheney]

Hi Horatio,

I don't see a way to resolve the very disparate set of opinions you've
highlighted below. It's also not clear from your email who is
responsible for making a decision.

I suggest reframing the discussion as user stories. ie

* As a Juju user with a Windows workstation I want to use juju
debug-log (this should work today)
* As a Juju user who has deployed a mixed environment (to the best of
my knowledge there is no requirement for the state servers to run on
Windows, this seems contrary to Canonical's goal of Free Software)
containing a windows workload charm I want to view the logs from that
charm.

Dave

On Wed, Aug 27, 2014 at 5:35 AM, Horacio Duran
 wrote:
> Hey, In an effort to move forward with juju's windows integration I have
> summarized what seems to be the core points of this discussion to the best
> of my ability (please excuse me if I missed or misunderstood something).
> The two core points of discussion on this thread are:
> * should we remove all-machines.log: which has been voted against, at least
> for the moment, since it is used for debug-log.
> * how do we support logging in windows: The strongest suggestions here are a
> syslog package by gabriel and logging into MongoDB by Gustavo.
>
> We do require some decision on the front of windows logging to have a
> complete windows support. Ideally we need senior citizens of juju dev
> community to weight into this in order to get a clear path to follow.
>
> Here is a summary I made to help myself while following this discussion:
>
> Nate original suggestion:
> * Remove all-machines.log: Claiming it takes a lot of space and it is not a
> multi platform solution
>
> Tim, John, Aaaron, etc:
> * all-machines.log is required for debug-log
> * makes it big and it would be nice to rotate it.
>
> Nate, gabriel:
> * keep all-machines.log
> * use a go-only solution (syslog package with ports from gabriel for
> windows)
> John
> * agrees.
>
> Nate, gabriel:
> * remove rsyslog from al OSes in favor of one solution that fits all OSes
> * Replace with go only solution.
>
> Dave:
> * Dont mind about the logs, make it just output and let external tools
> handle logging and rotation.
> * all-machines.log might be a bit bloated and it could contain less data
> that is more useful.
> (Here is the reference to 12factor that will later be attibuted to nate)
> Ian:
> * Agrees with dave, yet we should provide a rolling mechanism.
>
> Gabriel:
> * Windows does not support capturing stdout as a logging mechanism, it
> requires to explicitly log into the event log.
> * Thinks that using rsyslog to stream logs from agents to state server is
> too much overhead on external tools.
> * Proposes replacing external rsyslog with in app solution for the case of
> streaming logs.
> * Alternative solution, he does not recommend it, to create (and bundle with
> jujud.exe) a wrapper for windows only.
>
> Gustavo:
> * Present a possible alternative by using a MongoDB "capped collection"
> which will suit our use cases but does not recommend it because of the idea
> needs maturing on some details.
>
> Matt:
> * We should provide the option to log to stdout or syslog.
>
> Kapil:
> * Supports Gustavo's idea of logging in a structured form into Mongo as it
> makes sense to dump structured data with structure instead of serializing it
> to be de-serialized later.
> * We can send also messages to syslog and let OPS people collec them
> themselves.
>
> Gabriel (summarizing)
> * I will be looking into event log for local windows logging. This will
> probably require writing a package.
> * the syslog change will solve in the sort term, the aggregation issue from
> Windows nodes (when something better comes along, I will personally send a
> case of beer or ice-cream...or both, to whomever removes syslog as a
> dependency)
> * lumberjack works *now* for local logging on both Windows and Ubuntu. It
> simply removes 2 dependencies (for logging) with just a few lines of code...
>
> --
> Horacio
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Mark Ramm-Christensen (Canonical.com)]

>> you'll have to be more specific, there's been a shotgun of statements in
>> this thread, touching on logstash, aggregation removal, rsyslog removal,
>> log rotation, deferring to stderr/stdout, 12factor apps, working with ha
>> state servers, etc.
>
> I was referring to Nate's lumberjack package (PR seams to be gone) and
> the syslog streaming change. Lumberjack works on windows as well.


Just so it is easily accessable from this thread the URL for the
aformentioned lumberjack is here:

https://godoc.org/gopkg.in/natefinch/lumberjack.v2

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Mark Ramm-Christensen (Canonical.com)]

Thanks Horacio, it's very helpful to get this all in one place.
Adding my thoughts in line.

On Tue, Aug 26, 2014 at 3:35 PM, Horacio Duran
 wrote:

> Here is a summary I made to help myself while following this discussion:
>
> Nate original suggestion:
> * Remove all-machines.log: Claiming it takes a lot of space and it is not a
> multi platform solution
>
> Tim, John, Aaaron, etc:
> * all-machines.log is required for debug-log
> * makes it big and it would be nice to rotate it.

We do not have the option of removing debug-log, and therefore there
must be a "central" repository of log data.   This need not live in a
text file on the state servers called all-machines.log but it must
exist.

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[John Meinel]

My personal vote would be:
a) Use something that can write directly to multiple syslog receivers over
a TLS encrypted connection from inside the jujud binary (e.g. don't use
rsyslog to read the log files and forward them on to the state servers, but
just write directly)
I'd actually like to continue keeping a log file on local disk, as it can
help diagnose when the problem is that you are failing to connect to
somewhere else.
b) In the short term, continue using rsyslog as the aggregator on state
server machines.

c) Long term (not this cycle), look towards logging into an actual database
(be it another collection in mongo, or something more specifically designed
for handling logs like logstash, fluentd, scribe, flume, etc).
Ideally we'd provide something that people could easily integrate into
whatever system they wanted (and for that, all-machines.log is probably the
simplest thing to then tail and pull into another system.)

John
=:->



On Wed, Aug 27, 2014 at 7:19 AM, Mark Ramm-Christensen (Canonical.com) <
mark.ramm-christen...@canonical.com> wrote:

> Thanks Horacio, it's very helpful to get this all in one place.
> Adding my thoughts in line.
>
> On Tue, Aug 26, 2014 at 3:35 PM, Horacio Duran
>  wrote:
>
> > Here is a summary I made to help myself while following this discussion:
> >
> > Nate original suggestion:
> > * Remove all-machines.log: Claiming it takes a lot of space and it is
> not a
> > multi platform solution
> >
> > Tim, John, Aaaron, etc:
> > * all-machines.log is required for debug-log
> > * makes it big and it would be nice to rotate it.
>
> We do not have the option of removing debug-log, and therefore there
> must be a "central" repository of log data.   This need not live in a
> text file on the state servers called all-machines.log but it must
> exist.
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Ian Booth]

On 27/08/14 14:56, John Meinel wrote:
> My personal vote would be:
> a) Use something that can write directly to multiple syslog receivers over
> a TLS encrypted connection from inside the jujud binary (e.g. don't use
> rsyslog to read the log files and forward them on to the state servers, but
> just write directly)

I may be misremembering, but at the time that was the preferred approach. But
then someone said Go's inbuilt syslog APIs were broke, so the compromise was to
use rsyslog forwarding.

Does anyone else recall why it may have been said that Go's syslog APIs are 
broken?

> I'd actually like to continue keeping a log file on local disk, as it can
> help diagnose when the problem is that you are failing to connect to
> somewhere else.

Big +1 IMHO. I've always been a fan of logging to a text file readable in vi,
emacs, whatever. Lowest common denominator is often the only option for
debugging. I have no issue with logging elsewhere also, but can we please keep
the log file as well.


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[David Cheney]

On Wed, Aug 27, 2014 at 3:04 PM, Ian Booth  wrote:
>
>
> On 27/08/14 14:56, John Meinel wrote:
>> My personal vote would be:
>> a) Use something that can write directly to multiple syslog receivers over
>> a TLS encrypted connection from inside the jujud binary (e.g. don't use
>> rsyslog to read the log files and forward them on to the state servers, but
>> just write directly)
>
> I may be misremembering, but at the time that was the preferred approach. But
> then someone said Go's inbuilt syslog APIs were broke, so the compromise was 
> to
> use rsyslog forwarding.
>
> Does anyone else recall why it may have been said that Go's syslog APIs are 
> broken?

The reconnect logic is broken in all the version's of the syslog api.
The general consensus is that package is a mistake and should not be
used.

>
>> I'd actually like to continue keeping a log file on local disk, as it can
>> help diagnose when the problem is that you are failing to connect to
>> somewhere else.
>
> Big +1 IMHO. I've always been a fan of logging to a text file readable in vi,
> emacs, whatever. Lowest common denominator is often the only option for
> debugging. I have no issue with logging elsewhere also, but can we please keep
> the log file as well.
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[John Meinel]

...

> > I may be misremembering, but at the time that was the preferred
> approach. But
> > then someone said Go's inbuilt syslog APIs were broke, so the compromise
> was to
> > use rsyslog forwarding.
> >
> > Does anyone else recall why it may have been said that Go's syslog APIs
> are broken?
>
> The reconnect logic is broken in all the version's of the syslog api.
> The general consensus is that package is a mistake and should not be
> used.
>
>
I believe there is also an issue where we couldn't format the logs the way
we wanted to. (The prefix/timestamp are added by the package and cannot be
configured).

John
=:->
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Gabriel Samfira]

Hi David,

(some comments in-line)

As a user that wants to deploy a charm on a Windows machine, I want to 
be able to have a local log file on that machine for the machine agent 
and for the units deployed to it. I also want to be able to aggregate 
all those logs the same way Ubuntu workloads do (at the moment, to the 
syslog on the state machine(s) ).

Right now, the debug-log that juju generates works the same way on both 
platforms. However, windows services cannot redirect stdout to a file 
the same way we do it using upstart. So when starting juju as a service, 
that log does not get written.

So I guess the issues right now are:

* how can we have a local logfile on both Ubuntu as well as Windows? 
(lumberjack is one option that works on both platforms)
* how can we aggregate logs from both platforms? (using a go package to 
write directly to syslog is one option that works cross platform)

On 27.08.2014 04:32, David Cheney wrote:
> Hi Horatio,
>
> I don't see a way to resolve the very disparate set of opinions you've
> highlighted below. It's also not clear from your email who is
> responsible for making a decision.
>
> I suggest reframing the discussion as user stories. ie
>
> * As a Juju user with a Windows workstation I want to use juju
> debug-log (this should work today)
> * As a Juju user who has deployed a mixed environment (to the best of
> my knowledge there is no requirement for the state servers to run on
> Windows, this seems contrary to Canonical's goal of Free Software)
The goal is to eventually have a 1-to-1 feature set on both platforms. 
This includes a working state machine on windows as well (to be honest, 
it probably works already, just need to use WinRM instead of ssh to 
bootstrap it), containers using Hyper-V, and anything else that is 
supported by Ubuntu and feasible in Windows.

Thanks,
Gabriel

> containing a windows workload charm I want to view the logs from that
> charm.
>
> Dave
>
> On Wed, Aug 27, 2014 at 5:35 AM, Horacio Duran
>  wrote:
>> Hey, In an effort to move forward with juju's windows integration I have
>> summarized what seems to be the core points of this discussion to the best
>> of my ability (please excuse me if I missed or misunderstood something).
>> The two core points of discussion on this thread are:
>> * should we remove all-machines.log: which has been voted against, at least
>> for the moment, since it is used for debug-log.
>> * how do we support logging in windows: The strongest suggestions here are a
>> syslog package by gabriel and logging into MongoDB by Gustavo.
>>
>> We do require some decision on the front of windows logging to have a
>> complete windows support. Ideally we need senior citizens of juju dev
>> community to weight into this in order to get a clear path to follow.
>>
>> Here is a summary I made to help myself while following this discussion:
>>
>> Nate original suggestion:
>> * Remove all-machines.log: Claiming it takes a lot of space and it is not a
>> multi platform solution
>>
>> Tim, John, Aaaron, etc:
>> * all-machines.log is required for debug-log
>> * makes it big and it would be nice to rotate it.
>>
>> Nate, gabriel:
>> * keep all-machines.log
>> * use a go-only solution (syslog package with ports from gabriel for
>> windows)
>> John
>> * agrees.
>>
>> Nate, gabriel:
>> * remove rsyslog from al OSes in favor of one solution that fits all OSes
>> * Replace with go only solution.
>>
>> Dave:
>> * Dont mind about the logs, make it just output and let external tools
>> handle logging and rotation.
>> * all-machines.log might be a bit bloated and it could contain less data
>> that is more useful.
>> (Here is the reference to 12factor that will later be attibuted to nate)
>> Ian:
>> * Agrees with dave, yet we should provide a rolling mechanism.
>>
>> Gabriel:
>> * Windows does not support capturing stdout as a logging mechanism, it
>> requires to explicitly log into the event log.
>> * Thinks that using rsyslog to stream logs from agents to state server is
>> too much overhead on external tools.
>> * Proposes replacing external rsyslog with in app solution for the case of
>> streaming logs.
>> * Alternative solution, he does not recommend it, to create (and bundle with
>> jujud.exe) a wrapper for windows only.
>>
>> Gustavo:
>> * Present a possible alternative by using a MongoDB "capped collection"
>> which will suit our use cases but does not recommend it because of the idea
>> needs maturing on some details.
>>
>> Matt:
>> * We should provide the option to log to stdout or syslog.
>>
>> Kapil:
>> * Supports Gustavo's idea of logging in a structured form into Mongo as it
>> makes sense to dump structured data with structure instead of serializing it
>> to be de-serialized later.
>> * We can send also messages to syslog and let OPS people collec them
>> themselves.
>>
>> Gabriel (summarizing)
>> * I will be looking into event log for local windows logging. This will
>> probably require writing a package.
>> * the syslo

Re: getting rid of all-machines.log

[Gabriel Samfira]

On 27.08.2014 08:12, John Meinel wrote:
...
> I may be misremembering, but at the time that was the preferred approach. But
> then someone said Go's inbuilt syslog APIs were broke, so the compromise was 
> to
> use rsyslog forwarding.
>
> Does anyone else recall why it may have been said that Go's syslog APIs are 
> broken?

The reconnect logic is broken in all the version's of the syslog api.
The general consensus is that package is a mistake and should not be
used.


I believe there is also an issue where we couldn't format the logs the way we 
wanted to. (The prefix/timestamp are added by the package and cannot be 
configured).


I think that may have been in an older version of Go. For example:

http://paste.ubuntu.com/8158001/

will appear in syslog as:

Aug 27 13:01:36 rossak testing[3812]: hello

An example of log output streamed to all-machines.log using the syslog package:

unit-vanilla2-0[1424]: 2014-08-27 09:34:14 INFO juju.worker.uniter 
uniter.go:324 deploying charm "local:win2012hvr2/vanilla2-0"
unit-vanilla2-0[1424]: 2014-08-27 09:34:14 DEBUG juju.worker.uniter.charm 
manifest_deployer.go:126 preparing to deploy charm 
"local:win2012hvr2/vanilla2-0"
unit-vanilla2-0[1424]: 2014-08-27 09:34:14 DEBUG juju.worker.uniter.charm 
manifest_deployer.go:102 deploying charm "local:win2012hvr2/vanilla2-0"
unit-vanilla2-0[1424]: 2014-08-27 09:34:14 DEBUG juju.worker.uniter.filter 
filter.go:583 no new charm event

I have not looked at the reconnect part though.


Gabriel



John
=:->



-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[John Meinel]

So at the very least our default logging *doesn't* include the PID, though
the rest seems sane to me.

machine-0: 2014-04-08 02:00:53 INFO juju.cmd supercommand.go:296 running
juju-1.19.0-precise-amd64 [gc]
machine-0: 2014-04-08 02:00:53 INFO juju.cmd.jujud machine.go:129 machine
agent machine-0 start (1.19.0-precise-amd64 [gc])

John
=:->


On Wed, Aug 27, 2014 at 2:13 PM, Gabriel Samfira <
gsamf...@cloudbasesolutions.com> wrote:

>  On 27.08.2014 08:12, John Meinel wrote:
>
>   ...
>
>> > I may be misremembering, but at the time that was the preferred
>> approach. But
>> > then someone said Go's inbuilt syslog APIs were broke, so the
>> compromise was to
>> > use rsyslog forwarding.
>> >
>> > Does anyone else recall why it may have been said that Go's syslog APIs
>> are broken?
>>
>>  The reconnect logic is broken in all the version's of the syslog api.
>> The general consensus is that package is a mistake and should not be
>> used.
>>
>>
>  I believe there is also an issue where we couldn't format the logs the
> way we wanted to. (The prefix/timestamp are added by the package and cannot
> be configured).
>
>
>
> I think that may have been in an older version of Go. For example:
>
> http://paste.ubuntu.com/8158001/
>
> will appear in syslog as:
>
> Aug 27 13:01:36 rossak testing[3812]: hello
>
> An example of log output streamed to all-machines.log using the syslog
> package:
>
> unit-vanilla2-0[1424]: 2014-08-27 09:34:14 INFO juju.worker.uniter
> uniter.go:324 deploying charm "local:win2012hvr2/vanilla2-0"
> unit-vanilla2-0[1424]: 2014-08-27 09:34:14 DEBUG juju.worker.uniter.charm
> manifest_deployer.go:126 preparing to deploy charm
> "local:win2012hvr2/vanilla2-0"
> unit-vanilla2-0[1424]: 2014-08-27 09:34:14 DEBUG juju.worker.uniter.charm
> manifest_deployer.go:102 deploying charm "local:win2012hvr2/vanilla2-0"
> unit-vanilla2-0[1424]: 2014-08-27 09:34:14 DEBUG juju.worker.uniter.filter
> filter.go:583 no new charm event
>
> I have not looked at the reconnect part though.
>
>
> Gabriel
>
>
>
>  John
> =:->
>
>
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[Gabriel Samfira]

That one is an easy fix in any case. We are using a forked version of the 
syslog package. Removing the pid from the writeString() method should be 
trivial.


Gabriel

On 27.08.2014 13:45, John Meinel wrote:
So at the very least our default logging *doesn't* include the PID, though the 
rest seems sane to me.

machine-0: 2014-04-08 02:00:53 INFO juju.cmd supercommand.go:296 running 
juju-1.19.0-precise-amd64 [gc]
machine-0: 2014-04-08 02:00:53 INFO juju.cmd.jujud machine.go:129 machine agent 
machine-0 start (1.19.0-precise-amd64 [gc])

John
=:->


On Wed, Aug 27, 2014 at 2:13 PM, Gabriel Samfira 
mailto:gsamf...@cloudbasesolutions.com>> wrote:
On 27.08.2014 08:12, John Meinel wrote:
...
> I may be misremembering, but at the time that was the preferred approach. But
> then someone said Go's inbuilt syslog APIs were broke, so the compromise was 
> to
> use rsyslog forwarding.
>
> Does anyone else recall why it may have been said that Go's syslog APIs are 
> broken?

The reconnect logic is broken in all the version's of the syslog api.
The general consensus is that package is a mistake and should not be
used.


I believe there is also an issue where we couldn't format the logs the way we 
wanted to. (The prefix/timestamp are added by the package and cannot be 
configured).


I think that may have been in an older version of Go. For example:

http://paste.ubuntu.com/8158001/

will appear in syslog as:

Aug 27 13:01:36 rossak testing[3812]: hello

An example of log output streamed to all-machines.log using the syslog package:

unit-vanilla2-0[1424]: 2014-08-27 09:34:14 INFO juju.worker.uniter 
uniter.go:324 deploying charm "local:win2012hvr2/vanilla2-0"
unit-vanilla2-0[1424]: 2014-08-27 09:34:14 DEBUG juju.worker.uniter.charm 
manifest_deployer.go:126 preparing to deploy charm 
"local:win2012hvr2/vanilla2-0"
unit-vanilla2-0[1424]: 2014-08-27 09:34:14 DEBUG juju.worker.uniter.charm 
manifest_deployer.go:102 deploying charm "local:win2012hvr2/vanilla2-0"
unit-vanilla2-0[1424]: 2014-08-27 09:34:14 DEBUG juju.worker.uniter.filter 
filter.go:583 no new charm event

I have not looked at the reconnect part though.


Gabriel



John
=:->




--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev



-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev