Re: getting rid of all-machines.log

2014-08-14 Thread Gabriel Samfira
Please forgive the lengthy response :). The following are just a few 
thoughts regarding this subject.

On 14.08.2014 06:13, David Cheney wrote:
> Ian asked me to post my thoughts here.
>
> I am not in favour of applications rolling their own logs, I believe
> that applications should not know anything about their log output,
> they should just dump it all to stdout and another process should take
> care of shuttling the data, logging it, culling it, whatever.
> This is summarised here, http://12factor.net/logs
This philosophy works under the assumption that all applications that 
run as services, will only do so on linux/unix systems. Services on 
Windows have no way to capture stdout unless we use a wrapper to start 
the application as a service, capture its stdout and later on, do 
something with it. Windows services mostly stream to local event log. 
Given that we want juju to run both on linux and windows, there needs to 
be a common denominator.

The ability for juju to write its own logfile *works* across most, if 
not all platforms (as it does for other apps such as nginx, apache, 
openstack components, etc). If we want to have log handling outside of 
juju, we need to implement a separate configuration on separate 
platforms (adding more moving parts). Doing this in juju removes 
external dependencies, and gives us a unified way of handling logs 
across platforms.

I can understand how this philosophy is ok for applications that are 
meant to run on a PaaS where the application needs to adapt to its 
environment, and where the PaaS prefers to not allow the developer to 
make mistakes when it comes to saving the log of the application, but we 
are not running juju on a PaaS. Juju will only be run on a IaaS or on 
bare metal, where you are adapting the environment to your needs. What 
is good for a PaaS running apps mostly on Linux, is not necessarily good 
when you are targeting bare metal or IaaS using multiple platforms. We 
have to keep in mind that not all platforms are created equal. Some have 
limitations that others do not.
>
> I think our current system with rsyslog is working fine and there is
> no reason to remove it.
There is no rsyslog for windows (well, not free at least). Also, 
installing a 3rd party service that needs to be updated separately from 
the standard windows workflow opens up the system to vulnerabilities if 
not done right.

Windows aside, on non-bootstrap nodes we currently only use rsyslog to 
tail the logs generated by juju (and written by an append redirect from 
upstart), and then stream those logs to the a state machine. So you have 
3 components that have to work together just for the end result of 
having a log streamed to a state machine. IMO there are 2 moving parts 
too many.

Most languages have a syslog package that can stream directly to syslog. 
IMHO, the rsyslog forwarder is one external dependency that should be 
easily replaced. The less moving parts, the better. This also allows 
juju to stream logs from all platforms directly to syslog (its just a 
TCP/UDP connection in the end...).

This is just my 2 cents. Its up to you guys in the end. As far as 
windows is concerned, as long as I have clear guidelines defined, I can 
adapt the PRs.

An option would be to start writing a wrapper for Windows that will be 
bundled along side jujud.exe...but it would be a platform specific 
thing. This is something I would rather not do if I have a choice.

Cheers,
Gabriel
>
> The problems with all-machines.log being to large is independent of
> log rolling or any of these other arguments. We simply log too much.
> There would be no request for log rolling it all-machines.log
> contained only useful data.
>
> Dave
>
> On Sat, Aug 9, 2014 at 2:58 AM, Nate Finch  wrote:
>> So, rsyslog rotation works fine on Linux, but we can't do that on Windows.
>> If we have to do something different for Windows, I'd rather just do one
>> thing which is cross platform compatible for all our OSes, and not have to
>> support a different configuration for each OS.  Doing it all in-application
>> also insulates us from external dependencies... if some future or past
>> Ubuntu series (or CentOS) has a different version of rsyslog, it could
>> behave differently / require a different configuration, etc.
>>
>>
>>
>> On Fri, Aug 8, 2014 at 12:40 PM, David Britton 
>> wrote:
>>> On Fri, Aug 08, 2014 at 12:03:21PM -0400, Nate Finch wrote:
>>> [...]
 remote syslog and to the local file log, we wouldn't need to worry about
 log rotation of the local log screwing up what gets sent to the remote
>>> Do the standard rsyslog log rotation mechanisms not function well?
>>>
>>> On Windows, what about the event log (which has remote
>>> viewing/aggregation capabilities built in)?
>>>
>>> --
>>> David Britton 
>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>

-- 
Juju-dev mailing list
Ju

Re: getting rid of all-machines.log

2014-08-14 Thread Gabriel Samfira
On 14.08.2014 06:20, Ian Booth wrote:
> Just to back up Dave's arguments - all sys admins I know would be a big -1 on
> Juju doing it's own log rolling. It's a recipe for lost log files, missing 
> data
> etc. It's a mixing of responsibilities that should be handled separately.
I am unsure how juju rotating its own logs can lead to loss of data. Any 
input on this would be appreciated.
>
> Just on the volume point Dave raised - we do log a lot but that's also an
> orthogonal issue. Even if we logged less we'd still need a rolling mechanism.
>
> On 14/08/14 13:13, David Cheney wrote:
>> Ian asked me to post my thoughts here.
>>
>> I am not in favour of applications rolling their own logs, I believe
>> that applications should not know anything about their log output,
>> they should just dump it all to stdout and another process should take
>> care of shuttling the data, logging it, culling it, whatever.
>>
>> This is summarised here, http://12factor.net/logs
>>
>> I think our current system with rsyslog is working fine and there is
>> no reason to remove it.
>>
>> The problems with all-machines.log being to large is independent of
>> log rolling or any of these other arguments. We simply log too much.
>> There would be no request for log rolling it all-machines.log
>> contained only useful data.
>>
>> Dave
>>
>> On Sat, Aug 9, 2014 at 2:58 AM, Nate Finch  wrote:
>>> So, rsyslog rotation works fine on Linux, but we can't do that on Windows.
>>> If we have to do something different for Windows, I'd rather just do one
>>> thing which is cross platform compatible for all our OSes, and not have to
>>> support a different configuration for each OS.  Doing it all in-application
>>> also insulates us from external dependencies... if some future or past
>>> Ubuntu series (or CentOS) has a different version of rsyslog, it could
>>> behave differently / require a different configuration, etc.
>>>
>>>
>>>
>>> On Fri, Aug 8, 2014 at 12:40 PM, David Britton 
>>> wrote:
 On Fri, Aug 08, 2014 at 12:03:21PM -0400, Nate Finch wrote:
 [...]
> remote syslog and to the local file log, we wouldn't need to worry about
> log rotation of the local log screwing up what gets sent to the remote
 Do the standard rsyslog log rotation mechanisms not function well?

 On Windows, what about the event log (which has remote
 viewing/aggregation capabilities built in)?

 --
 David Britton 
>>>
>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: getting rid of all-machines.log

2014-08-14 Thread Gustavo Niemeyer
As a side note, MongoDB offers a "capped collection" mechanism with
the semantics that you can insert rows at will, and it rolls around
automatically by replacing oldest entries with the newest ones. This
tends to be a very convenient way to do structured logging, both on
the writing and on the reading side. Using that as a general rsyslog
might be unwise, as there are a number of details to get right and the
volume may get wild depending on the application, but at least for the
juju implementation itself it might be quite convenient. Imagine being
able to ask "please tell me the output of the last run of the
db-relation-joined hook on unit db/3".

On Wed, Aug 6, 2014 at 11:24 AM, John Meinel  wrote:
> all-machines.log is where we aggregate the messages from all
> machines/units/etc.
> It is likely to get big, which is why we want log rotate, but if you want to
> be able to "juju debug-log" and actually get the feed of everything that is
> going on, that needs to be *somewhere*. And yes, we'd like to switch to
> something like log stash instead, but until we get there we do still need
> it.
> John
> =:->
>
>
>
> On Wed, Aug 6, 2014 at 4:13 PM, Tim Penhey  wrote:
>>
>> On 06/08/14 16:11, Nate Finch wrote:
>> > all-machines.log seems both redundant and a ticking time bomb of disk
>> > space usage.  Do we really need it?  Can we drop it and maybe later
>> > schedule some time to use something like logstash that is both more
>> > featureful and is cross platform compatible (unlike rsyslog)?
>>
>> not yet...
>>
>> debug-log uses all-machines.log, we cannot get rid of it at this stage.
>>
>> We can't drop it until a replacement is in place.
>>
>> Tim
>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>

-- 
gustavo @ http://niemeyer.net

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: Intentionally introducing failures into Juju

2014-08-14 Thread Gustavo Niemeyer
On Thu, Aug 14, 2014 at 3:42 AM, Stuart Bishop
 wrote:
> Further to just injecting failures, I'm interested in controlling when
> and the order hooks can run. A sort of manual mode, which could be
> driven by a test harness such as Amulet.

This sounds quite heavyweight and intrusive. Introducing delays in
certain actions so that the flow is altered sounds okay, but manually
modifying the order of hooks defined to be the correct one doesn't
sound right. The former is also more useful in stress testing, while
the latter is going to be seldom used because people need to think
through the cases that could explode, and then hand-code the failure
scenario.

> Perhaps all hooks in the
> queue are initially held, and I can unhold them one at a time.

You should be able to do that one with debug-hooks today already, right?

> This would let me test the odd edge cases, such as peers departing
> peer relations during handshaking, or what happens when a new client

That's not just reordering, but introducing a failure scenario where a
peer unit leaves the relation. It would indeed be useful to support
that sort of failure injection, but with a proper mechanism for it
rather than fiddling with hook ordering.

> unit is added and its relation-changed hooks manages to run before the
> relation-joined hooks at the server end.

Similar case.

> If you could do this, you could inject your failures by actually
> breaking your units using juju run or juju ssh. Deploy your units, run
> the install hooks, juju ssh in breaking one of the units (rm -rf /,
> whatever), run the peer relation hooks, confirm that the service is
> still usable despite the failed unit.

Similar case as well. We should be able to inject those failures
without having to manually fiddle with hook ordering.


gustavo @ http://niemeyer.net

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Reviewboard

2014-08-14 Thread Nate Finch
What's the status on this?  I think this could really help our workflow.
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: Reviewboard

2014-08-14 Thread Nate Finch
Sorry, more context for those who haven't been in on the talks in Juju-core:

We're trying to get Reviewboard set up for juju-core use with github (and a
plugin we wrote so we can log in with our github usernames).


On Thu, Aug 14, 2014 at 10:27 AM, Nate Finch 
wrote:

> What's the status on this?  I think this could really help our workflow.
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: getting rid of all-machines.log

2014-08-14 Thread Matt Rae
Many operations teams already have a standard log collecting systems. I
think it would be best to be flexible enough to work in environments with
existing systems.

Standard ways are logging to syslog so any syslog implementation can be
used, or logging to stdout so a supervisor like djb daemontools can collect
and rotate logs.


On Thu, Aug 14, 2014 at 3:00 AM, Gabriel Samfira <
gsamf...@cloudbasesolutions.com> wrote:

> On 14.08.2014 06:20, Ian Booth wrote:
> > Just to back up Dave's arguments - all sys admins I know would be a big
> -1 on
> > Juju doing it's own log rolling. It's a recipe for lost log files,
> missing data
> > etc. It's a mixing of responsibilities that should be handled separately.
> I am unsure how juju rotating its own logs can lead to loss of data. Any
> input on this would be appreciated.
> >
> > Just on the volume point Dave raised - we do log a lot but that's also an
> > orthogonal issue. Even if we logged less we'd still need a rolling
> mechanism.
> >
> > On 14/08/14 13:13, David Cheney wrote:
> >> Ian asked me to post my thoughts here.
> >>
> >> I am not in favour of applications rolling their own logs, I believe
> >> that applications should not know anything about their log output,
> >> they should just dump it all to stdout and another process should take
> >> care of shuttling the data, logging it, culling it, whatever.
> >>
> >> This is summarised here, http://12factor.net/logs
> >>
> >> I think our current system with rsyslog is working fine and there is
> >> no reason to remove it.
> >>
> >> The problems with all-machines.log being to large is independent of
> >> log rolling or any of these other arguments. We simply log too much.
> >> There would be no request for log rolling it all-machines.log
> >> contained only useful data.
> >>
> >> Dave
> >>
> >> On Sat, Aug 9, 2014 at 2:58 AM, Nate Finch 
> wrote:
> >>> So, rsyslog rotation works fine on Linux, but we can't do that on
> Windows.
> >>> If we have to do something different for Windows, I'd rather just do
> one
> >>> thing which is cross platform compatible for all our OSes, and not
> have to
> >>> support a different configuration for each OS.  Doing it all
> in-application
> >>> also insulates us from external dependencies... if some future or past
> >>> Ubuntu series (or CentOS) has a different version of rsyslog, it could
> >>> behave differently / require a different configuration, etc.
> >>>
> >>>
> >>>
> >>> On Fri, Aug 8, 2014 at 12:40 PM, David Britton <
> david.brit...@canonical.com>
> >>> wrote:
>  On Fri, Aug 08, 2014 at 12:03:21PM -0400, Nate Finch wrote:
>  [...]
> > remote syslog and to the local file log, we wouldn't need to worry
> about
> > log rotation of the local log screwing up what gets sent to the
> remote
>  Do the standard rsyslog log rotation mechanisms not function well?
> 
>  On Windows, what about the event log (which has remote
>  viewing/aggregation capabilities built in)?
> 
>  --
>  David Britton 
> >>>
> >>>
> >>> --
> >>> Juju-dev mailing list
> >>> Juju-dev@lists.ubuntu.com
> >>> Modify settings or unsubscribe at:
> >>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >>>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: getting rid of all-machines.log

2014-08-14 Thread Nate Finch
The front page of 12factor.net says "offering maximum portability between
execution environments" ... that's exactly what I'm going for.

We're going to support Windows. Windows does not have rsyslog or logrotate.
 We *have* a solution which is cross platform.

My main concern is that if we use rsyslog on linux and something else on
Windows, now we're supporting and maintaining two things, when they could
both be the same.  Why support two things when you can support just one?

I'm sure writing to stdout and redirecting to a logfile that gets watched
by rsyslog is the linux devop standard.  But it has also proved annoyingly
complex for us to configure and maintain. We've wasted several developer
*weeks* of time getting it to work right.

Why would we continue to maintain the rsyslog stuff when it has been such
trouble for us?  Just because it's the standard, doesn't mean it's the
right tool for this specific job.

I talked with Ian about this a little this morning, and my suggestion was
that we give users a way to turn off our in-app log rotation and let them
set it up themselves external to the app (with rsyslog or whatever they
like) I just don't want to have to configure it or support it.

-Nate






On Thu, Aug 14, 2014 at 9:04 AM, Gustavo Niemeyer <
gustavo.nieme...@canonical.com> wrote:

> As a side note, MongoDB offers a "capped collection" mechanism with
> the semantics that you can insert rows at will, and it rolls around
> automatically by replacing oldest entries with the newest ones. This
> tends to be a very convenient way to do structured logging, both on
> the writing and on the reading side. Using that as a general rsyslog
> might be unwise, as there are a number of details to get right and the
> volume may get wild depending on the application, but at least for the
> juju implementation itself it might be quite convenient. Imagine being
> able to ask "please tell me the output of the last run of the
> db-relation-joined hook on unit db/3".
>
> On Wed, Aug 6, 2014 at 11:24 AM, John Meinel 
> wrote:
> > all-machines.log is where we aggregate the messages from all
> > machines/units/etc.
> > It is likely to get big, which is why we want log rotate, but if you
> want to
> > be able to "juju debug-log" and actually get the feed of everything that
> is
> > going on, that needs to be *somewhere*. And yes, we'd like to switch to
> > something like log stash instead, but until we get there we do still need
> > it.
> > John
> > =:->
> >
> >
> >
> > On Wed, Aug 6, 2014 at 4:13 PM, Tim Penhey 
> wrote:
> >>
> >> On 06/08/14 16:11, Nate Finch wrote:
> >> > all-machines.log seems both redundant and a ticking time bomb of disk
> >> > space usage.  Do we really need it?  Can we drop it and maybe later
> >> > schedule some time to use something like logstash that is both more
> >> > featureful and is cross platform compatible (unlike rsyslog)?
> >>
> >> not yet...
> >>
> >> debug-log uses all-machines.log, we cannot get rid of it at this stage.
> >>
> >> We can't drop it until a replacement is in place.
> >>
> >> Tim
> >>
> >>
> >> --
> >> Juju-dev mailing list
> >> Juju-dev@lists.ubuntu.com
> >> Modify settings or unsubscribe at:
> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >
> >
> >
> > --
> > Juju-dev mailing list
> > Juju-dev@lists.ubuntu.com
> > Modify settings or unsubscribe at:
> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >
>
> --
> gustavo @ http://niemeyer.net
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Running godeps -u dependencies.tsv easily

2014-08-14 Thread Dimiter Naydenov
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi all,

Is it just me or others are also getting frustrated for the N-th time
that godeps doesn't fetch new revisions when needed, and you need to
manually go into each dependency's work dir, pull/fetch/update as
needed, then run godeps again.. You know the drill all too well I guess :)

Not anymore - check out these nifty scripts:

run-godeps: http://pastebin.ubuntu.com/8046143/
up-dep: http://paste.ubuntu.com/8046172/

Put both of these somewhere in $PATH, chmod +x both of them, modify
the *_DIR variables inside as needed.

Then, from anywhere, just run:
$ run-godeps

It will go to your juju-core WORK_DIR, run godeps, catch errors like
"missing import path" or "revision not found", do the right thing and
re-run godeps as many times as needed, until all dependencies are updated.

Sweet! :)

I hope somebody else finds it useful, and I'd appreciate feedback for it.

Cheers,
- -- 
Dimiter Naydenov 
juju-core team
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEcBAEBAgAGBQJT7NVmAAoJENzxV2TbLzHwlsEH/1GuYET5r/NvDQX7o9S3naEO
rPlmAM3+jeU3k3Pte50Gn6xpIY0whIjiDnrGE+UJaHaqFTQo1W6hkwTpdFyhioLA
lEaJmDD2yS6nilO1F/y0b4ny6SMSCCeBQxSic29gAJkNcRLRJSXUaAkMi3oyPaz5
N1aPout1xqOtb0ThZk9L4ZcUu4z5C83tck9E1VMGjFsgVJaNr/+5Gi1LCMWMe/aN
BijG5oGiMW3oDtgz5iU8tFbaB3K5FXbyfhq5Od0kTyX69RNRhDzXOlIvC5CDgmX8
L+dIc5G2oUKLh+D+IF/Y5vvVN/IkMXTtuo6tUs/0O0zKxCr4Y0Y1ajXzOWsIp7U=
=xSVd
-END PGP SIGNATURE-

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: Running godeps -u dependencies.tsv easily

2014-08-14 Thread Nate Finch
Godeps should pull down new revisions, as of a month or so ago.  You just
need to do go get -u launchpad.net/godeps  and you'll get the new revision.
 I don't believe it go-gets packages you don't already have, though.


On Thu, Aug 14, 2014 at 11:27 AM, Dimiter Naydenov <
dimiter.nayde...@canonical.com> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi all,
>
> Is it just me or others are also getting frustrated for the N-th time
> that godeps doesn't fetch new revisions when needed, and you need to
> manually go into each dependency's work dir, pull/fetch/update as
> needed, then run godeps again.. You know the drill all too well I guess :)
>
> Not anymore - check out these nifty scripts:
>
> run-godeps: http://pastebin.ubuntu.com/8046143/
> up-dep: http://paste.ubuntu.com/8046172/
>
> Put both of these somewhere in $PATH, chmod +x both of them, modify
> the *_DIR variables inside as needed.
>
> Then, from anywhere, just run:
> $ run-godeps
>
> It will go to your juju-core WORK_DIR, run godeps, catch errors like
> "missing import path" or "revision not found", do the right thing and
> re-run godeps as many times as needed, until all dependencies are updated.
>
> Sweet! :)
>
> I hope somebody else finds it useful, and I'd appreciate feedback for it.
>
> Cheers,
> - --
> Dimiter Naydenov 
> juju-core team
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1
>
> iQEcBAEBAgAGBQJT7NVmAAoJENzxV2TbLzHwlsEH/1GuYET5r/NvDQX7o9S3naEO
> rPlmAM3+jeU3k3Pte50Gn6xpIY0whIjiDnrGE+UJaHaqFTQo1W6hkwTpdFyhioLA
> lEaJmDD2yS6nilO1F/y0b4ny6SMSCCeBQxSic29gAJkNcRLRJSXUaAkMi3oyPaz5
> N1aPout1xqOtb0ThZk9L4ZcUu4z5C83tck9E1VMGjFsgVJaNr/+5Gi1LCMWMe/aN
> BijG5oGiMW3oDtgz5iU8tFbaB3K5FXbyfhq5Od0kTyX69RNRhDzXOlIvC5CDgmX8
> L+dIc5G2oUKLh+D+IF/Y5vvVN/IkMXTtuo6tUs/0O0zKxCr4Y0Y1ajXzOWsIp7U=
> =xSVd
> -END PGP SIGNATURE-
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: getting rid of all-machines.log

2014-08-14 Thread Matt Rae
On Thu, Aug 14, 2014 at 8:23 AM, Nate Finch 
wrote:

> The front page of 12factor.net says "offering maximum portability between
> execution environments" ... that's exactly what I'm going for.
>
> We're going to support Windows. Windows does not have rsyslog or
> logrotate.  We *have* a solution which is cross platform.
>
> My main concern is that if we use rsyslog on linux and something else on
> Windows, now we're supporting and maintaining two things, when they could
> both be the same.  Why support two things when you can support just one?
>
>
I'm sure writing to stdout and redirecting to a logfile that gets watched
> by rsyslog is the linux devop standard.  But it has also proved annoyingly
> complex for us to configure and maintain. We've wasted several developer
> *weeks* of time getting it to work right.
>
> Why would we continue to maintain the rsyslog stuff when it has been such
> trouble for us?  Just because it's the standard, doesn't mean it's the
> right tool for this specific job.
>
>
I talked with Ian about this a little this morning, and my suggestion was
> that we give users a way to turn off our in-app log rotation and let them
> set it up themselves external to the app (with rsyslog or whatever they
> like) I just don't want to have to configure it or support it.
>

Yeah this sounds like a good suggestion. Maybe we can solve the problem in
a cross platform way but still have options to log to stdout/syslog in case
a user wants to integrate with an existing logging system themselves. A lot
of applications have options of logging to stdout or to syslog depending on
which you prefer.

Principle of least astonishment and taco bell programming [2] are a couple
ideas that might be good to keep in mind when deciding how we do it.

[1] http://en.wikipedia.org/wiki/Principle_of_least_astonishment
[2] http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html


>
>
> -Nate
>
>
>
>
>
>
> On Thu, Aug 14, 2014 at 9:04 AM, Gustavo Niemeyer <
> gustavo.nieme...@canonical.com> wrote:
>
>> As a side note, MongoDB offers a "capped collection" mechanism with
>> the semantics that you can insert rows at will, and it rolls around
>> automatically by replacing oldest entries with the newest ones. This
>> tends to be a very convenient way to do structured logging, both on
>> the writing and on the reading side. Using that as a general rsyslog
>> might be unwise, as there are a number of details to get right and the
>> volume may get wild depending on the application, but at least for the
>> juju implementation itself it might be quite convenient. Imagine being
>> able to ask "please tell me the output of the last run of the
>> db-relation-joined hook on unit db/3".
>>
>> On Wed, Aug 6, 2014 at 11:24 AM, John Meinel 
>> wrote:
>> > all-machines.log is where we aggregate the messages from all
>> > machines/units/etc.
>> > It is likely to get big, which is why we want log rotate, but if you
>> want to
>> > be able to "juju debug-log" and actually get the feed of everything
>> that is
>> > going on, that needs to be *somewhere*. And yes, we'd like to switch to
>> > something like log stash instead, but until we get there we do still
>> need
>> > it.
>> > John
>> > =:->
>> >
>> >
>> >
>> > On Wed, Aug 6, 2014 at 4:13 PM, Tim Penhey 
>> wrote:
>> >>
>> >> On 06/08/14 16:11, Nate Finch wrote:
>> >> > all-machines.log seems both redundant and a ticking time bomb of disk
>> >> > space usage.  Do we really need it?  Can we drop it and maybe later
>> >> > schedule some time to use something like logstash that is both more
>> >> > featureful and is cross platform compatible (unlike rsyslog)?
>> >>
>> >> not yet...
>> >>
>> >> debug-log uses all-machines.log, we cannot get rid of it at this stage.
>> >>
>> >> We can't drop it until a replacement is in place.
>> >>
>> >> Tim
>> >>
>> >>
>> >> --
>> >> Juju-dev mailing list
>> >> Juju-dev@lists.ubuntu.com
>> >> Modify settings or unsubscribe at:
>> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> >
>> >
>> >
>> > --
>> > Juju-dev mailing list
>> > Juju-dev@lists.ubuntu.com
>> > Modify settings or unsubscribe at:
>> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> >
>>
>> --
>> gustavo @ http://niemeyer.net
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: getting rid of all-machines.log

2014-08-14 Thread Gustavo Niemeyer
On Thu, Aug 14, 2014 at 12:23 PM, Nate Finch  wrote:
> The front page of 12factor.net says "offering maximum portability between
> execution environments" ... that's exactly what I'm going for.

This can be taken as an excuse to do just about anything.

> We're going to support Windows. Windows does not have rsyslog or logrotate.
> We have a solution which is cross platform.
>
> My main concern is that if we use rsyslog on linux and something else on
> Windows, now we're supporting and maintaining two things, when they could
> both be the same.  Why support two things when you can support just one?

Just to be clear, you really mean "why support two existing and well
known things when I can implement a third thing", right?


gustavo @ http://niemeyer.net

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: getting rid of all-machines.log

2014-08-14 Thread Nate Finch
On Thu, Aug 14, 2014 at 12:24 PM, Gustavo Niemeyer <
gustavo.nieme...@canonical.com> wrote:

>
> > Why support two things when you can support just one?
>
> Just to be clear, you really mean "why support two existing and well
> known things when I can implement a third thing", right?
>

Yes, that is exactly what I mean.
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: getting rid of all-machines.log

2014-08-14 Thread Gustavo Niemeyer
On Thu, Aug 14, 2014 at 1:35 PM, Nate Finch  wrote:
> On Thu, Aug 14, 2014 at 12:24 PM, Gustavo Niemeyer
>  wrote:
>>
>> > Why support two things when you can support just one?
>>
>> Just to be clear, you really mean "why support two existing and well
>> known things when I can implement a third thing", right?
>
> Yes, that is exactly what I mean.

Okay, on that basis and without any better rationale than "12factor
says I can do anything" I'd be tempted to request further analysis on
the problem if the decision was on my hands. There are more
interesting problems to solve than redoing what already exists.


gustavo @ http://niemeyer.net

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: getting rid of all-machines.log

2014-08-14 Thread Nate Finch
I didn't bring up 12 factor, it's irrelevant to my argument.

I'm trying to make our product simpler and easier to maintain.  That is
all.  If there's another cross-platform solution that we can use, I'd be
happy to consider it.  We have to change the code to support Windows.  I'd
rather the diff be +50 -150  than +75 -0.  I don't know how to state it any
simpler than that.


On Thu, Aug 14, 2014 at 1:35 PM, Gustavo Niemeyer <
gustavo.nieme...@canonical.com> wrote:

> On Thu, Aug 14, 2014 at 1:35 PM, Nate Finch 
> wrote:
> > On Thu, Aug 14, 2014 at 12:24 PM, Gustavo Niemeyer
> >  wrote:
> >>
> >> > Why support two things when you can support just one?
> >>
> >> Just to be clear, you really mean "why support two existing and well
> >> known things when I can implement a third thing", right?
> >
> > Yes, that is exactly what I mean.
>
> Okay, on that basis and without any better rationale than "12factor
> says I can do anything" I'd be tempted to request further analysis on
> the problem if the decision was on my hands. There are more
> interesting problems to solve than redoing what already exists.
>
>
> gustavo @ http://niemeyer.net
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: Running godeps -u dependencies.tsv easily

2014-08-14 Thread roger peppe
Indeed, godeps does already have this functionality.
I should probably have announced it, sorry.

It does go-get packages you don't already have too.

Our 'bot script now does something like:

go get -d $ourpackage
godeps -u $ourpackagedir/dependencies.tsv

which means that we don't pull down any dependencies that
we don't need to.

   cheers,
 rog.

On 14 August 2014 16:31, Nate Finch  wrote:
> Godeps should pull down new revisions, as of a month or so ago.  You just
> need to do go get -u launchpad.net/godeps  and you'll get the new revision.
> I don't believe it go-gets packages you don't already have, though.
>
>
> On Thu, Aug 14, 2014 at 11:27 AM, Dimiter Naydenov
>  wrote:
>>
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> Hi all,
>>
>> Is it just me or others are also getting frustrated for the N-th time
>> that godeps doesn't fetch new revisions when needed, and you need to
>> manually go into each dependency's work dir, pull/fetch/update as
>> needed, then run godeps again.. You know the drill all too well I guess :)
>>
>> Not anymore - check out these nifty scripts:
>>
>> run-godeps: http://pastebin.ubuntu.com/8046143/
>> up-dep: http://paste.ubuntu.com/8046172/
>>
>> Put both of these somewhere in $PATH, chmod +x both of them, modify
>> the *_DIR variables inside as needed.
>>
>> Then, from anywhere, just run:
>> $ run-godeps
>>
>> It will go to your juju-core WORK_DIR, run godeps, catch errors like
>> "missing import path" or "revision not found", do the right thing and
>> re-run godeps as many times as needed, until all dependencies are updated.
>>
>> Sweet! :)
>>
>> I hope somebody else finds it useful, and I'd appreciate feedback for it.
>>
>> Cheers,
>> - --
>> Dimiter Naydenov 
>> juju-core team
>> -BEGIN PGP SIGNATURE-
>> Version: GnuPG v1
>>
>> iQEcBAEBAgAGBQJT7NVmAAoJENzxV2TbLzHwlsEH/1GuYET5r/NvDQX7o9S3naEO
>> rPlmAM3+jeU3k3Pte50Gn6xpIY0whIjiDnrGE+UJaHaqFTQo1W6hkwTpdFyhioLA
>> lEaJmDD2yS6nilO1F/y0b4ny6SMSCCeBQxSic29gAJkNcRLRJSXUaAkMi3oyPaz5
>> N1aPout1xqOtb0ThZk9L4ZcUu4z5C83tck9E1VMGjFsgVJaNr/+5Gi1LCMWMe/aN
>> BijG5oGiMW3oDtgz5iU8tFbaB3K5FXbyfhq5Od0kTyX69RNRhDzXOlIvC5CDgmX8
>> L+dIc5G2oUKLh+D+IF/Y5vvVN/IkMXTtuo6tUs/0O0zKxCr4Y0Y1ajXzOWsIp7U=
>> =xSVd
>> -END PGP SIGNATURE-
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Fwd: Running godeps -u dependencies.tsv easily

2014-08-14 Thread roger peppe
Indeed, godeps does already have this functionality.
I should probably have announced it, sorry.

It does go-get packages you don't already have too.

Our 'bot script now does something like:

go get -d $ourpackage
godeps -u $ourpackagedir/dependencies.tsv

which means that we don't pull down any dependencies that
we don't need to.

   cheers,
 rog.

On 14 August 2014 16:31, Nate Finch  wrote:
> Godeps should pull down new revisions, as of a month or so ago.  You just
> need to do go get -u launchpad.net/godeps  and you'll get the new revision.
> I don't believe it go-gets packages you don't already have, though.
>
>
> On Thu, Aug 14, 2014 at 11:27 AM, Dimiter Naydenov
>  wrote:
>>
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> Hi all,
>>
>> Is it just me or others are also getting frustrated for the N-th time
>> that godeps doesn't fetch new revisions when needed, and you need to
>> manually go into each dependency's work dir, pull/fetch/update as
>> needed, then run godeps again.. You know the drill all too well I guess :)
>>
>> Not anymore - check out these nifty scripts:
>>
>> run-godeps: http://pastebin.ubuntu.com/8046143/
>> up-dep: http://paste.ubuntu.com/8046172/
>>
>> Put both of these somewhere in $PATH, chmod +x both of them, modify
>> the *_DIR variables inside as needed.
>>
>> Then, from anywhere, just run:
>> $ run-godeps
>>
>> It will go to your juju-core WORK_DIR, run godeps, catch errors like
>> "missing import path" or "revision not found", do the right thing and
>> re-run godeps as many times as needed, until all dependencies are updated.
>>
>> Sweet! :)
>>
>> I hope somebody else finds it useful, and I'd appreciate feedback for it.
>>
>> Cheers,
>> - --
>> Dimiter Naydenov 
>> juju-core team
>> -BEGIN PGP SIGNATURE-
>> Version: GnuPG v1
>>
>> iQEcBAEBAgAGBQJT7NVmAAoJENzxV2TbLzHwlsEH/1GuYET5r/NvDQX7o9S3naEO
>> rPlmAM3+jeU3k3Pte50Gn6xpIY0whIjiDnrGE+UJaHaqFTQo1W6hkwTpdFyhioLA
>> lEaJmDD2yS6nilO1F/y0b4ny6SMSCCeBQxSic29gAJkNcRLRJSXUaAkMi3oyPaz5
>> N1aPout1xqOtb0ThZk9L4ZcUu4z5C83tck9E1VMGjFsgVJaNr/+5Gi1LCMWMe/aN
>> BijG5oGiMW3oDtgz5iU8tFbaB3K5FXbyfhq5Od0kTyX69RNRhDzXOlIvC5CDgmX8
>> L+dIc5G2oUKLh+D+IF/Y5vvVN/IkMXTtuo6tUs/0O0zKxCr4Y0Y1ajXzOWsIp7U=
>> =xSVd
>> -END PGP SIGNATURE-
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: getting rid of all-machines.log

2014-08-14 Thread Gustavo Niemeyer
On Thu, Aug 14, 2014 at 3:14 PM, Nate Finch  wrote:
> I didn't bring up 12 factor, it's irrelevant to my argument.

Is there someone else sending messages under your name?

On Thu, Aug 14, 2014 at 12:23 PM, Nate Finch  wrote:
> The front page of 12factor.net says "offering maximum portability between
> execution environments" ... that's exactly what I'm going for.

> I'm trying to make our product simpler and easier to maintain.  That is all.
> If there's another cross-platform solution that we can use, I'd be happy to
> consider it.  We have to change the code to support Windows.  I'd rather the
> diff be +50 -150  than +75 -0.  I don't know how to state it any simpler
> than that.

How about simply allowing people to select their own rsyslog target?


gustavo @ http://niemeyer.net

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: getting rid of all-machines.log

2014-08-14 Thread Kapil Thangavelu
On Thu, Aug 14, 2014 at 2:14 PM, Nate Finch 
wrote:

> I didn't bring up 12 factor, it's irrelevant to my argument.
>
> I'm trying to make our product simpler and easier to maintain.  That is
> all.  If there's another cross-platform solution that we can use, I'd be
> happy to consider it.  We have to change the code to support Windows.  I'd
> rather the diff be +50 -150  than +75 -0.  I don't know how to state it
> any simpler than that.
>
>
the abrogation of responsibility which is what ic you adocating for in this
thread,  also makes our product quite a lot less usable imo... Our product
is a distributed system with emergent behavior. Having a debug log is one
of the most useful things you can have to observe the system and back in py
days was one of the most used features and it was just a simple dump to the
db with querying. Its unfortunate that ability to use it usefully didn't
land to core till recently and did so in broken fashion (still requiring
internal tag names for filtering).. or lots more people would be using it.
Gustavo's suggestion of storing the structured log data  in mongo sounds
really good to me. Yes, features are work and require code but that sort of
implementation is also cross platform portable. The current implementation
and proposed alternatives I find somewhat ridicolous in that we basically
dump structured data into an unstructured format only to reparse it every
time we look at it (or ingest into logstash) given that we already have the
structured data. Asking people to setup one of those distributed log
aggregation systems systems and configure them is a huge task, and anyone
suggesting punting that to an end user or charm developer has never setup
one up themselves i suspect. ie. an analogy imo
http://xahlee.info/comp/i/fault-tolerance_NoSQL.png As for the operations
follks who do have them.. we can continue sending messages send to local
syslog and let them collect per their preference.

-k




>
> On Thu, Aug 14, 2014 at 1:35 PM, Gustavo Niemeyer <
> gustavo.nieme...@canonical.com> wrote:
>
>> On Thu, Aug 14, 2014 at 1:35 PM, Nate Finch 
>> wrote:
>> > On Thu, Aug 14, 2014 at 12:24 PM, Gustavo Niemeyer
>> >  wrote:
>> >>
>> >> > Why support two things when you can support just one?
>> >>
>> >> Just to be clear, you really mean "why support two existing and well
>> >> known things when I can implement a third thing", right?
>> >
>> > Yes, that is exactly what I mean.
>>
>> Okay, on that basis and without any better rationale than "12factor
>> says I can do anything" I'd be tempted to request further analysis on
>> the problem if the decision was on my hands. There are more
>> interesting problems to solve than redoing what already exists.
>>
>>
>> gustavo @ http://niemeyer.net
>>
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: getting rid of all-machines.log

2014-08-14 Thread Kapil Thangavelu
if we stopped redirecting the log on linux in our upstart jobs, log start
would rotate the per agent/job log files for us.



On Thu, Aug 14, 2014 at 11:49 AM, Matt Rae  wrote:

>
> On Thu, Aug 14, 2014 at 8:23 AM, Nate Finch 
> wrote:
>
>> The front page of 12factor.net says "offering maximum portability
>> between execution environments" ... that's exactly what I'm going for.
>>
>> We're going to support Windows. Windows does not have rsyslog or
>> logrotate.  We *have* a solution which is cross platform.
>>
>> My main concern is that if we use rsyslog on linux and something else on
>> Windows, now we're supporting and maintaining two things, when they could
>> both be the same.  Why support two things when you can support just one?
>>
>>
> I'm sure writing to stdout and redirecting to a logfile that gets watched
>> by rsyslog is the linux devop standard.  But it has also proved annoyingly
>> complex for us to configure and maintain. We've wasted several developer
>> *weeks* of time getting it to work right.
>>
>> Why would we continue to maintain the rsyslog stuff when it has been such
>> trouble for us?  Just because it's the standard, doesn't mean it's the
>> right tool for this specific job.
>>
>>
> I talked with Ian about this a little this morning, and my suggestion was
>> that we give users a way to turn off our in-app log rotation and let them
>> set it up themselves external to the app (with rsyslog or whatever they
>> like) I just don't want to have to configure it or support it.
>>
>
> Yeah this sounds like a good suggestion. Maybe we can solve the problem in
> a cross platform way but still have options to log to stdout/syslog in case
> a user wants to integrate with an existing logging system themselves. A lot
> of applications have options of logging to stdout or to syslog depending on
> which you prefer.
>
> Principle of least astonishment and taco bell programming [2] are a couple
> ideas that might be good to keep in mind when deciding how we do it.
>
> [1] http://en.wikipedia.org/wiki/Principle_of_least_astonishment
> [2] http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html
>
>
>>
>>
>> -Nate
>>
>>
>>
>>
>>
>>
>> On Thu, Aug 14, 2014 at 9:04 AM, Gustavo Niemeyer <
>> gustavo.nieme...@canonical.com> wrote:
>>
>>> As a side note, MongoDB offers a "capped collection" mechanism with
>>> the semantics that you can insert rows at will, and it rolls around
>>> automatically by replacing oldest entries with the newest ones. This
>>> tends to be a very convenient way to do structured logging, both on
>>> the writing and on the reading side. Using that as a general rsyslog
>>> might be unwise, as there are a number of details to get right and the
>>> volume may get wild depending on the application, but at least for the
>>> juju implementation itself it might be quite convenient. Imagine being
>>> able to ask "please tell me the output of the last run of the
>>> db-relation-joined hook on unit db/3".
>>>
>>> On Wed, Aug 6, 2014 at 11:24 AM, John Meinel 
>>> wrote:
>>> > all-machines.log is where we aggregate the messages from all
>>> > machines/units/etc.
>>> > It is likely to get big, which is why we want log rotate, but if you
>>> want to
>>> > be able to "juju debug-log" and actually get the feed of everything
>>> that is
>>> > going on, that needs to be *somewhere*. And yes, we'd like to switch to
>>> > something like log stash instead, but until we get there we do still
>>> need
>>> > it.
>>> > John
>>> > =:->
>>> >
>>> >
>>> >
>>> > On Wed, Aug 6, 2014 at 4:13 PM, Tim Penhey 
>>> wrote:
>>> >>
>>> >> On 06/08/14 16:11, Nate Finch wrote:
>>> >> > all-machines.log seems both redundant and a ticking time bomb of
>>> disk
>>> >> > space usage.  Do we really need it?  Can we drop it and maybe later
>>> >> > schedule some time to use something like logstash that is both more
>>> >> > featureful and is cross platform compatible (unlike rsyslog)?
>>> >>
>>> >> not yet...
>>> >>
>>> >> debug-log uses all-machines.log, we cannot get rid of it at this
>>> stage.
>>> >>
>>> >> We can't drop it until a replacement is in place.
>>> >>
>>> >> Tim
>>> >>
>>> >>
>>> >> --
>>> >> Juju-dev mailing list
>>> >> Juju-dev@lists.ubuntu.com
>>> >> Modify settings or unsubscribe at:
>>> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>> >
>>> >
>>> >
>>> > --
>>> > Juju-dev mailing list
>>> > Juju-dev@lists.ubuntu.com
>>> > Modify settings or unsubscribe at:
>>> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>> >
>>>
>>> --
>>> gustavo @ http://niemeyer.net
>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> http

Juju trunk now is compatible with Mongo 2.6

2014-08-14 Thread Ian Booth
A little while back we started work as a background task to port Juju to work
with Mongo 2.6. After a final push from Andrew to get everything finalised, the
current Juju trunk now has Mongo 2.6 support finished/working.

Note that Mongo 2.4.x (as shipped in cloud archive and/or trusty) is still the
officially supported version for Juju.

The reason for this work was due to the fact that Mongo 2.6 offers various fixes
and improvements over 2.4; some of these fixes are for issues Juju encounters
but which will not be fixed for 2.4
eg https://jira.mongodb.org/browse/SERVER-11807
Also as time goes on, we need to have a viable pathway to move to Mongo 2.6 (and
later versions) since that's where most of the development focus will be. Plus
Mongo 2.6 is already in proposed for Utopic, so we now have the option to
package juju-mongodb with a consistent version of Mongo as ships with the 
distro.

What we will do now is set up regular test runs using Mongo 2.6 on CI so we'll
have data available about that combination's reliability etc.


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: Juju trunk now is compatible with Mongo 2.6

2014-08-14 Thread David Cheney
Will CI failures using mongodb-2.6 be considered critical and block the build ?

On Fri, Aug 15, 2014 at 10:07 AM, Ian Booth  wrote:
> A little while back we started work as a background task to port Juju to work
> with Mongo 2.6. After a final push from Andrew to get everything finalised, 
> the
> current Juju trunk now has Mongo 2.6 support finished/working.
>
> Note that Mongo 2.4.x (as shipped in cloud archive and/or trusty) is still the
> officially supported version for Juju.
>
> The reason for this work was due to the fact that Mongo 2.6 offers various 
> fixes
> and improvements over 2.4; some of these fixes are for issues Juju encounters
> but which will not be fixed for 2.4
> eg https://jira.mongodb.org/browse/SERVER-11807
> Also as time goes on, we need to have a viable pathway to move to Mongo 2.6 
> (and
> later versions) since that's where most of the development focus will be. Plus
> Mongo 2.6 is already in proposed for Utopic, so we now have the option to
> package juju-mongodb with a consistent version of Mongo as ships with the 
> distro.
>
> What we will do now is set up regular test runs using Mongo 2.6 on CI so we'll
> have data available about that combination's reliability etc.
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: Juju trunk now is compatible with Mongo 2.6

2014-08-14 Thread Ian Booth
No, not yet. The landing tests are still run using 2.4.
The job to test with mongo 2.6 is/will initially be separate.

On 15/08/14 11:11, David Cheney wrote:
> Will CI failures using mongodb-2.6 be considered critical and block the build 
> ?
> 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: Juju trunk now is compatible with Mongo 2.6

2014-08-14 Thread David Cheney
Sorry, I was not clear.

If a bug is found in the mongo 2.6 build will that be considered
critical and block the landing bot.

On Fri, Aug 15, 2014 at 12:32 PM, Ian Booth  wrote:
> No, not yet. The landing tests are still run using 2.4.
> The job to test with mongo 2.6 is/will initially be separate.
>
> On 15/08/14 11:11, David Cheney wrote:
>> Will CI failures using mongodb-2.6 be considered critical and block the 
>> build ?
>>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: Juju trunk now is compatible with Mongo 2.6

2014-08-14 Thread Ian Booth
There is no mongo 2.6 build; it's all the one code base.
juju-core works out-of-the-box on both mongo 2.4 and 2.6.
The landing tests are run on top of mongo 2.4 as usual. If any of the tests
fail, then the bot will be blocked.

Perhaps I don't understand the question?

On 15/08/14 13:47, David Cheney wrote:
> Sorry, I was not clear.
> 
> If a bug is found in the mongo 2.6 build will that be considered
> critical and block the landing bot.
> 
> On Fri, Aug 15, 2014 at 12:32 PM, Ian Booth  wrote:
>> No, not yet. The landing tests are still run using 2.4.
>> The job to test with mongo 2.6 is/will initially be separate.
>>
>> On 15/08/14 11:11, David Cheney wrote:
>>> Will CI failures using mongodb-2.6 be considered critical and block the 
>>> build ?
>>>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: Juju trunk now is compatible with Mongo 2.6

2014-08-14 Thread Andrew Wilkins
On Fri, Aug 15, 2014 at 11:58 AM, Ian Booth  wrote:

> There is no mongo 2.6 build; it's all the one code base.
> juju-core works out-of-the-box on both mongo 2.4 and 2.6.
> The landing tests are run on top of mongo 2.4 as usual. If any of the tests
> fail, then the bot will be blocked.
>
> Perhaps I don't understand the question?


If someone manually tests with Mongo 2.6, will we file that as a critical
stop-the-line bug?


> On 15/08/14 13:47, David Cheney wrote:
> > Sorry, I was not clear.
> >
> > If a bug is found in the mongo 2.6 build will that be considered
> > critical and block the landing bot.
> >
> > On Fri, Aug 15, 2014 at 12:32 PM, Ian Booth 
> wrote:
> >> No, not yet. The landing tests are still run using 2.4.
> >> The job to test with mongo 2.6 is/will initially be separate.
> >>
> >> On 15/08/14 11:11, David Cheney wrote:
> >>> Will CI failures using mongodb-2.6 be considered critical and block
> the build ?
> >>>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: Juju trunk now is compatible with Mongo 2.6

2014-08-14 Thread Andrew Wilkins
(if someone tests and finds a bug)...


On Fri, Aug 15, 2014 at 12:17 PM, Andrew Wilkins <
andrew.wilk...@canonical.com> wrote:

> On Fri, Aug 15, 2014 at 11:58 AM, Ian Booth 
> wrote:
>
>> There is no mongo 2.6 build; it's all the one code base.
>> juju-core works out-of-the-box on both mongo 2.4 and 2.6.
>> The landing tests are run on top of mongo 2.4 as usual. If any of the
>> tests
>> fail, then the bot will be blocked.
>>
>> Perhaps I don't understand the question?
>
>
> If someone manually tests with Mongo 2.6, will we file that as a critical
> stop-the-line bug?
>
>
>>  On 15/08/14 13:47, David Cheney wrote:
>> > Sorry, I was not clear.
>> >
>> > If a bug is found in the mongo 2.6 build will that be considered
>> > critical and block the landing bot.
>> >
>> > On Fri, Aug 15, 2014 at 12:32 PM, Ian Booth 
>> wrote:
>> >> No, not yet. The landing tests are still run using 2.4.
>> >> The job to test with mongo 2.6 is/will initially be separate.
>> >>
>> >> On 15/08/14 11:11, David Cheney wrote:
>> >>> Will CI failures using mongodb-2.6 be considered critical and block
>> the build ?
>> >>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


Re: Juju trunk now is compatible with Mongo 2.6

2014-08-14 Thread Ian Booth


On 15/08/14 14:17, Andrew Wilkins wrote:
> On Fri, Aug 15, 2014 at 11:58 AM, Ian Booth  wrote:
> 
>> There is no mongo 2.6 build; it's all the one code base.
>> juju-core works out-of-the-box on both mongo 2.4 and 2.6.
>> The landing tests are run on top of mongo 2.4 as usual. If any of the tests
>> fail, then the bot will be blocked.
>>
>> Perhaps I don't understand the question?
> 
> 
> If someone manually tests with Mongo 2.6, will we file that as a critical
> stop-the-line bug?
>

Right now, we won't block landing but the bug will be treated as very important
and we would want a fix ASAP. Essentially, we're still in an evaluation phase.

Once we have data to enable us to make a decision that Juju works well with
mongo 2.6 and we may want to consider packaging that version into juju-mongodb,
then things become different. But until then there's little to gain from
blocking everyone based on a bug in something we're not officially ready to 
support.

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev