Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

2016-05-18 Thread Christian Muirhead 
Yeah, I think so too - it seems like one of those performance bugs where
the fix would be obvious to someone familiar with the codebase. But the
response to Gustavo's bug (from October!) didn't give me much hope of it
being fixed very soon.

On Wed, May 18, 2016 at 11:36 AM David Cheney 
wrote:

> This feels like a bug in mongodb. We store approximately zero data in
> mongodb during test runs -- seriously, one machine, a charm at most,
> that's it. It mongodb has such amazingly high overheads just start to
> store data that sounds like a serious problem that is being ignored.
>
> On Wed, May 18, 2016 at 7:32 PM, Christian Muirhead
>  wrote:
> > WiredTiger is *much* slower at creating and dropping indexes and
> > collections. I haven't worked out why that is, other than doing some
> > stracing and seeing that a lot of time is spent in fdatasync - I haven't
> dug
> > into the mongo source code.
> > There's a bit more detail in these bugs:
> > https://bugs.launchpad.net/juju-core/+bug/1573294
> > https://jira.mongodb.org/browse/SERVER-21198
> >
> > I've changed the tests so that instead of dropping and recreating
> databases
> > in teardown and setup we clear out all of the collections (except the
> > transaction collections) between tests. Obviously that's worse from the
> > perspective of test isolation, but it seems to work well - better than I
> was
> > expecting, to be honest.
> >
> > Cheers,
> > Christian
> >
> > On Wed, May 18, 2016 at 9:58 AM roger peppe 
> > wrote:
> >>
> >> Out of interest, what's causing the 3.2 slowdown and what's the hack to
> >> speed it up again?
> >>
> >> On 18 May 2016 09:51, "Christian Muirhead"
> >>  wrote:
> >>>
> >>>
> >>>
> >>> On Wed, May 18, 2016 at 2:04 AM David Cheney <
> david.che...@canonical.com>
> >>> wrote:
> 
>  100x more webscale
> >>>
> >>> Ha!
> >>>
> >>> I'm *just about* finished the hack to make the state tests on 3.2 run
> in
> >>> about the same time as on 2.4. On my machine the state tests take
> 6m24s on
> >>> 3.2 and the old version took 4m56s. Which is still worse,
> unfortunately, but
> >>> at least it isn't 100x worse. So if there are stability benefits to
> running
> >>> the tests on 3.2 it's still a win, I guess?
> >>>
> >>>
> 
> 
>  On Wed, May 18, 2016 at 11:02 AM, Horacio Duran
>   wrote:
>  > For now we are trying to go around mongo issues that make the tests
>  > 100x
>  > slower (yes one hundred) once this is fixed we should start using
>  > mongo 3.2
>  > exclusively since 2.4 iirc is EOL or near. The issue lies in the new
>  > storage
>  > engine, which we could skip if mmapv1 ( the old one) wasn't also
>  > nearing EOL
>  > I am currently on the phone but if You want more details I can dig
> up
>  > the
>  > bug with details of what I am talking about.
>  >
>  >
>  > On Tuesday, 17 May 2016, David Cheney 
>  > wrote:
>  >>
>  >> What's the plan for mongo 3.2 ? Will we be required to support 2.x
>  >> versions for the foreseeable future, or is there a possibility to
>  >> make
>  >> it a build or run time failure if mongo < 3.2 is installed on the
>  >> host
>  >> ?
>  >>
>  >> On Wed, May 18, 2016 at 9:01 AM, Martin Packman
>  >>  wrote:
>  >> > On 17/05/2016, Curtis Hovey-Canonical 
> wrote:
>  >> >>
>  >> >> The juju-mongo2.6 package will be be preferred by juju 1.2.5 in
>  >> >> xenial
>  >> >> and without other changes, 2.4 will be used by all other 1.25
>  >> >> series.
>  >> >
>  >> > This isn't yet true, there's a bug open for it:
>  >> >
>  >> > "Use juju-mongodb2.6 for 1.25 on xenial"
>  >> > <
> https://bugs.launchpad.net/ubuntu/+source/juju-core/+bug/1570650>
>  >> >
>  >> > I had made the packaging change, but without juju code changes as
>  >> > well
>  >> > it just went and installed the old (2.4) juju-mongodb anyway when
>  >> > setting up a state server.
>  >> >
>  >> > Martin
>  >> >
>  >> > --
>  >> > Juju-dev mailing list
>  >> > Juju-dev@lists.ubuntu.com
>  >> > Modify settings or unsubscribe at:
>  >> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
>  >>
>  >> --
>  >> Juju-dev mailing list
>  >> Juju-dev@lists.ubuntu.com
>  >> Modify settings or unsubscribe at:
>  >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> 
>  --
>  Juju-dev mailing list
>  Juju-dev@lists.ubuntu.com
>  Modify settings or unsubscribe at:
>  https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >>>
> >>>
> >>> --
> >>> Juju-dev mailing list
> >>> Juju-dev@lists.ubuntu.com
> >>> Modify settings or unsubscribe at:
> >>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >>>
> >
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

2016-05-18 Thread David Cheney 
This feels like a bug in mongodb. We store approximately zero data in
mongodb during test runs -- seriously, one machine, a charm at most,
that's it. It mongodb has such amazingly high overheads just start to
store data that sounds like a serious problem that is being ignored.

On Wed, May 18, 2016 at 7:32 PM, Christian Muirhead
 wrote:
> WiredTiger is *much* slower at creating and dropping indexes and
> collections. I haven't worked out why that is, other than doing some
> stracing and seeing that a lot of time is spent in fdatasync - I haven't dug
> into the mongo source code.
> There's a bit more detail in these bugs:
> https://bugs.launchpad.net/juju-core/+bug/1573294
> https://jira.mongodb.org/browse/SERVER-21198
>
> I've changed the tests so that instead of dropping and recreating databases
> in teardown and setup we clear out all of the collections (except the
> transaction collections) between tests. Obviously that's worse from the
> perspective of test isolation, but it seems to work well - better than I was
> expecting, to be honest.
>
> Cheers,
> Christian
>
> On Wed, May 18, 2016 at 9:58 AM roger peppe 
> wrote:
>>
>> Out of interest, what's causing the 3.2 slowdown and what's the hack to
>> speed it up again?
>>
>> On 18 May 2016 09:51, "Christian Muirhead"
>>  wrote:
>>>
>>>
>>>
>>> On Wed, May 18, 2016 at 2:04 AM David Cheney 
>>> wrote:

 100x more webscale
>>>
>>> Ha!
>>>
>>> I'm *just about* finished the hack to make the state tests on 3.2 run in
>>> about the same time as on 2.4. On my machine the state tests take 6m24s on
>>> 3.2 and the old version took 4m56s. Which is still worse, unfortunately, but
>>> at least it isn't 100x worse. So if there are stability benefits to running
>>> the tests on 3.2 it's still a win, I guess?
>>>
>>>


 On Wed, May 18, 2016 at 11:02 AM, Horacio Duran
  wrote:
 > For now we are trying to go around mongo issues that make the tests
 > 100x
 > slower (yes one hundred) once this is fixed we should start using
 > mongo 3.2
 > exclusively since 2.4 iirc is EOL or near. The issue lies in the new
 > storage
 > engine, which we could skip if mmapv1 ( the old one) wasn't also
 > nearing EOL
 > I am currently on the phone but if You want more details I can dig up
 > the
 > bug with details of what I am talking about.
 >
 >
 > On Tuesday, 17 May 2016, David Cheney 
 > wrote:
 >>
 >> What's the plan for mongo 3.2 ? Will we be required to support 2.x
 >> versions for the foreseeable future, or is there a possibility to
 >> make
 >> it a build or run time failure if mongo < 3.2 is installed on the
 >> host
 >> ?
 >>
 >> On Wed, May 18, 2016 at 9:01 AM, Martin Packman
 >>  wrote:
 >> > On 17/05/2016, Curtis Hovey-Canonical  wrote:
 >> >>
 >> >> The juju-mongo2.6 package will be be preferred by juju 1.2.5 in
 >> >> xenial
 >> >> and without other changes, 2.4 will be used by all other 1.25
 >> >> series.
 >> >
 >> > This isn't yet true, there's a bug open for it:
 >> >
 >> > "Use juju-mongodb2.6 for 1.25 on xenial"
 >> > 
 >> >
 >> > I had made the packaging change, but without juju code changes as
 >> > well
 >> > it just went and installed the old (2.4) juju-mongodb anyway when
 >> > setting up a state server.
 >> >
 >> > Martin
 >> >
 >> > --
 >> > Juju-dev mailing list
 >> > Juju-dev@lists.ubuntu.com
 >> > Modify settings or unsubscribe at:
 >> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
 >>
 >> --
 >> Juju-dev mailing list
 >> Juju-dev@lists.ubuntu.com
 >> Modify settings or unsubscribe at:
 >> https://lists.ubuntu.com/mailman/listinfo/juju-dev

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

2016-05-18 Thread Christian Muirhead 
Michael, thanks for all the clear info in the bugs by the way!

I also got good results from running the tests under tmpfs - the 3.2 run
was almost acceptably fast. But obviously it's not practical to require
every machine running the tests to have a tmpfs mounted (or somehow mount
one in the test run).

On Wed, May 18, 2016 at 10:36 AM Michael Hudson-Doyle <
michael.hud...@canonical.com> wrote:

> On 18 May 2016 at 21:32, Christian Muirhead
>  wrote:
> > WiredTiger is *much* slower at creating and dropping indexes and
> > collections. I haven't worked out why that is, other than doing some
> > stracing and seeing that a lot of time is spent in fdatasync - I haven't
> dug
> > into the mongo source code.
>
> Yeah, this is what I concluded too. I tried running mongo under
> eatmydata but it didn't work for reasons I didn't get around to
> understanding. I even built a mongo with the fdatasync call commented
> out but then I moved onto other things...
>
> Cheers,
> mwh
>
> > There's a bit more detail in these bugs:
> > https://bugs.launchpad.net/juju-core/+bug/1573294
> > https://jira.mongodb.org/browse/SERVER-21198
> >
> > I've changed the tests so that instead of dropping and recreating
> databases
> > in teardown and setup we clear out all of the collections (except the
> > transaction collections) between tests. Obviously that's worse from the
> > perspective of test isolation, but it seems to work well - better than I
> was
> > expecting, to be honest.
> >
> > Cheers,
> > Christian
> >
> > On Wed, May 18, 2016 at 9:58 AM roger peppe 
> > wrote:
> >>
> >> Out of interest, what's causing the 3.2 slowdown and what's the hack to
> >> speed it up again?
> >>
> >> On 18 May 2016 09:51, "Christian Muirhead"
> >>  wrote:
> >>>
> >>>
> >>>
> >>> On Wed, May 18, 2016 at 2:04 AM David Cheney <
> david.che...@canonical.com>
> >>> wrote:
> 
>  100x more webscale
> >>>
> >>> Ha!
> >>>
> >>> I'm *just about* finished the hack to make the state tests on 3.2 run
> in
> >>> about the same time as on 2.4. On my machine the state tests take
> 6m24s on
> >>> 3.2 and the old version took 4m56s. Which is still worse,
> unfortunately, but
> >>> at least it isn't 100x worse. So if there are stability benefits to
> running
> >>> the tests on 3.2 it's still a win, I guess?
> >>>
> >>>
> 
> 
>  On Wed, May 18, 2016 at 11:02 AM, Horacio Duran
>   wrote:
>  > For now we are trying to go around mongo issues that make the tests
>  > 100x
>  > slower (yes one hundred) once this is fixed we should start using
>  > mongo 3.2
>  > exclusively since 2.4 iirc is EOL or near. The issue lies in the new
>  > storage
>  > engine, which we could skip if mmapv1 ( the old one) wasn't also
>  > nearing EOL
>  > I am currently on the phone but if You want more details I can dig
> up
>  > the
>  > bug with details of what I am talking about.
>  >
>  >
>  > On Tuesday, 17 May 2016, David Cheney 
>  > wrote:
>  >>
>  >> What's the plan for mongo 3.2 ? Will we be required to support 2.x
>  >> versions for the foreseeable future, or is there a possibility to
>  >> make
>  >> it a build or run time failure if mongo < 3.2 is installed on the
>  >> host
>  >> ?
>  >>
>  >> On Wed, May 18, 2016 at 9:01 AM, Martin Packman
>  >>  wrote:
>  >> > On 17/05/2016, Curtis Hovey-Canonical 
> wrote:
>  >> >>
>  >> >> The juju-mongo2.6 package will be be preferred by juju 1.2.5 in
>  >> >> xenial
>  >> >> and without other changes, 2.4 will be used by all other 1.25
>  >> >> series.
>  >> >
>  >> > This isn't yet true, there's a bug open for it:
>  >> >
>  >> > "Use juju-mongodb2.6 for 1.25 on xenial"
>  >> > <
> https://bugs.launchpad.net/ubuntu/+source/juju-core/+bug/1570650>
>  >> >
>  >> > I had made the packaging change, but without juju code changes as
>  >> > well
>  >> > it just went and installed the old (2.4) juju-mongodb anyway when
>  >> > setting up a state server.
>  >> >
>  >> > Martin
>  >> >
>  >> > --
>  >> > Juju-dev mailing list
>  >> > Juju-dev@lists.ubuntu.com
>  >> > Modify settings or unsubscribe at:
>  >> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
>  >>
>  >> --
>  >> Juju-dev mailing list
>  >> Juju-dev@lists.ubuntu.com
>  >> Modify settings or unsubscribe at:
>  >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> 
>  --
>  Juju-dev mailing list
>  Juju-dev@lists.ubuntu.com
>  Modify settings or unsubscribe at:
>  https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >>>
> >>>
> >>> --
> >>> Juju-dev mailing list
> >>> Juju-dev@lists.ubuntu.com
> >>> Modify settings or unsubscribe at:
> >>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >>>
> >
> > --
> > Juju-dev mailing list
> > Juju-dev@lists.ubuntu.com
> > Modify settings or unsubscribe at:
> > https://lists.

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

2016-05-18 Thread Michael Hudson-Doyle 
On 18 May 2016 at 21:32, Christian Muirhead
 wrote:
> WiredTiger is *much* slower at creating and dropping indexes and
> collections. I haven't worked out why that is, other than doing some
> stracing and seeing that a lot of time is spent in fdatasync - I haven't dug
> into the mongo source code.

Yeah, this is what I concluded too. I tried running mongo under
eatmydata but it didn't work for reasons I didn't get around to
understanding. I even built a mongo with the fdatasync call commented
out but then I moved onto other things...

Cheers,
mwh

> There's a bit more detail in these bugs:
> https://bugs.launchpad.net/juju-core/+bug/1573294
> https://jira.mongodb.org/browse/SERVER-21198
>
> I've changed the tests so that instead of dropping and recreating databases
> in teardown and setup we clear out all of the collections (except the
> transaction collections) between tests. Obviously that's worse from the
> perspective of test isolation, but it seems to work well - better than I was
> expecting, to be honest.
>
> Cheers,
> Christian
>
> On Wed, May 18, 2016 at 9:58 AM roger peppe 
> wrote:
>>
>> Out of interest, what's causing the 3.2 slowdown and what's the hack to
>> speed it up again?
>>
>> On 18 May 2016 09:51, "Christian Muirhead"
>>  wrote:
>>>
>>>
>>>
>>> On Wed, May 18, 2016 at 2:04 AM David Cheney 
>>> wrote:

 100x more webscale
>>>
>>> Ha!
>>>
>>> I'm *just about* finished the hack to make the state tests on 3.2 run in
>>> about the same time as on 2.4. On my machine the state tests take 6m24s on
>>> 3.2 and the old version took 4m56s. Which is still worse, unfortunately, but
>>> at least it isn't 100x worse. So if there are stability benefits to running
>>> the tests on 3.2 it's still a win, I guess?
>>>
>>>


 On Wed, May 18, 2016 at 11:02 AM, Horacio Duran
  wrote:
 > For now we are trying to go around mongo issues that make the tests
 > 100x
 > slower (yes one hundred) once this is fixed we should start using
 > mongo 3.2
 > exclusively since 2.4 iirc is EOL or near. The issue lies in the new
 > storage
 > engine, which we could skip if mmapv1 ( the old one) wasn't also
 > nearing EOL
 > I am currently on the phone but if You want more details I can dig up
 > the
 > bug with details of what I am talking about.
 >
 >
 > On Tuesday, 17 May 2016, David Cheney 
 > wrote:
 >>
 >> What's the plan for mongo 3.2 ? Will we be required to support 2.x
 >> versions for the foreseeable future, or is there a possibility to
 >> make
 >> it a build or run time failure if mongo < 3.2 is installed on the
 >> host
 >> ?
 >>
 >> On Wed, May 18, 2016 at 9:01 AM, Martin Packman
 >>  wrote:
 >> > On 17/05/2016, Curtis Hovey-Canonical  wrote:
 >> >>
 >> >> The juju-mongo2.6 package will be be preferred by juju 1.2.5 in
 >> >> xenial
 >> >> and without other changes, 2.4 will be used by all other 1.25
 >> >> series.
 >> >
 >> > This isn't yet true, there's a bug open for it:
 >> >
 >> > "Use juju-mongodb2.6 for 1.25 on xenial"
 >> > 
 >> >
 >> > I had made the packaging change, but without juju code changes as
 >> > well
 >> > it just went and installed the old (2.4) juju-mongodb anyway when
 >> > setting up a state server.
 >> >
 >> > Martin
 >> >
 >> > --
 >> > Juju-dev mailing list
 >> > Juju-dev@lists.ubuntu.com
 >> > Modify settings or unsubscribe at:
 >> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
 >>
 >> --
 >> Juju-dev mailing list
 >> Juju-dev@lists.ubuntu.com
 >> Modify settings or unsubscribe at:
 >> https://lists.ubuntu.com/mailman/listinfo/juju-dev

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

2016-05-18 Thread Christian Muirhead 
WiredTiger is *much* slower at creating and dropping indexes and
collections. I haven't worked out why that is, other than doing some
stracing and seeing that a lot of time is spent in fdatasync - I haven't
dug into the mongo source code.
There's a bit more detail in these bugs:
https://bugs.launchpad.net/juju-core/+bug/1573294
https://jira.mongodb.org/browse/SERVER-21198

I've changed the tests so that instead of dropping and recreating databases
in teardown and setup we clear out all of the collections (except the
transaction collections) between tests. Obviously that's worse from the
perspective of test isolation, but it seems to work well - better than I
was expecting, to be honest.

Cheers,
Christian

On Wed, May 18, 2016 at 9:58 AM roger peppe 
wrote:

> Out of interest, what's causing the 3.2 slowdown and what's the hack to
> speed it up again?
> On 18 May 2016 09:51, "Christian Muirhead" <
> christian.muirh...@canonical.com> wrote:
>
>>
>>
>> On Wed, May 18, 2016 at 2:04 AM David Cheney 
>> wrote:
>>
>>> 100x more webscale
>>>
>> Ha!
>>
>> I'm *just about* finished the hack to make the state tests on 3.2 run in
>> about the same time as on 2.4. On my machine the state tests take 6m24s on
>> 3.2 and the old version took 4m56s. Which is still worse, unfortunately,
>> but at least it isn't 100x worse. So if there are stability benefits to
>> running the tests on 3.2 it's still a win, I guess?
>>
>>
>>
>>>
>>> On Wed, May 18, 2016 at 11:02 AM, Horacio Duran
>>>  wrote:
>>> > For now we are trying to go around mongo issues that make the tests
>>> 100x
>>> > slower (yes one hundred) once this is fixed we should start using
>>> mongo 3.2
>>> > exclusively since 2.4 iirc is EOL or near. The issue lies in the new
>>> storage
>>> > engine, which we could skip if mmapv1 ( the old one) wasn't also
>>> nearing EOL
>>> > I am currently on the phone but if You want more details I can dig up
>>> the
>>> > bug with details of what I am talking about.
>>> >
>>> >
>>> > On Tuesday, 17 May 2016, David Cheney 
>>> wrote:
>>> >>
>>> >> What's the plan for mongo 3.2 ? Will we be required to support 2.x
>>> >> versions for the foreseeable future, or is there a possibility to make
>>> >> it a build or run time failure if mongo < 3.2 is installed on the host
>>> >> ?
>>> >>
>>> >> On Wed, May 18, 2016 at 9:01 AM, Martin Packman
>>> >>  wrote:
>>> >> > On 17/05/2016, Curtis Hovey-Canonical  wrote:
>>> >> >>
>>> >> >> The juju-mongo2.6 package will be be preferred by juju 1.2.5 in
>>> xenial
>>> >> >> and without other changes, 2.4 will be used by all other 1.25
>>> series.
>>> >> >
>>> >> > This isn't yet true, there's a bug open for it:
>>> >> >
>>> >> > "Use juju-mongodb2.6 for 1.25 on xenial"
>>> >> > 
>>> >> >
>>> >> > I had made the packaging change, but without juju code changes as
>>> well
>>> >> > it just went and installed the old (2.4) juju-mongodb anyway when
>>> >> > setting up a state server.
>>> >> >
>>> >> > Martin
>>> >> >
>>> >> > --
>>> >> > Juju-dev mailing list
>>> >> > Juju-dev@lists.ubuntu.com
>>> >> > Modify settings or unsubscribe at:
>>> >> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>> >>
>>> >> --
>>> >> Juju-dev mailing list
>>> >> Juju-dev@lists.ubuntu.com
>>> >> Modify settings or unsubscribe at:
>>> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

2016-05-18 Thread roger peppe 
Out of interest, what's causing the 3.2 slowdown and what's the hack to
speed it up again?
On 18 May 2016 09:51, "Christian Muirhead" 
wrote:

>
>
> On Wed, May 18, 2016 at 2:04 AM David Cheney 
> wrote:
>
>> 100x more webscale
>>
> Ha!
>
> I'm *just about* finished the hack to make the state tests on 3.2 run in
> about the same time as on 2.4. On my machine the state tests take 6m24s on
> 3.2 and the old version took 4m56s. Which is still worse, unfortunately,
> but at least it isn't 100x worse. So if there are stability benefits to
> running the tests on 3.2 it's still a win, I guess?
>
>
>
>>
>> On Wed, May 18, 2016 at 11:02 AM, Horacio Duran
>>  wrote:
>> > For now we are trying to go around mongo issues that make the tests 100x
>> > slower (yes one hundred) once this is fixed we should start using mongo
>> 3.2
>> > exclusively since 2.4 iirc is EOL or near. The issue lies in the new
>> storage
>> > engine, which we could skip if mmapv1 ( the old one) wasn't also
>> nearing EOL
>> > I am currently on the phone but if You want more details I can dig up
>> the
>> > bug with details of what I am talking about.
>> >
>> >
>> > On Tuesday, 17 May 2016, David Cheney 
>> wrote:
>> >>
>> >> What's the plan for mongo 3.2 ? Will we be required to support 2.x
>> >> versions for the foreseeable future, or is there a possibility to make
>> >> it a build or run time failure if mongo < 3.2 is installed on the host
>> >> ?
>> >>
>> >> On Wed, May 18, 2016 at 9:01 AM, Martin Packman
>> >>  wrote:
>> >> > On 17/05/2016, Curtis Hovey-Canonical  wrote:
>> >> >>
>> >> >> The juju-mongo2.6 package will be be preferred by juju 1.2.5 in
>> xenial
>> >> >> and without other changes, 2.4 will be used by all other 1.25
>> series.
>> >> >
>> >> > This isn't yet true, there's a bug open for it:
>> >> >
>> >> > "Use juju-mongodb2.6 for 1.25 on xenial"
>> >> > 
>> >> >
>> >> > I had made the packaging change, but without juju code changes as
>> well
>> >> > it just went and installed the old (2.4) juju-mongodb anyway when
>> >> > setting up a state server.
>> >> >
>> >> > Martin
>> >> >
>> >> > --
>> >> > Juju-dev mailing list
>> >> > Juju-dev@lists.ubuntu.com
>> >> > Modify settings or unsubscribe at:
>> >> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> >>
>> >> --
>> >> Juju-dev mailing list
>> >> Juju-dev@lists.ubuntu.com
>> >> Modify settings or unsubscribe at:
>> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

2016-05-18 Thread Christian Muirhead 
On Wed, May 18, 2016 at 2:04 AM David Cheney 
wrote:

> 100x more webscale
>
Ha!

I'm *just about* finished the hack to make the state tests on 3.2 run in
about the same time as on 2.4. On my machine the state tests take 6m24s on
3.2 and the old version took 4m56s. Which is still worse, unfortunately,
but at least it isn't 100x worse. So if there are stability benefits to
running the tests on 3.2 it's still a win, I guess?



>
> On Wed, May 18, 2016 at 11:02 AM, Horacio Duran
>  wrote:
> > For now we are trying to go around mongo issues that make the tests 100x
> > slower (yes one hundred) once this is fixed we should start using mongo
> 3.2
> > exclusively since 2.4 iirc is EOL or near. The issue lies in the new
> storage
> > engine, which we could skip if mmapv1 ( the old one) wasn't also nearing
> EOL
> > I am currently on the phone but if You want more details I can dig up the
> > bug with details of what I am talking about.
> >
> >
> > On Tuesday, 17 May 2016, David Cheney 
> wrote:
> >>
> >> What's the plan for mongo 3.2 ? Will we be required to support 2.x
> >> versions for the foreseeable future, or is there a possibility to make
> >> it a build or run time failure if mongo < 3.2 is installed on the host
> >> ?
> >>
> >> On Wed, May 18, 2016 at 9:01 AM, Martin Packman
> >>  wrote:
> >> > On 17/05/2016, Curtis Hovey-Canonical  wrote:
> >> >>
> >> >> The juju-mongo2.6 package will be be preferred by juju 1.2.5 in
> xenial
> >> >> and without other changes, 2.4 will be used by all other 1.25 series.
> >> >
> >> > This isn't yet true, there's a bug open for it:
> >> >
> >> > "Use juju-mongodb2.6 for 1.25 on xenial"
> >> > 
> >> >
> >> > I had made the packaging change, but without juju code changes as well
> >> > it just went and installed the old (2.4) juju-mongodb anyway when
> >> > setting up a state server.
> >> >
> >> > Martin
> >> >
> >> > --
> >> > Juju-dev mailing list
> >> > Juju-dev@lists.ubuntu.com
> >> > Modify settings or unsubscribe at:
> >> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >>
> >> --
> >> Juju-dev mailing list
> >> Juju-dev@lists.ubuntu.com
> >> Modify settings or unsubscribe at:
> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

2016-05-17 Thread David Cheney 
100x more webscale

On Wed, May 18, 2016 at 11:02 AM, Horacio Duran
 wrote:
> For now we are trying to go around mongo issues that make the tests 100x
> slower (yes one hundred) once this is fixed we should start using mongo 3.2
> exclusively since 2.4 iirc is EOL or near. The issue lies in the new storage
> engine, which we could skip if mmapv1 ( the old one) wasn't also nearing EOL
> I am currently on the phone but if You want more details I can dig up the
> bug with details of what I am talking about.
>
>
> On Tuesday, 17 May 2016, David Cheney  wrote:
>>
>> What's the plan for mongo 3.2 ? Will we be required to support 2.x
>> versions for the foreseeable future, or is there a possibility to make
>> it a build or run time failure if mongo < 3.2 is installed on the host
>> ?
>>
>> On Wed, May 18, 2016 at 9:01 AM, Martin Packman
>>  wrote:
>> > On 17/05/2016, Curtis Hovey-Canonical  wrote:
>> >>
>> >> The juju-mongo2.6 package will be be preferred by juju 1.2.5 in xenial
>> >> and without other changes, 2.4 will be used by all other 1.25 series.
>> >
>> > This isn't yet true, there's a bug open for it:
>> >
>> > "Use juju-mongodb2.6 for 1.25 on xenial"
>> > 
>> >
>> > I had made the packaging change, but without juju code changes as well
>> > it just went and installed the old (2.4) juju-mongodb anyway when
>> > setting up a state server.
>> >
>> > Martin
>> >
>> > --
>> > Juju-dev mailing list
>> > Juju-dev@lists.ubuntu.com
>> > Modify settings or unsubscribe at:
>> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

2016-05-17 Thread Horacio Duran 
For now we are trying to go around mongo issues that make the tests 100x
slower (yes one hundred) once this is fixed we should start using mongo 3.2
exclusively since 2.4 iirc is EOL or near. The issue lies in the new
storage engine, which we could skip if mmapv1 ( the old one) wasn't also
nearing EOL I am currently on the phone but if You want more details I can
dig up the bug with details of what I am talking about.

On Tuesday, 17 May 2016, David Cheney  wrote:

> What's the plan for mongo 3.2 ? Will we be required to support 2.x
> versions for the foreseeable future, or is there a possibility to make
> it a build or run time failure if mongo < 3.2 is installed on the host
> ?
>
> On Wed, May 18, 2016 at 9:01 AM, Martin Packman
> > wrote:
> > On 17/05/2016, Curtis Hovey-Canonical  > wrote:
> >>
> >> The juju-mongo2.6 package will be be preferred by juju 1.2.5 in xenial
> >> and without other changes, 2.4 will be used by all other 1.25 series.
> >
> > This isn't yet true, there's a bug open for it:
> >
> > "Use juju-mongodb2.6 for 1.25 on xenial"
> > 
> >
> > I had made the packaging change, but without juju code changes as well
> > it just went and installed the old (2.4) juju-mongodb anyway when
> > setting up a state server.
> >
> > Martin
> >
> > --
> > Juju-dev mailing list
> > Juju-dev@lists.ubuntu.com 
> > Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com 
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

2016-05-17 Thread David Cheney 
What's the plan for mongo 3.2 ? Will we be required to support 2.x
versions for the foreseeable future, or is there a possibility to make
it a build or run time failure if mongo < 3.2 is installed on the host
?

On Wed, May 18, 2016 at 9:01 AM, Martin Packman
 wrote:
> On 17/05/2016, Curtis Hovey-Canonical  wrote:
>>
>> The juju-mongo2.6 package will be be preferred by juju 1.2.5 in xenial
>> and without other changes, 2.4 will be used by all other 1.25 series.
>
> This isn't yet true, there's a bug open for it:
>
> "Use juju-mongodb2.6 for 1.25 on xenial"
> 
>
> I had made the packaging change, but without juju code changes as well
> it just went and installed the old (2.4) juju-mongodb anyway when
> setting up a state server.
>
> Martin
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

2016-05-17 Thread Martin Packman 
On 17/05/2016, Curtis Hovey-Canonical  wrote:
>
> The juju-mongo2.6 package will be be preferred by juju 1.2.5 in xenial
> and without other changes, 2.4 will be used by all other 1.25 series.

This isn't yet true, there's a bug open for it:

"Use juju-mongodb2.6 for 1.25 on xenial"


I had made the packaging change, but without juju code changes as well
it just went and installed the old (2.4) juju-mongodb anyway when
setting up a state server.

Martin

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

2016-05-17 Thread Curtis Hovey-Canonical 
On Tue, May 17, 2016 at 4:22 AM, David Cheney
 wrote:
> What is the story with mongo ? It's constantly causing builds to fail
> CI because of it's complete shitness.
>
> I've heard that some people have moved to mongo 3.2 which fixes the
> problem, but as CI clearly is running the old rubbish version, this
> clearly isn't a problem which can be called fixed.

Ci runs the juju unit tests as specified by the juju makefile. eg
make install-deps
make test

^ I don't see mongodb3.2 in the make file. Their rules actually call
for juju-mongodb or mongodb-server. Someone can easily fix their
makefile. I made changes recently to support lxd and the golang 1.6
compiler used by ppc64el, arm64 and s390x.

There are cases where the makefile doesn't work, such as windows and
centos. We manually installed mongo on the hosts. We can put a card on
the board to change the mongo versions if Cheryl and Alexis want.
Note, that centos and windows don't host mongo in real life, so
something is wrong if the test suite requires a controller run on an
unsupported OS.

> What is the solution ? Will mongo 2.6 be deprecated entirely and the
> build fail it the mongo version installed is < 3.2 ?

The juju-mongo2.6 package will be be preferred by juju 1.2.5 in xenial
and without other changes, 2.4 will be used by all other 1.25 series.

Precise wont ever have juju-mongodb3.2 (and it does not use
juju-mongodb2.4 either). Nor will wily in it remaining two months.
Trusty will get juju-mongodb3.2 soon. We have two packaging changed
queued for yakkety before we can begin the backports to xenial and
trusty.

-- 
Curtis Hovey
Canonical Cloud Development and Operations
http://launchpad.net/~sinzui

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

2016-05-17 Thread David Cheney 
What is the story with mongo ? It's constantly causing builds to fail
CI because of it's complete shitness.

I've heard that some people have moved to mongo 3.2 which fixes the
problem, but as CI clearly is running the old rubbish version, this
clearly isn't a problem which can be called fixed.

What is the solution ? Will mongo 2.6 be deprecated entirely and the
build fail it the mongo version installed is < 3.2 ?

Thanks

Dave

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev