Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

On Wed, May 18, 2016 at 2:04 AM David Cheney wrote: > 100x more webscale > Ha! I'm *just about* finished the hack to make the state tests on 3.2 run in about the same time as on 2.4. On my machine the state tests take 6m24s on 3.2 and the old version took 4m56s. Which is still worse, unfortunate

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

Out of interest, what's causing the 3.2 slowdown and what's the hack to speed it up again? On 18 May 2016 09:51, "Christian Muirhead" wrote: > > > On Wed, May 18, 2016 at 2:04 AM David Cheney > wrote: > >> 100x more webscale >> > Ha! > > I'm *just about* finished the hack to make the state tests

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

WiredTiger is *much* slower at creating and dropping indexes and collections. I haven't worked out why that is, other than doing some stracing and seeing that a lot of time is spent in fdatasync - I haven't dug into the mongo source code. There's a bit more detail in these bugs: https://bugs.launch

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

On 18 May 2016 at 21:32, Christian Muirhead wrote: > WiredTiger is *much* slower at creating and dropping indexes and > collections. I haven't worked out why that is, other than doing some > stracing and seeing that a lot of time is spent in fdatasync - I haven't dug > into the mongo source code.

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

Michael, thanks for all the clear info in the bugs by the way! I also got good results from running the tests under tmpfs - the 3.2 run was almost acceptably fast. But obviously it's not practical to require every machine running the tests to have a tmpfs mounted (or somehow mount one in the test

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

This feels like a bug in mongodb. We store approximately zero data in mongodb during test runs -- seriously, one machine, a charm at most, that's it. It mongodb has such amazingly high overheads just start to store data that sounds like a serious problem that is being ignored. On Wed, May 18, 2016

Re: Today I submitted 5 PR's to be merged, 3 failed because mongo shat itself

Yeah, I think so too - it seems like one of those performance bugs where the fix would be obvious to someone familiar with the codebase. But the response to Gustavo's bug (from October!) didn't give me much hope of it being fixed very soon. On Wed, May 18, 2016 at 11:36 AM David Cheney wrote: >

Re: The mean time for a CI run has risen to 33 minutes

Not sure if this is the isolation problem you're speaking of, but if so, it's been fixed. https://launchpad.net/bugs/1564511 Martin Packman writes: > On 16/05/2016, David Cheney wrote: >> This got significantly worse in the last 6 weeks. What happened ? > > Either the juju tests are slower,

Three CI builds in the last 48 hours have failed because launchpad had a lie down

We already have godeps which can take a set of vcs repos and flick them to the right revisions. Why does CI check out every single dependency from upstream every single time we do a build ? That introduces wc -l dependencies.tsv points of failure to every single CI run -- not to mention the severa

Re: Three CI builds in the last 48 hours have failed because launchpad had a lie down

CI does a build from scratch in a pristine VM. There is some discussion about changing that in a variety of ways (in a container in a long running VM, etc), but it doesn't have a local copy to work from so it has to pull it from upstream each time. John =:-> On Wed, May 18, 2016 at 8:05 PM, Davi

Re: Three CI builds in the last 48 hours have failed because launchpad had a lie down

This is a huge waste of wall time and bandwidth, and increases the chance of failure significantly. This is something I would like to see changed. On Thu, May 19, 2016 at 12:06 PM, John Meinel wrote: > CI does a build from scratch in a pristine VM. There is some discussion > about changing that i

Re: Three CI builds in the last 48 hours have failed because launchpad had a lie down

I agree. We should snapshot a known-good starting point and start from there. It's a huge waste of time to do apt update / upgrade / install git, bzr, mercurial, etc every single time we run CI. That's like 1/3rd the time it takes to run the unit test CI job. On Wed, May 18, 2016 at 10:35 PM D

Awful dependency problem caused by romulus

Hello, github.com/juju/juju/cmd/juju/commands: github.com/juju/romulus/cmd/commands: github.com/juju/romulus/cmd/setplan: < github.com/juju/juju/api/service: github.com/juju/juju/cmd/modelcmd: cmd/juju depends on the romulus repository, and the romulus reposi

Re: Awful dependency problem caused by romulus

On Wed, May 18, 2016 at 11:02 PM, David Cheney wrote: > Hello, > > github.com/juju/juju/cmd/juju/commands: > github.com/juju/romulus/cmd/commands: > github.com/juju/romulus/cmd/setplan: < > github.com/juju/juju/api/service: > github.com/juju/juju/cmd/modelcmd

Re: Three CI builds in the last 48 hours have failed because launchpad had a lie down

On 19 May 2016 at 02:05, David Cheney wrote: > We already have godeps which can take a set of vcs repos and flick > them to the right revisions. > > Why does CI check out every single dependency from upstream every > single time we do a build ? That introduces wc -l dependencies.tsv > points of fa