Re: Moving udd away from sqlite

2012-07-23 Thread Jelmer Vernooij
On Thu, Jun 14, 2012 at 06:23:24PM -0400, James Westby wrote: > We're interested in moving our deployment of udd away from sqlite to > postgres, and we're interested in doing the same thing for the package > importer deployment. > There are two main reasons for this: 1) that we were asked to by >

Re: Moving udd away from sqlite

2012-07-04 Thread James Westby
On Tue, 03 Jul 2012 18:12:06 +0100, James Westby wrote: > I think I've addressed all the comments so far, and I'm keen to move > ahead with the deployment before we get too close to the > weekend. Therefore if there are no objections I'd like to merge these > and deploy Wed morning UK time. Hi,

Re: Moving udd away from sqlite

2012-07-03 Thread James Westby
On Mon, 02 Jul 2012 11:03:41 +0100, James Westby wrote: > My apologies, there was some wrong default branch somewhere. They are > all re-proposed now: > > https://code.launchpad.net/~james-w/udd/storm/+merge/113003 > https://code.launchpad.net/~james-w/udd/storm-unicode-fixes/+merge/113004 >

Re: Moving udd away from sqlite

2012-07-02 Thread James Westby
On Mon, 02 Jul 2012 11:54:00 +0200, John Arbash Meinel wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 7/2/2012 9:36 AM, James Westby wrote: > > On Thu, 21 Jun 2012 11:29:08 -0400, James Westby > > wrote: > >> I need to do that, as well as some unicode fixes. I'll get those > >>

Re: Moving udd away from sqlite

2012-07-02 Thread John Arbash Meinel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 7/2/2012 9:36 AM, James Westby wrote: > On Thu, 21 Jun 2012 11:29:08 -0400, James Westby > wrote: >> I need to do that, as well as some unicode fixes. I'll get those >> done and up for review by the end of this week. > > Belatedly here they are: >

Re: Moving udd away from sqlite

2012-07-02 Thread James Westby
On Thu, 21 Jun 2012 11:29:08 -0400, James Westby wrote: > I need to do that, as well as some unicode fixes. I'll get those done > and up for review by the end of this week. Belatedly here they are: https://code.launchpad.net/~james-w/udd/storm/+merge/112983 https://code.launchpad.net/~james

Re: Moving udd away from sqlite

2012-06-25 Thread James Westby
On Thu, 21 Jun 2012 11:29:08 -0400, James Westby wrote: > I need to do that, as well as some unicode fixes. I'll get those done > and up for review by the end of this week. I didn't manage to get all the changes made, so I'll continue with this now and we'll push the deployment back until the re

Re: Moving udd away from sqlite

2012-06-21 Thread James Westby
On Thu, 21 Jun 2012 18:22:22 +0200, Vincent Ladeuil wrote: > > Rollback is to revert the storm code again > > Restore the dbs. I don't think we should do that if we have no evidence of data corruption. We'd be repeating work for no benefit. > > Rollback is to stop the importer, revert

Re: Moving udd away from sqlite

2012-06-21 Thread Vincent Ladeuil
> James Westby writes: > On Thu, 21 Jun 2012 15:35:44 +0200, John Arbash Meinel wrote: >> As I understand it, James still needs to do some work to get Storm to >> not require as strict of an isolation level. For me personally, I'd >> rather he was the one doing the rollout a

Re: Moving udd away from sqlite

2012-06-21 Thread James Westby
On Thu, 21 Jun 2012 15:35:44 +0200, John Arbash Meinel wrote: > As I understand it, James still needs to do some work to get Storm to > not require as strict of an isolation level. For me personally, I'd > rather he was the one doing the rollout and we can help monitor it. > But we certainly want

Re: Moving udd away from sqlite

2012-06-21 Thread John Arbash Meinel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 ... > We'd definitely be in favour of this. Do we need to co-ordinate > timings with you, or can we just fire off the MP and let you > handle it, the deploy & rollback? > > From our end, it would probably be best if you could do the > rollout with t

Re: Moving udd away from sqlite

2012-06-21 Thread Jonathan Lange
On 21 June 2012 12:23, John Arbash Meinel wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > ... > >> We haven't had any movement on this for a few days. >> >> To sum up, this is my understanding of the situation. 1. We need to >> switch our udd to postgres 2. We need to make code change

Re: Moving udd away from sqlite

2012-06-21 Thread John Arbash Meinel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 ... > We haven't had any movement on this for a few days. > > To sum up, this is my understanding of the situation. 1. We need to > switch our udd to postgres 2. We need to make code changes to do > so 3. We want to integrate these changes into trunk

Re: Moving udd away from sqlite

2012-06-21 Thread Jonathan Lange
On 19 June 2012 12:43, John Arbash Meinel wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > ... >> Some routes are: - move to postgresql using the lower level APIs >> and add storm later - work on the storm patchset to make it >> reliable with sqlite - do both postgresql and sqlite at t

Re: Moving udd away from sqlite

2012-06-19 Thread John Arbash Meinel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 ... > Some routes are: - move to postgresql using the lower level APIs > and add storm later - work on the storm patchset to make it > reliable with sqlite - do both postgresql and sqlite at the same > time. > > My recommendation, given the existing i

Re: Moving udd away from sqlite

2012-06-18 Thread Robert Collins
On Tue, Jun 19, 2012 at 3:09 AM, James Westby wrote: >> Since https://bugs.launchpad.net/udd/+bug/724893 has been filed, we >> haven't understood *when* the importer fails but we know it can fail and >> it failed more often recently (and a lot during your first attempt if I >> remember correctly).

Re: Moving udd away from sqlite

2012-06-18 Thread James Westby
On Mon, 18 Jun 2012 10:45:26 +0200, Vincent Ladeuil wrote: > > James Westby writes: > > > On Fri, 15 Jun 2012 12:34:12 +0200, Vincent Ladeuil > wrote: > >> > It's not magic. It's moving from a database that's not designed for > >> > concurrent use to one that is designed for c

Re: Moving udd away from sqlite

2012-06-18 Thread Vincent Ladeuil
> James Westby writes: > On Fri, 15 Jun 2012 12:34:12 +0200, Vincent Ladeuil wrote: >> > It's not magic. It's moving from a database that's not designed for >> > concurrent use to one that is designed for concurrent use. >> >> Despite not being designed for concurrent u

Re: Moving udd away from sqlite

2012-06-15 Thread James Westby
On Fri, 15 Jun 2012 12:34:12 +0200, Vincent Ladeuil wrote: > > It's not magic. It's moving from a database that's not designed for > > concurrent use to one that is designed for concurrent use. > > Despite not being designed for concurrent use, it *is* used this way and > lock contention

Re: Moving udd away from sqlite

2012-06-15 Thread Vincent Ladeuil
> Jonathan Lange writes: >> There is a work in progress regarding the jubany package importer >> deployment, let's try to no step on each other feet ;) >> > What's the work in progress? Setting up a quantal lxc container that can be used for both test and production with

Re: Moving udd away from sqlite

2012-06-15 Thread Jonathan Lange
On 15 June 2012 09:56, Vincent Ladeuil wrote: >> James Westby writes: > >    > Hi, > >    > We're interested in moving our deployment of udd away from sqlite >    > to postgres, and we're interested in doing the same thing for the >    > package importer deployment. > > There is a work in pro

Re: Moving udd away from sqlite

2012-06-15 Thread Vincent Ladeuil
> James Westby writes: > Hi, > We're interested in moving our deployment of udd away from sqlite > to postgres, and we're interested in doing the same thing for the > package importer deployment. There is a work in progress regarding the jubany package importer deployment, l

Re: Moving udd away from sqlite

2012-06-14 Thread Dmitrijs Ledkovs
On 14/06/12 23:23, James Westby wrote: > > 3) Deploy the storm code, but migrate the db to postgres at the same > time. It introduces more changes at the same time so is riskier, > but we're fairly sure we won't see the locking issues with > postgres. I'm pretty sure that we can s

Moving udd away from sqlite

2012-06-14 Thread James Westby
Hi, We're interested in moving our deployment of udd away from sqlite to postgres, and we're interested in doing the same thing for the package importer deployment. There are two main reasons for this: 1) that we were asked to by canonical sysadmins, and 2) because it's the right thing to do. udd