Re: Moving udd away from sqlite

[James Westby]

On Tue, 03 Jul 2012 18:12:06 +0100, James Westby  
wrote:
> I think I've addressed all the comments so far, and I'm keen to move
> ahead with the deployment before we get too close to the
> weekend. Therefore if there are no objections I'd like to merge these
> and deploy Wed morning UK time.

Hi,

The importer is now running on storm again.

There were a couple of bugs that I fixed, but I don't believe bad data
should have come from either of them.

There have been some database locked errors, but so far everything has
kept running without deadlocking.

I'll continue to keep an eye on it while we work on the postgres stuff.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Moving udd away from sqlite

[James Westby]

On Mon, 02 Jul 2012 11:03:41 +0100, James Westby  
wrote:
> My apologies, there was some wrong default branch somewhere. They are
> all re-proposed now:
> 
>   https://code.launchpad.net/~james-w/udd/storm/+merge/113003
>   https://code.launchpad.net/~james-w/udd/storm-unicode-fixes/+merge/113004
>   
> https://code.launchpad.net/~james-w/udd/storm-sqlite-db-provider/+merge/113005

Hi,

I think I've addressed all the comments so far, and I'm keen to move
ahead with the deployment before we get too close to the
weekend. Therefore if there are no objections I'd like to merge these
and deploy Wed morning UK time.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Moving udd away from sqlite

[James Westby]

On Mon, 02 Jul 2012 11:54:00 +0200, John Arbash Meinel  
wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> On 7/2/2012 9:36 AM, James Westby wrote:
> > On Thu, 21 Jun 2012 11:29:08 -0400, James Westby
> >  wrote:
> >> I need to do that, as well as some unicode fixes. I'll get those
> >> done and up for review by the end of this week.
> > 
> > Belatedly here they are:
> > 
> > https://code.launchpad.net/~james-w/udd/storm/+merge/112983 
> > https://code.launchpad.net/~james-w/udd/storm-unicode-fixes/+merge/112984
> >
> > 
> https://code.launchpad.net/~james-w/udd/storm-sqlite-db-provider/+merge/112985
> > 
> 
> I think you used the wrong target. All of those are requests to merge
> into lp:~james-w/udd/storm-sqlite
> 
> And I think it has already landed there (all of the diffs are empty).

My apologies, there was some wrong default branch somewhere. They are
all re-proposed now:

  https://code.launchpad.net/~james-w/udd/storm/+merge/113003
  https://code.launchpad.net/~james-w/udd/storm-unicode-fixes/+merge/113004
  https://code.launchpad.net/~james-w/udd/storm-sqlite-db-provider/+merge/113005

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Moving udd away from sqlite

[James Westby]

On Thu, 21 Jun 2012 11:29:08 -0400, James Westby  
wrote:
> I need to do that, as well as some unicode fixes. I'll get those done
> and up for review by the end of this week.

Belatedly here they are:

  https://code.launchpad.net/~james-w/udd/storm/+merge/112983
  https://code.launchpad.net/~james-w/udd/storm-unicode-fixes/+merge/112984
  https://code.launchpad.net/~james-w/udd/storm-sqlite-db-provider/+merge/112985

> I'll then spin up an ec2 instance to run in parallel and do everything
> except push the branches back, and leave this running over the weekend.

I spun up three udd instances on ec2, each rigged to not hit
codehosting, but to have something like the db access pattern of doing
imports.

There was one running tip of trunk, one running the code we tried to
deploy last time, and one running my current code.

After leaving them for a weekend grep "database is locked" debug_log*
gives:

  trunk: 0
  old code: 7199
  new code: 0

(with me logging in a couple of times to kill the old code because it
was in a deadlocked state.)

which is hopefully good evidence that we should have fewer issues this
time.

The new code also imported roughly the same number of packages as trunk
in that time, so there doesn't seem to be any large performance impact
from the changes.

> Then on Tuesday morning my time (starting 1300 UTC) I'll do a deployment
> of the code to production if (reviews, ec2) don't show any problems. We
> can then carefully monitor the service for the next few days.

I'm ready to do this once the code is reviewed.

I'll send another email before I do it.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Moving udd away from sqlite

[James Westby]

On Thu, 21 Jun 2012 11:29:08 -0400, James Westby  
wrote:
> I need to do that, as well as some unicode fixes. I'll get those done
> and up for review by the end of this week.

I didn't manage to get all the changes made, so I'll continue with this
now and we'll push the deployment back until the rest of the steps have
been completed. Therefore there won't be a deployment of storm tomorrow.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Moving udd away from sqlite

[James Westby]

On Thu, 21 Jun 2012 18:22:22 +0200, Vincent Ladeuil  
wrote:
> > Rollback is to revert the storm code again 
> 
> Restore the dbs.

I don't think we should do that if we have no evidence of data
corruption. We'd be repeating work for no benefit.

> > Rollback is to stop the importer, revert the config changes,
> 
> restore the dbs.

No need, as changes would be written to postgres, so just switching back
to sqlite is an effective restore.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Moving udd away from sqlite

[James Westby]

On Thu, 21 Jun 2012 15:35:44 +0200, John Arbash Meinel  
wrote:
> As I understand it, James still needs to do some work to get Storm to
> not require as strict of an isolation level. For me personally, I'd
> rather he was the one doing the rollout and we can help monitor it.
> But we certainly want to know when it is going to happen.

I need to do that, as well as some unicode fixes. I'll get those done
and up for review by the end of this week.

I'll then spin up an ec2 instance to run in parallel and do everything
except push the branches back, and leave this running over the weekend.

Then on Tuesday morning my time (starting 1300 UTC) I'll do a deployment
of the code to production if (reviews, ec2) don't show any problems. We
can then carefully monitor the service for the next few days.

Rollback is to revert the storm code again and requeue any packages that
failed due to storm-related errors.

Once this is rolled out we can start work on the migration to postgres:

  * Obtaining a postgres server and credentials from IS
  * Prepare config changes to use postgres
  * Stop the importer
  * Migrate dbs in to postgres
  * Deploy the config changes
  * Restart the importer

Rollback is to stop the importer, revert the config changes, then
restart the importer and let it catch back up.

How does that sound to everyone?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Moving udd away from sqlite

[James Westby]

On Mon, 18 Jun 2012 10:45:26 +0200, Vincent Ladeuil  
wrote:
> >>>>> James Westby  writes:
> 
> > On Fri, 15 Jun 2012 12:34:12 +0200, Vincent Ladeuil 
>  wrote:
> >> > It's not magic. It's moving from a database that's not designed for
> >> > concurrent use to one that is designed for concurrent use.
> >> 
> >> Despite not being designed for concurrent use, it *is* used this
> >> way and lock contentions have been encountered leading me to
> >> believe that the actual *design* needs to be fixed. The fact that
> >> changing the db is triggering more contentions is a symptom of a
> >> deeper issue.
> 
> > Changing the db access layer is triggering that, we were still
> > running on the same (non-multi-user db). I agree that the design
> > needs to be fixed, and that's exactly what we're taking about,
> > fixing it by moving a db that is designed for multi-user use.
> 
> It looks like your understanding of the issue is better than mine here,
> would you mind sharing that knowledge in an automated test (with the
> added benefit that we won't regress in this area) ?
> 
> Just this week-end we had an add-import-jobs failure:
> 
> Traceback (most recent call last):
>   File "/srv/package-import.canonical.com/new/scripts/bin/add-import-jobs", 
> line 5, in 
> sys.exit(main())
>   File 
> "/srv/package-import.canonical.com/new/scripts/udd/scripts/add_import_jobs.py",
>  line 17, in main
> icommon.create_import_jobs(lp, status_db)
>   File "/srv/package-import.canonical.com/new/scripts/udd/icommon.py", line 
> 304, in create_import_jobs
> status_db.add_import_jobs(checked, newest_published)
>   File "/srv/package-import.canonical.com/new/scripts/udd/icommon.py", line 
> 633, in add_import_jobs
> self._add_job(c, package, self.JOB_TYPE_NEW)
>   File "/srv/package-import.canonical.com/new/scripts/udd/icommon.py", line 
> 615, in _add_job
> datetime.utcnow(), None, None))
> sqlite3.OperationalError: database is locked
> 
> So we already know that add_import_jobs is involved in the bug (with the
> current sqlite-based implementation), who is the other user in this case
> and how can this be reproduced ?

Each connection to sqlite is another "user", so each of the cron
scripts, as well as the imports themselves, and several connections
within mass-import are all the users.

When a write operation is started a global lock is acquired that locks
out any other writers until the operation is complete.

If the lock is held then the library will wait up to a timeout
(configured to be 30s for udd) for the lock to be released before giving
up.

The errors like the above occur when the timeout is reached, so either
another transaction took more than 30s to release the lock, or there
were lots of connections trying to take the lock, and this one didn't
win before the 30s was up.

When we change to storm it forces pysqlite in to a higher isolation level,
so that transactions are started when any statement is executed. My
guess is that this means locks are taken more frequently and are held
for longer, giving more contention errors.

Postgres doesn't have a global lock, it has table or row locks, so that
clients will only hit lock contention if they are changing the same
data, which will be much less frequent.

How can I show that in an automated test? I can write an XFAIL test that
if two connections are opened, one starts a transaction and then the
other hits an locking exception if it tries to do anything, but that
doesn't seem to prove much about the operation of the system.

> I.e. reproducing the add_import_jobs failure in a test that will fail
> with sqlite and succeed with your changes will demonstrate we've
> captured (and fixed) at least one lock contention.

We are dealing with probabilistic failure though. I can demonstrate that
in a deterministic situation changing two separate tables under sqlite
will take global locks, but I can't prove that we will never get
contention under postgres.

> If the test suite cannot be trusted to catch most of the issues that
> happen in production, the test suite should be fixed.
> 
> You're not implying that testing in production being needed, the test
> suite is useless right ?

No, I'm saying that the only measure of whether something runs correctly
in production is whether it runs in production.

> From that, can we imagine a test that will import a few packages and
> compare the corresponding dbs ?

We can do that as part of testing the migration script.

> > It can be restarted with the dbs from whenever the transition starts a

Re: Moving udd away from sqlite

[James Westby]

On Fri, 15 Jun 2012 12:34:12 +0200, Vincent Ladeuil  
wrote:
> > It's not magic. It's moving from a database that's not designed for
> > concurrent use to one that is designed for concurrent use.
> 
> Despite not being designed for concurrent use, it *is* used this way and
> lock contentions have been encountered leading me to believe that the
> actual *design* needs to be fixed. The fact that changing the db is
> triggering more contentions is a symptom of a deeper issue.

Changing the db access layer is triggering that, we were still running
on the same (non-multi-user db). I agree that the design needs to be
fixed, and that's exactly what we're taking about, fixing it by moving a
db that is designed for multi-user use.

> Well, when the correctness and safety is demonstrated, the context (and
> hence my own answer) will probably be different but until then I just
> can't say.
> 
> > And I'm very reluctant to fork without an actual plan for merging
> > back: how to know when it's safe & how to actually achieve it.
> 
> And I have no idea (nor time right now) to debug the fallouts of such a
> change that the actual package importer doesn't need. Hence my tendency
> to consider that demonstrating the validity of this change should be
> achieved first.

But you just said above that you *do* think it needs to be fixed?

How can we demonstrate the validity of the change? We can only
demonstrate that it doesn't break production by running the changes in
production. What would satisfy you that it was unlikely to break
production?

> Would there be a script to migrate from sqlite to PG ?

Yes.

> Can the package importer be re-started with empty dbs and catch up (how
> long will it take ? Days ? Weeks ?). Can this trigger bugs because the
> importer don't remember what it pushed to lp ?

It can be restarted with the dbs from whenever the transition starts and
it will catch up in roughly the between starting the transition and
rolling back. There may be a few bugs due to replaying things, but we do
it all the time (e.g. removing revids and re-importing when someone does
push --overwrite)

> Or do you expect us to see another peek like
> http://webnumbr.com/ubuntu-package-import-failures.from%282012-01-24%29 ?

Hopefully not.

> Yes, that's why we're not in a position to safely accept such a change !
> 
> And all the time spent on integrating these changes is not spent on
> allowing them to be accepted in good conditions.

Sorry, I don't understand this, could you explain?

> > but it demonstrated the locking problem and is how he came up with
> > those options.
> 
> Then the test improvements are certainly valuable to backport to lp:udd
> or is there nothing to reuse from the EC2 experiment ?

It wasn't a set of unit tests, it was a set of tests of a live system,
so nothing to backport.

> > We have had a lot of experience recently working with Canonical IS
> > to get new servers and new staging servers deployed. If you want a
> > staging server, we'd be happy to help you and would gladly
> > advocate for you in their priority queue.
> 
> Great to hear :) I should come back soon on this topic, just a quick
> question though: are the new servers running lucid or precise ?

New servers are being installed with precise. If an existing server is
used it may be lucid, but there is a concerted effort to upgrade all
machines to precise currently underway.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Upgrading pristine-xz on jubany

[James Westby]

On Fri, 15 Jun 2012 10:32:59 +0200, Vincent Ladeuil  
wrote:
> > Barry Warsaw  writes:
> 
> > On Jun 14, 2012, at 05:21 PM, Vincent Ladeuil wrote:
> >> - I'm already running successful tests inside a quantal lxc container 
> :)
> 
> > It has become for many of us not just a nice-to-have but a
> > must-have for Ubuntu development.
> 
> That's my understanding as well.
> 
> Here are my last achievements for the week:
> 
> - I got in touch with pristine-tar maintainers resulting in a trivial
>   bugfix included in 1.25. This is a small step in getting *known* as a
>   primary consumer but it also demonstrates that we can get fixes
>   upstream quickly (1.25 has already been uploaded to sid and quantal).
> 
> - I got in touch with xz maintainers and a fix is on its way there
>   (many thanks to Lasse Collin for its invaluable help here). This will
>   require an additional fix to pristine-xz which I will submit as soon
>   as I can test the xz fix).
> 
> With these fixes in place, on quantal, it should remain only < 10
> pristine-tar import failures out of the current 338 on jubany.

That's great work Vincent, I look forward to it being deployed, however
that happens :-)

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Moving udd away from sqlite

[James Westby]

Hi,

We're interested in moving our deployment of udd away from sqlite to
postgres, and we're interested in doing the same thing for the package
importer deployment.

There are two main reasons for this: 1) that we were asked to by
canonical sysadmins, and 2) because it's the right thing to do. udd has
evolved past the point where sqlite is a good choice for the db.

It's not purely about cleanliness though. Currently udd will wait up to
30s to acquire a sqlite lock, and I believe that it frequently uses a
significant portion of that time. Therefore moving to postgres with its
improved locking would improve the performance of the importer (though I
have no numbers to show that it would significantly affect latency or
throughput.)

As you know, we've tried a first step towards this previously. We ported
the db access to storm, so that we could replace the db underneath,
taking advantage of storm's support for multiple database types.

That didn't go so well, as after a few hours/days of usage it would
deadlock due to sqlite locking. That's one of the reasons why sqlite
isn't a great choice, as it shows we are currently operating close to
the threshold that sqlite allows.

After lots of head-scratching I believe I've worked out why changing to
storm made this happen when the underlying db was the same. Storm forces
sqlite to operate at a higher isolation level, so udd was taking locks
more frequently or holding them for longer, leading to more contention
and eventually the deadlock.

I haven't proven that it would fix the deadlock issue, just that reduces
the incidence of "database is locked" errors on a test instance rigged
to stress the db even more.

As I see it we have N options:

  1) Deploy the same code as last time (with some fixes for bugs that
 Max spotted that were unrelated to locking.) Live with the degraded
 performance while we migrate to postgres.

  2) Deploy the same code with a change to stop storm forcing the
 isolation level. This may fix the deadlock issues, but we won't be
 sure. It's also not clear that storm will function correctly in
 that condition.

  3) Deploy the storm code, but migrate the db to postgres at the same
 time. It introduces more changes at the same time so is riskier,
 but we're fairly sure we won't see the locking issues with
 postgres. I'm pretty sure that we can still rollback from this
 fairly easily, as we can just go back to the sqlite dbs and the
 importer will pick up from where it left off, duplicating some
 work.

  4) Leave the package importer on sqlite and do something else for our
 instance (fork basically)

Are there any others?

What do you think of these options? We'd like to avoid option 4 if
possible.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: DEB_VENDOR & package imports

[James Westby]

On Thu, 03 May 2012 14:49:19 +0100, Dmitrijs Ledkovs 
 wrote:
> I think, the lp:debian/* imports should be run with
> 
>export DEB_VENDOR=Debian
> 
> This should make the lp:debian/package to match the source package as it
> is unpacked/built on Debian.

I'm agreed on this, and Colin pointed it out too and I filed a bug:
https://bugs.launchpad.net/udd/+bug/911496 .

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Importer add-import-jobs cronjob temporarily disabled

[James Westby]

On Sat, 28 Apr 2012 10:52:44 +0100, Max Bowsher <_...@maxb.eu> wrote:
> See https://bugs.launchpad.net/udd/+bug/990394

Thanks for catching this.

Fwiw it's not normally an issue as it normally takes longer to run
branch-distro.py and re-enable things, and once there has been another
package published to the new release it only acts on all the packages
once. For whatever reason that didn't happen this time.

I'll reply to the bug with some comments about how it could be better
handled in future.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Stormified importer has introduced unintended change in data being written to sqlite

[James Westby]

On Sun, 22 Apr 2012 01:52:16 +0100, Max Bowsher <_...@maxb.eu> wrote:
> I've repaired the database, mainly by making lots of use of:
> 
> UPDATE foo SET bar = CAST(bar AS text) WHERE typeof(bar) = 'blob';
> 
> However, in two tables (revids and should_retry), it wasn't that simple,
> since the above technique would give a key uniqueness violation.
> 
> For those, I manually consulted the data and merged it appropriately. In
> the revids table, this only affected 4 packages (6tunnel aalib acct ace).

Thanks for going through that to reconstruct the data.

> Because of the above woes and because of the deadlocking issue, and some
> discussion with Jelmer on #bzr, I've reverted Storm out of lp:udd.

I'm fine with that, but we should aim to push it back in, as it is a
step to moving away from sqlite, which is one of the causes of these
issues.

> I guess we can put it back in, in time, but this time let's be really
> really sure we don't create operational issues.

How do we go about doing that. We have only the production environment
available to give us good feedback on the reliability of the code it
seems.

The revert should have happened sooner, and I would have done it
immediately if the issues hadn't takes more than 24 hours to show up in
the first place.

I am genuinely interested in ideas for handling this better though.

> With the above accomplished, the importer is now happily alive, and
> chugging through the extreme backlog from the previous malfunction at a
> concurrency level of 12 threads.

Thanks for ensuring good service.


James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD importer making a nuisance of itself with v3 source format branches

[James Westby]

On Fri, 13 Apr 2012 01:13:22 +0100, Max Bowsher <_...@maxb.eu> wrote:
> I've just had a conversation with cjwatson and slangasek on
> #ubuntu-release about the importer making a nuisance of itself by
> declaring a perfectly reasonable commit to be a collision / difference,
> and replacing it with one of its own.
> 
> The key pain point here is the .pc/ directory.
> 
> It's practically impossible to maintain a .pc/ directory checked into
> VCS without unnatural jumping through hoops. As a result, packages being
> seriously developed in bzr by humans, rather than being primarily
> imported, tend NOT to have a .pc/ :
> 
>  patches applied doesn't have to imply .pc in vcs; it's
> unfortunate that the importer took that particular decision
>  (I've been using patches-applied-in-bzr since well before the
> importer did, *without* .pc)
> 
> I think, as a short term fix, we should modify the collision-is-clean
> check to ignore the absence of a .pc directory in packager-committed
> revisions.

That sounds reasonable.

It does mean that there starts to be some differences in how the
branches should be handled, but they exist already depending on whether
the importer or a human was last to push.

I think there's more to a transition that just ignoring .pc, but it
should be considered if that's how humans prefer to deal with the
branches.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Importer stopped since Sunday?

[James Westby]

On Tue, 10 Apr 2012 01:08:25 +0100, Max Bowsher <_...@maxb.eu> wrote:
> It looks like someone stopped the UDD importer on Sunday?
> 
> James W. has a 'crontab -e' editor open from around that time.
> 
> Anyone know what's going on, and if it's safe to restart?
> 
> We're failing rather dismally at providing prompt imports at the moment.

Hi,

Firstly my apologies for not sending a mail about this at the time.

Some time on Saturday night the importer started failing hard with
sqlite contention errors. The mass-import process was stuck, so nothing
was being imported, and the cronjobs were failing causing two emails to
be sent every 5 minutes.

Given that nothing was being imported anyway, I stopped the process
until I would have more time to investigate after the long weekend.

I suspect the changes are somehow related to my storm changes, but I
don't know what the relationship is yet. It ran fine for a couple of
days, so we should be fine to restart and process the queue. It may well
fail again though.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: A huge waste of jubany diskspace...

[James Westby]

On Tue, 10 Apr 2012 01:11:07 +0100, Max Bowsher <_...@maxb.eu> wrote:
> pkg_import@jubany:/srv/package-import.canonical.com/new/tmp$ du -hsc  .
> 174G  .
> 
> Yikes!
> 
> Anyone see any reason *NOT* to rm -rf ?

Not really. It may lead to a strange failure if any imports are
currently in progress, but they will obviously be easily fixed by
requeuing.

Looking at the largest individual users they are an .orig.tar.gz and two
extracted dirs, likely indicating that something that extracts source
packages is not cleaning up correctly.

However, a quick look doesn't show anything obviously not handling
cleanup correctly, so I suspect it may just be leftovers when an import
is killed. The process isn't always going to get a chance to cleanup, so
this isn't entirely surprising.

Perhaps we should request a tmp reaper be run on that directory
periodically?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: What to do when a packaging branch is out of date

[James Westby]

On Wed, 04 Apr 2012 15:51:21 -0300, Andreas Hasenack  
wrote:
> Hi, can you take another look? It seems there is a delay again:
> 
> $ bzr branch ubuntu:landscape-client precise-already-done
> Most recent Ubuntu version: 12.04.3-0ubuntu1
> 
> 
> Packaging branch version: 12.04.2-0ubuntu1
> Packaging branch status: OUT-OF-DATE
> Branched 44 revisions.
> 
> There is an error for it in the package importer:
> http://package-import.ubuntu.com/status/landscape-client.html#2012-03-30
> 06:12:01.518169
> 
> bzrlib.errors.UnknownErrorFromSmartServer: Server sent an unexpected
> error: ('error', 'xmlrpclib.Fault', " exception: RequestExpired: request expired.'>")

Hi,

It is now up to date again.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: bzr merge-upstream: why delete and add the same unchanged file?

[James Westby]

On Thu, 29 Mar 2012 12:21:41 +0200, Jelmer Vernooij  
wrote:
> Am 29/03/12 05:14, schrieb James Westby:
> > On Tue, 20 Mar 2012 18:06:52 -0300, Andreas Hasenack 
> >  wrote:
> >> I understand they are isolated and separated branches. I thought
> >> supporting a bzr branch for the upstream branch was more of a
> >> convenience and that merge-upstream would actually just export it to a
> >> temporary tarball and then move on like if I had given it a tarball to
> >> work with, but I see now that's not the case.
> > Yeah, there is some old code in bzr-builddeb to do that, but I've
> > forgotten how to activate it now :-)
> I think I'm missing some context - are you talking about running "bzr
> merge-upstream " ? That should still work.

That will merge in the branch as well as export the tarball right?

I think Andreas was asking about a way to just export the tarball and
use it in merge-upstream without also merging the branch.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: bzr merge-upstream: why delete and add the same unchanged file?

[James Westby]

On Tue, 20 Mar 2012 18:06:52 -0300, Andreas Hasenack  
wrote:
> Yes, and I don't have commit or upload rights to
> ubuntu:landscape-client, so it will always be the first time :)

If your sponsor pushes to the branch then it will be there for the next
time that you make a change.

> I understand they are isolated and separated branches. I thought
> supporting a bzr branch for the upstream branch was more of a
> convenience and that merge-upstream would actually just export it to a
> temporary tarball and then move on like if I had given it a tarball to
> work with, but I see now that's not the case.

Yeah, there is some old code in bzr-builddeb to do that, but I've
forgotten how to activate it now :-)

> It also makes it very hard to review the changes this first time,
> right? The diff isn't helpful.

Yeah, the best thing to do I think is to fall back to "diff -Nru" of two
trees so that you diff based on paths rather than file ids. I'm not
aware of anyway to get bzr to do this.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: bzr merge-upstream: why delete and add the same unchanged file?

[James Westby]

On Tue, 20 Mar 2012 16:20:29 -0300, Andreas Hasenack  
wrote:
> Why is it removing and adding the same file? This file (and several
> others) didn't change between ubuntu:landscape-client and
> lp:landscape-client, it's exactly the same.

I'm assuming that this is the first time you've done any sort of merge
between the two?

Because they are unrelated branches from bzr's point of view, it has to
reconcile the history and file-ids. For the history it joins the two
revisions history together, which is fine. The file ids isn't so easy
though, as they can't be joined.

Therefore it replaces the ubuntu file ids with the upstream ones, which
is why you see everything as removed and added.

This is annoying, but it allows you to move forward, and only needs to
happen once.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: What to do when a packaging branch is out of date

[James Westby]

On Tue, 20 Mar 2012 10:00 -0400, Scott Kitterman  wrote:
> On Tuesday, March 20, 2012 10:56:27 AM Andreas Hasenack wrote:
> > So apparently a package update can be prepared using udd, released
> > into the archive, and the packaging branch still be out of date:
> > 
> > """
> > $ bzr branch
> > ubuntu:landscape-client landscape-client-12.04-0ubuntu1
> > Most recent Ubuntu version: 12.04-0ubuntu1
> > 
> > 
> > Packaging branch version: 11.07.1.1-0ubuntu2
> > Packaging branch status: OUT-OF-DATE
> > Branched 40 revisions.
> > $
> > """
> > 
> > It's like it was released before being committed.
> > 
> > What should I do if I want to prepare another fix? Wait? Ping someone
> > to commit and then branch again? Do it with debdiffs and not worry
> > about branches anymore?
> > 
> > I emailed ubuntu-devel-discuss and someone told me that somebody has
> > to manually commit to the udd branch, and that this is normal.
> 
> It's normal for there to be some delay.  It will (normally) be automatically 
> updated from what was uploaded.  The only way there isn't some delay is if 
> someone manually commits the branch.
> 
> I'm not sure how much delay is considered normal these days.

The logs show that this package was updated at

  Time (UTC): 2012-03-20 13:07

Approximately 14 hours after the upload.

The reason for this is that the importer was throttled a week or so ago
in order to help tracking down a service that was overloading
Launchpad. That service was found and corrected but the importer was
never turned back up to a reasonable speed. I've just done that
now. Apologies for the delay.

When the importer isn't restricted like that the branch will usually be
updated before the package has been published, so if it's been more than
an hour and it's still saying it is out of date then there is likely a
problem. This page can help in seeing what that problem is:

  http://package-import.ubuntu.com/status/

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: problem with recipe build

[James Westby]

On Fri, 2 Mar 2012 17:15:24 -0500 (EST), Scott Moser  wrote:
> I'm sure I'm doing something wrong, but not sure what.
> I'm trying to just set up a launchpad build of two branches, one 'trunk'
> that launchpad is pulling from an svn repo, and a packaging only branch.
> 
> I'm not quite sure why there would be expected to be a tag
> 'upstream-0.9.4+r4177' as that string is something that was created due to
> this build recipe.  But the working-dir has in it a directory named
> 'madwifi-0.9.4+r4177' and debian/changelog in that says:
> | madwifi (0.9.4+r4177-0ubuntu0+7) precise; urgency=low
> |
> |   * Auto build.
> |
> |  -- Scott Moser   Fri, 02 Mar 2012 16:58:43 -0500
> 
> Which seems sane to me.  The upstream source doesn't have tags named
> 'upstream-', but only 'release-'.
> 
> Help?

[...]

> bzr: ERROR: Unable to find the upstream source. Import it as tag 
> upstream-0.9.4+r4177 or build with --allow-fallback-to-native.

Your version number is saying this is a non-native package, which
requires an upstream tarball to build against.

bzr-builder expects that tarball to have been imported by bzr-builddeb,
which would have created that tag.

You can fix this by either:

  * Importing the upstream tarball
  * Changing the version number to have no "-", making it a native
version number.
  * Or build with --allow-fallback-to-native, which will ignore all of
this and build the native package despite the version number

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD breakdown when building orig.tar.gz from upstream VCS

[James Westby]

On Wed, 15 Feb 2012 08:21:37 -0500, Barry Warsaw  wrote:
> I just want to put this out there for the historical record.  I think this is
> a rare enough use case that UDD doesn't need to address, certainly not any
> time soon, if ever.  OTOH, maybe there's an easy workaround.
> 
> I was working on an NBS for the fgfs-atlas package (LP: #903225).  The
> solution was straightforward enough: upstream had all the necessary fixes in
> their CVS repository, but hadn't done a release in a long time.  I twiddled
> the packaging to build an orig.tar.gz from CVS, and the googlez helped find
> some good general packaging information on how to do this.
> 
> Unfortunately, UDD is essentially useless here.  The problem is that after
> creating the tarball from CVS, `bzr bd -S` can't be used because dpkg-source
> will complain too much about deltas between the tarball and the source tree.
> It'll warn about a lot of stuff, but then fail with some unrepresentable
> changes to source.
> 
> I worked around this by unpacking the new orig.tar.gz and `cp -a` the debian/
> directory from the precise version of the package (with my changes) over into
> the unpacked tarball.  After a few rounds of tweaking, and using `debuild -S
> -sa`, I had a debian/ that built locally, so I uploaded it and will let the
> importer (hopefully) sort out the mess.
> 
> I still did all my changes to debian/ in a source branch though, because that
> made it easier to get a diff for the linked Debian bug.
> 
> Is there's a magical udd switch or config setting that would have helped me
> keep all the changes in the source branch?  It seems like this is somewhat
> similar to the merge-upstream issue when upstream has a rather large released
> tarball delta.

What's the merge-upstream issue?

I think that using merge-upstream is what should be done here, passing
it the tarball you created. That will merge in the new code, keep the
debian directory and any changes, with conflicts if necessary. Once all
that is resolved `bzr bd -S` should just work.

Of course there may be a large gulf between theory and reality :-)

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Moving udd to django

[James Westby]

On Thu, 15 Dec 2011 08:09:00 +1300, Robert Collins  
wrote:
> On Thu, Dec 15, 2011 at 7:08 AM, James Westby  
> wrote:
> > My plan is delete add-import-jobs, and an a POST handler that gets told
> > when there are new packages to scan.
> 
> 'there is work to do now' is a classic pub-sub situation. Rather than
> a post handler, I suggest you want one of:
>  - a webhook

http://wiki.webhooks.org/w/page/13385124/FrontPage

  The concept of a WebHook is simple. A WebHook is an HTTP callback: an
  HTTP POST that occurs when something happens; a simple
  event-notification via HTTP POST.

Are you referring to a different webhook?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Moving udd to django

[James Westby]

On Wed, 14 Dec 2011 09:38:53 +0100, Vincent Ladeuil  
wrote:
> > it will be less risky to deploy changes gradually,
> 
> Only if we have tested these changes before deployment which we can't do
> for now (don't take my word for it, just look at the lp:udd
> history). Even my recent changes in this regard gave us the ability to
> *manually* test locally (and there, look at revno 555 history for cases
> I had to fix on top of our actual test suite).

Why not set up a staging instance of udd to test changes before they are
deployed to production?

> > and we are better off not forking if we can avoid it.  Eventually
> > we can split it into separate services/components.
> 
> I'm all in favor of splitting, I'm advocating the less risky (and as
> such the cheapest) way to get there.

The way I see it the same code is going to be changing either way, and
it can either do so incrementally, or in one big jump.

If everything goes well then one big jump is less work, but if things
break, and you are saying that they are very likely to, then it's easier
to deal with that incrementally, where you have fewer things to reason
about.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Moving udd to django

[James Westby]

On Wed, 14 Dec 2011 09:32:28 +0100, Vincent Ladeuil  
wrote:
> True, but I'm not saying your plan is *bad* for udd, quite the
> contrary. And yes, sharing some service to query launchpad sounds also
> like a good idea (I think I mention adding pkgme to mass_import and
> that's one of the points I had in mind). But this can achieved only by
> either running both pkgme and udd on the same host or add yet another
> layer to access the db remotely. In both cases, what I'm saying is that
> udd has already higher priorities.

My plan is delete add-import-jobs, and an a POST handler that gets told
when there are new packages to scan.

add-import-jobs would then move somewhere else (could be the same
machine)

That's perhaps what you mean by another layer. It's not a priority for
udd, but even while drafting this mail there's a conversation going on
about another service that would want the same information.

We can take what we have from udd and turn it in to an enabler for lots
of other useful things.

> > I would be worried about the risks involved with changing
> > production udd in such a large way at once. The steps I outlined
> > here would involve targeted production changes that should be much
> > easier to debug.
> 
> But if you follow these steps during your rewrite, nothing forbids
> following them when deploying once we know you've fully debugged them
> right ?

Except that any feedback from earlier steps has to be incorporated after
completing later steps, so you may have to undo lots of work.

> >> Both projects will benefit from this separation: 
> >> 
> >> - pkgme can go ahead without caring for udd needs, as long as the 
> actual
> >> code base evolve by separating old features from the new ones with a
> >> reasonable effort to make the new ones easier to integrate.
> 
> > I don't think that's true. It's not like we don't care about udd,
> > and if we make changes without regard for it
> 
> Why would you do that ?

"pkgme can go ahead without caring for udd needs"

The easiest way for us to make changes caring for udd is to have prompt
feedback about the way it is affecting udd.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Moving udd to django

[James Westby]

On Tue, 13 Dec 2011 09:11:36 +0100, Vincent Ladeuil  
wrote:
> >   3. It would also allow for starting to move udd to an SOA, or at least
> >   make it easier.
> 
> Not a concern for udd so far.

Actually I'd like to turn add_import_jobs in to a separate service, as
it could be shared between udd and pkgme, and other services that people
want to build, reducing load on Launchpad. That would be much easier if
udd was in django (or twisted, or go, or ...)

> >   4. It would be nice to have a query builder, rather than all the
> >   hand-written sql.
> 
> Not a big problem for udd so far.

Indeed, but I consider it technical debt, and it makes the code harder
to read and change.

> > 1) Install python-django in production
> >- So that it is there when it is needed.
> 
> This should be cheap. On the other hand, if newer versions are needed
> (which would be known during the dev/test phase) this will also be
> useless.

Coding to django 1.1 (lucid) is not too restrictive. IS can also run
django 1.3 if absolutely required.

> > What do you think? Is this worth doing?
> 
> Frankly ? This certainly sounds like a very good plan.
> 
> I also think udd doesn't need any of that in the short term.
> 
> Now, if you do it for pkgme, we will certainly be interested in
> cannibalizing part or all of it.
> 
> In the mean time, the less disruptive changes we do to udd, the more we
> can focus on the import failures which is where the paint points are.

I do think dealing with import failures is good, but I'm not sure the
approach you advocate is the right one.

> What I mean is that forcing the use of the same code base for two
> projects with different goals sounds like a sure way to trigger failures
> for the other project without benefits for the project needing the new
> features. We've already encountered such issues at a small scale and
> until the test suite become rock-solid, I fail to see while we won't
> encounter new issues.
> 
> I.e.: It sounds easier to separate common parts from specific parts in
> the actual udd code base while starting a *new* project than doing so in
> the udd code base that is already in production. Once those parts are
> clearly separated, the package importer can look at integrating the new
> and well-separated udd.

I would be worried about the risks involved with changing production udd
in such a large way at once. The steps I outlined here would involve
targeted production changes that should be much easier to debug.

> Both projects will benefit from this separation: 
> 
> - pkgme can go ahead without caring for udd needs, as long as the actual
>   code base evolve by separating old features from the new ones with a
>   reasonable effort to make the new ones easier to integrate.

I don't think that's true. It's not like we don't care about udd, and if
we make changes without regard for it then it's likely that either lots
of changes will be required to support udd again, or udd will never used
the new codebase.

> - udd won't have to integrate incremental steps that bring no added
>   value but still cost deployment/debugging time or do so *only* when a
>   tangible benefit emerge *and* addresses issues.

If you believe this then we can go away and ignore udd, but I fear it
will be more work and more risk in the end.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Moving udd to django

[James Westby]

On Sun, 11 Dec 2011 06:38:25 +1300, Robert Collins  
wrote:
> It might be interesting - as a thought experiment if nothing else - to
> consider failures a form of crash and upload them to a crash database
> rather than processing them inside udd - e.g. toss them out over amqp.

The current design is that it also uses them to avoid acting on that
package until there is manual intervention.

So it may be interesting to use oopses to communicate the problems and
do analysis of them, but it won't avoid storing the data as well unless
that model is changed.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Moving udd to django

[James Westby]

On Sat, 10 Dec 2011 17:52:40 +1300, Robert Collins  
wrote:
> On Sat, Dec 10, 2011 at 3:10 PM, James Westby  
> wrote:
> > Hi,
> >
> > I think there are a few reasons that we should consider moving udd to
> > django (more on what this actually means later.)
> 
> I'm curious what data udd stores.

There are two sets.

There's the bookkeeping data for running import-package each time a
package is uploaded to Debian or Ubuntu, keeping track of failures, etc.

Then there's the bookkeeping data for import-package itself, about what
it thinks the state of bzr is etc.

These should be treated differently, with the former the one that is
more important for the goals laid out here.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Moving udd to django

[James Westby]

Hi,

I think there are a few reasons that we should consider moving udd to
django (more on what this actually means later.)

  1. We want to have remote commands so that we don't have to ssh in and
  run scripts for common tasks

  2. There are scripts in cron that generate html pages that are
  served. It would be nice to have these generated on request so that
  there is no latency.

  3. It would also allow for starting to move udd to an SOA, or at least
  make it easier.

  4. It would be nice to have a query builder, rather than all the
  hand-written sql.

So, what exactly do I mean by moving to django?

  * Moving all database access to go through django's ORM.

  * Moving all HTML generation to be done in django views.

This has fairly wide-ranging consequences though, and obviously needs to
be broken down in to manageable steps.

Here's my idea for what those steps should be.

1) Install python-django in production
   - So that it is there when it is needed.

2) Create a django project in the udd codebase.
   - Providing the skeleton to use, but not actually changing any
   behaviour

3) Write django management commands for each current script that call
the same code.
   - This gets them running in the django env, but without changing
   anything else.
   - At this point we'll want to deploy, and change the init script and
   cronjobs to run the new scripts.

4) Configure the django project to use multiple databases corresponding
to the existing database.
   - At this point we'll want to deploy and run some sanity checks to
   make sure that django can read the dbs.

5) Pick a db table and write a django model to match. Convert users of
that table to call through the query builder. Repeat.
   - I think this will be ok table-at-a-time, but we should obviously
   test well.

6) At this point we'll be using django just as an ORM, but can turn on
the admin site if we want.

7) We can also start to write django views to replace the
html-generating scripts. Once they are written we can deploy and request
that apache actually run django and serve the views, and then turn of
the scripts.

8) Then we'll be able to start adding forms to the views to accomplish
common tasks.

That's a fairly long road, but it is incremental, and the changed code
deployed at each point can be fairly small.

What do you think? Is this worth doing?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: bzr bd -S --package-merge

[James Westby]

Hi,

I think this is likely to be

  https://bugs.launchpad.net/ubuntu/+source/bzr-builddeb/+bug/876888

Thanks,

James


On Thu, 27 Oct 2011 07:39:09 -0400, Barry Warsaw  wrote:
> How exactly does --package-merge calculate the version it passes to
> dpkg-genchanges -v?  I'm wondering because it doesn't seem to be doing what
> I'd expect it to do, and I'm reluctant to use it.
> 
> Latest case in point: I'm merging boost1.46 from testing into precise, so I do
> the following:
> 
> $ bzr branch ubuntu:boost1.46 precise
> $ cd precise
> $ bzr merge-package debianlp:boost1.46
> Most recent Debian version: MISSING
> Text conflict in debian/control
> 1 conflicts encountered.
> The merge resulted in 1 conflicts. Please resolve these and commit the 
> changes with "bzr commit".
> 
> 
> 
> $ bzr diff debian/changelog
> === modified file 'debian/changelog'
> --- debian/changelog  2011-06-03 20:28:58 +
> +++ debian/changelog  2011-10-27 01:57:00 +
> @@ -1,3 +1,35 @@
> +boost1.46 (1.46.1-7ubuntu1) precise; urgency=low
> +
> +  * Merge with Debian testing.  Remaining Ubuntu changes:
> +- Detect gcc atomic intrinsics, needed for arm spinlock (LP: #513721)
> +- Drop libboost-mpi, libboost-mpi-python, and libboost-graph-parallel 
> (and
> +  related -dev packages): we don't want to pull all of the mpi packages
> +  into main.  These are provided in a separate boost-mpi-source1.46
> +  package
> +- Drop libboost1.46-all-dev and provide from boost-mpi-source1.46
> +- Adjust debian/rules and debian/control
> +
> + -- Barry Warsaw   Wed, 26 Oct 2011 21:56:45 -0400
> +
> +boost1.46 (1.46.1-7) unstable; urgency=low
> +
> +  * control: Fix ungrammatical description for iostreams packages.
> +Closes: #633865.
> +  
> +  * rules: Dump boostrap log file if bootstrapping fails.
> +  
> +  * libboost-doc.README.Debian: Remove reference to packages bjam and
> +boost-build; now only need boostX.YZ-dev.  Closes: #630529.
> +
> + -- Steve M. Robbins   Wed, 17 Aug 2011 23:18:52 -0500
> +
> +boost1.46 (1.46.1-6) unstable; urgency=low
> +
> +  * control(libboost-mpi-python1.46.1, libboost-python1.46.1): Suggests a
> +python interpreter.  Closes: #620775.
> +
> + -- Steve M. Robbins   Sun, 12 Jun 2011 00:37:42 -0500
> +
>  boost1.46 (1.46.1-5ubuntu2) oneiric; urgency=low
>  
>* No change rebuild to drop Python 3.1 support.
> 
> 
> 
> $ bzr bd -S --package-merge
> 
> If I now look at boost1.46_1.46.1-7ubuntu1_source.changes I see *all* the
> version entries from 1.46.1-7ubuntu1 all the way back to 1.21.1-1, the first
> in the changelog.
> 
> If I drop the --package-merge option, I get a much trimmer changes file, with
> just the last changelog entry in it.  That's not right either though.  I'd
> expect to see entries for -6, -7 and -7ubuntu1.
> 
> I've pushed the branch to lp:~barry/ubuntu/precise/boost1.46/debian-merge in
> case you'd like to take a look at see what's going on.  I'll hold off on
> uploading for a bit in case there's something obvious going on.
> 
> Cheers,
> -Barry
Attachment: signature.asc (application/pgp-signature)
> -- 
> ubuntu-distributed-devel mailing list
> ubuntu-distributed-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Update on package import failures involving non-ascii filenames

[James Westby]

Hi Martin,

Thanks to you and the others that worked on this, it's great to have
some progress on this long-standing problem.

Thanks,

James


On Wed, 26 Oct 2011 09:56:58 +0100, Martin Packman 
 wrote:
> Yesterday we deployed a fix for  to how the
> package importer handles non-ascii filenames. Some packages have now
> been successfully imported, and better feedback, including the problem
> filename, is now given where there are still errors.
> 
> 
> Jonathan Riddell correctly noted in the bug that some packages have
> filenames that aren't UTF-8, but the issue was also preventing some
> that did have names the importer could decode from succeeding, such
> as:
> 
> 
> 
> Several of the remaining failures look similarly to be from test
> suites, given the full filenames now listed in the error message. It's
> encouraging that programs care about this and want to test they handle
> non-ascii filenames correctly, though generating them at test time
> would be a better approach. :)
> 
> 
> The overall results are:
> * Before there were 69 package with failures across four similar
> UnicodeDecodeError signatures.
> * Now there are 36 across three BadFilenameEncoding signatures, plus 3
> new failures with InvalidNormalization, and 1 other issue exposed.
> 
> Filenames in encodings other than UTF-8:
> 
> 
> 
> 
> Many of these are filenames in legacy single byte encodings, some are
> double byte, and some are clearly junk. A complete fix needs
>  in bzr core resolving, but hacking some fallback
> into the importer would be possible if it was deemed worthwhile.
> 
> Packages with filenames that are not in unicode normal form 'NFC':
> 
> 
> This is a symptom of the incomplete unicode normalisation code in bzr,
> which OSX  also runs into
> 
> Funky symlink issue:
> 
> 
> Probably a bug in the bzr-builddeb (copied from bzrtools) import_dir function.
> 
> 
> There were a few wrinkles in getting this deployed. Some existing
> fallout from recent lp:udd changes needed tackling first, and then
> using the latest lp:bzr-builddeb broke a few things that needed
> interface updates on the udd side. Most excitingly the first
> normalisation failure caused a loop that lead to infinitely repeating
> tracebacks. Fortunately we were monitoring the process at the time so
> could kill it and fix the problem.
> 
> Martin
> 
> -- 
> ubuntu-distributed-devel mailing list
> ubuntu-distributed-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Collision branches

[James Westby]

On Thu, 6 Oct 2011 15:27:54 +0100, Jonathan Riddell  wrote:
> On Wed, Oct 05, 2011 at 12:24:44PM -0400, James Westby wrote:
> > In order to get some data on this I just looked back at 45 of these from
> > the last month, and found:
> > 
> >   9 that looked real, or at least feasible
> >   2 that were caused by updates to .po files
> >   4 that were caused by automatically generated debian/control files
> >   1 caused by updates to config.{guess,sub}
> >   29 caused by quilt interaction
> > 
> > So 64% of these were caused by quilt interactions, and all the other
> > spurious ones were less than 50% of what's left, indicating that quilt
> > is the place to focus our efforts to make these merge proposals have
> > much more signal than noise.
> 
> I've looked through a bunch and many of them seem to be conflicts
> caused by adding a file.  The version uploaded then adds the file to
> its own brancha and bzr considers it a conflict (even though the files
> are identical), I'd have thought the importer would have some magic to
> detect this?

I think this is discussed in

https://bugs.launchpad.net/udd/+bug/806940

It's very confusing as there are a couple of hidden things going on:

  1) The importer isn't using the diff that you see on the MP to decide
  whether there is a collision, so sometimes it does odd things.

  2) When doing the re-import when it detects a collision it doesn't
  have any reference to the moved-aside branch, so any added files will
  get different file ids, leading to the conflicts that you see.

The second should actually be pretty easy to fix, passing the tree from
the side branch as a file_ids_from when the import is done. The former
could be mitigated in several ways, and may be the wrong thing to do
conceptually.

> I also found a strange case in python3-defaults where there is nothing
> to merge, spooky.

That's an artefact of 1) above, there is a diff between the branches,
but it leads to no changes when merged.

> There's also several such as checkbox, openssl, gnome-color-manager where I 
> get an error on checking out the moved-aside branch
> 
> >bzr branch lp:~ubuntu-branches/ubuntu/oneiric/openssl/oneiric-201110041452
> bzr: ERROR: Revision {cjwat...@canonical.com-20111004123628-4ybe8782rytyb0dc} 
> not present in 
> "Graph(StackedParentsProvider(bzrlib.repository._LazyListJoin(([CachingParentsProvider(None)],
>  []".

Yeah, there's a few of these, there's clearly a bug somewhere, but I'm
not sure of the cause.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Collision branches

[James Westby]

On Thu, 8 Sep 2011 14:51:44 +1000, Martin Pool  wrote:
> I looked into a few of them and they weren't all clearly due to quilt
> problems, but perhaps most of them are (or I didn't understand the
> cause from a glance.)

In order to get some data on this I just looked back at 45 of these from
the last month, and found:

  9 that looked real, or at least feasible
  2 that were caused by updates to .po files
  4 that were caused by automatically generated debian/control files
  1 caused by updates to config.{guess,sub}
  29 caused by quilt interaction

So 64% of these were caused by quilt interactions, and all the other
spurious ones were less than 50% of what's left, indicating that quilt
is the place to focus our efforts to make these merge proposals have
much more signal than noise.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Our biweekly meetings

[James Westby]

On Tue, 6 Sep 2011 08:57:03 -0400, Barry Warsaw  wrote:
> At our last meeting, we talked about starting up the UDD bi-weekly meetings
> again this week (Wednesday Sept 7 @ 1100 UTC).  However, both Martin and I are
> pretty tired of the old format, so let's think about how we can restructure
> the meeting to be most effective for everyone.  If we really don't think we
> *need* them anymore, that's fine too.
> 
> What would *you* like to get out of these meetings?

Hi,

Thanks for starting the discussion Barry.

The reason that I haven't been at the last few meetings is that I'm not
someone who is naturally awake at 7am, so I've tended to miss them. I
can make a more consistent effort if desired. To be clear, I don't
resent the time, I realise we are in a near-impossible situation, it's
just hard to argue with my sleeping self sometimes :-)

I'm not sure that it's directly related to the meetings, but I wonder if
everyone is in a similar position to me, where they aren't sure where we
are on the roadmap we outlined at UDS, and hence what would be next to
work on. Indeed whether that roadmap is still the right order in which
to be doing things.

I agree that highlighting success would be good, and a roadmap at least
defines what success is, but to know if you are progressing you need to
keep track of where you are.

In addition I don't know if we aren't making progress in some area we
could be because it isn't clear. I would hate to think that I was
holding things (or even worse: people) up because I wasn't doing
something.

If I'm not the only one, then we could look at ways of fixing that,
which may involve the meeting. If I am the only one then I'll work out
what I need to do to stay up to date, and maybe that involves setting my
alarm clock :-)

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Collision branches

[James Westby]

On Thu, 8 Sep 2011 14:51:44 +1000, Martin Pool  wrote:
> I looked into a few of them and they weren't all clearly due to quilt
> problems, but perhaps most of them are (or I didn't understand the
> cause from a glance.)

Unfortunately it's necessary to look at the importer log for the package
to see why the importer felt it necessary to file the merge proposal.

This is because the diff you are seeing is the result of merging the
branch in to the new tip, but the importer decides based on the diff of
the two revisions. These frequently differ and mean that looking at the
merge diff doesn't tell you why the importer chose to do that
(particularly if the merge diff is empty.)

Perhaps that's the wrong check, but that's the way it is currently.

> I think we can handle this without blocking on looms by doing a
> smarter merge that unapplies and reapplies the patches.  There is some
> work towards this in eg  which
> Jelmer is working on - we may need extra work to hook it up into the
> udd importer.

Would it also need to be used by LP when generating the merge diffs?

> What we should probably do next is look at the merge proposals that
> were filed and work out whether each one
> 
> - is a real conflict in a sensible form
> - is not a real conflict and shouldn't be generated at all (some have zero 
> diff)
> - could be either avoided or better presented by smarter quilt
> handling or something else

That would be good. Is someone able to look at this analysis?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Collision branches

[James Westby]

Hi,

Thanks for fixing the bugs that were preventing merge proposals for
getting filed for collisions.

This had led to a surge in the number of such merge proposals. This is
mainly due to a backlog, but there have been 10 or so in the two days
since.

You can see the extent of this by searching for ubuntu-branches at
http://reports.qa.ubuntu.com/reports/sponsoring/index.html

Colin has valiantly reviewed some of them (maybe half, thanks Colin,)
and has found that in none of the cases so far were the collisions
"real" in the sense that someone pushed and someone else uploaded
something different.

There was one case at

  
https://code.launchpad.net/~ubuntu-branches/ubuntu/oneiric/ibus/oneiric-201108121834/+merge/73916

which seems to indicate a bug though.

>From my previous experience going through these merge proposals the
majority of issues will be caused by the representation of quilt
in the branch.

Can the Bazaar team do something to stop this influx of merge proposals
that must be sorted, leaving just "real" ones? Does this have to involve
work on looms?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Terminating the append_revisions_only experiment, for now

[James Westby]

On Tue, 23 Aug 2011 09:41:24 +0200, vila  wrote:
> This is also very close to my feelings and I like to add that we should
> *really* write tests to capture them and make sure we don't regress
> again in the future.
> 
> Overall, I feel we have far too much failures when landing *good* stuff
> *because* we lack a proper way to test. It's a long known issue that we
> miss a launchpad test server but *not* having tests to guard against
> regressions is clearly a path we don't want to pursue as demonstrated
> here (IMNSHO).
> 
> Whether we address that by using staging or any sort of local launchpad
> test server is open to discussion but the sooner we start this
> discussion the better.

I think there are tests that could be written that would have caught
some of these issues that don't require a Launchpad instance to run.

There are problems when the append-revisions-only is set on the local
branches for instance.

Whether the code is structured such that you can test those code paths
without a Launchpad instance is a different matter though :-)

I'm not saying that we shouldn't work out a way to test against
Launchpad, just that there may be useful things that can be done without
solving that problem.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: The append-revisions-only can of worms

[James Westby]

On Tue, 19 Jul 2011 00:06:33 +0100, Max Bowsher <_...@maxb.eu> wrote:
> Well... The logic *should* be to have imported into the Debian branch
> first and then merged into the Ubuntu branch, such that this situation
> never arises.

That's true.

It still seems to be the wrong behaviour in any case?

> Perhaps we should investigate whether Launchpad's Debian importer could
> run more frequently, or if the UDD importer could query Debian and defer
> Ubuntu imports that appear to be syncs of not-yet-known Debian versions?

They sounds like useful things to do.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: The append-revisions-only can of worms

[James Westby]

On Sat, 16 Jul 2011 11:17:33 +0100, Max Bowsher <_...@maxb.eu> wrote:
> I've tracked down a cause of some of the AppendRevisionsOnlyViolation
> failures.
> 
> When a package is uploaded to Debian and synced to Ubuntu, there's a
> race condition between Launchpad's Debian metadata importer and the UDD
> package importer.
> 
> If the UDD package importer tries to import the package in the window
> when Launchpad knows of the Ubuntu publication but not of the Debian
> publication, then the new package version gets imported first into the
> Ubuntu branch.
> 
> Later, when the Debian publication becomes visible, Launchpad sees that
> this particular (package,version) has already been imported and
> tries to pull the Ubuntu branch into the Debian sid branch.
> 
> And of course, if the package has *ever* had an Ubuntu specific upload,
> that's an AppendRevisionsOnlyViolation.
> 
> So we've actually found a bug here :-(

Yeah, I think the logic should be to do a merge taking all the content
from Ubuntu, with the parents being (debian, ubuntu)?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: RFC: Minor collisions handling, both manual and automatic

[James Westby]

On Wed, 13 Jul 2011 08:57:19 +0100, Max Bowsher <_...@maxb.eu> wrote:
> I just dealt with a requeue/zap-revids for bootchart to get that import
> going again.
> 
> The importer created various extra and fairly pointless revisions in the
> lucid and maverick branches... all to remove a vim backup file which was
> included in the human-committed branch but not in the archive.
> 
> I think we should do two things to deal with minor collisions like this:
> 
> 1) Automation
> =
> 
> In the collision detection code, we should have heuristics for pathnames
> which are safe to disregard and still treat a collision as clean. I'm
> thinking of:
> 
> .*.swp (vim backup)
> 
> .pc
> .pc/.version
> .pc/.quilt_series
> .pc/.quilt_patches
> 
> (The creation or removal of an initialized quilt control directory
> containing no actual content is pretty harmless)
> 
> I'm sure there are more.

That sounds ok to me. Provided the branch remains usable I don't see why
these would cause a problem.

> 2) Manual override
> ==
> 
> We should have a new tool that can be used to forcible add/update rows
> in the importer's meta.db revids table, by package, suite and
> revision-id, so that jubany operators can placate the importer about
> one-off oddities determined to be harmless on human inspection.

Agreed. I just tended to use the big-but-easy hammer of requeue --full.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Testing/feedback wanted: Using dpkg-mergechangelogs in bzr-builddeb

[James Westby]

On Fri, 8 Jul 2011 09:16:53 +1000, Andrew Bennetts 
 wrote:
> James Westby wrote:
> […]
> > bzrlib.plugins.builddeb.tests.test_merge_changelog.TestMergeChangelog.test_not_valid_changelog
> > 
> > This is testing what happens with an invalid changelog. I'm not sure
> > what the effect of returning "not_applicable" versus "success" with the
> > invalid text would be. I don't think it's a particularly important case
> > though.
> 
> “not_applicable” gives other merge hooks, including the default one
> (which is always applicable, but might conflict), a chance to try the
> merge.  The other outcomes (“success”, “conflicted” and “deleted”) are
> final result for the merge.
> 
> So that test is saying “if it's not a valid changelog, give up and let
> the default logic (or other plugins) merge it.”  I agree that that
> doesn't seem particularly important.  The main thing I think is that
> invalid changelogs shouldn't cause tracebacks or other completely
> unreasonable output (and *perhaps* ideally also produce a gentle warning).
> If so, “success” vs. “not_applicable” are both fine, assuming the
> apparent success isn't actually total garbage.

Yeah, sounds about right, given that it only acts on files called
"debian/changelog."

> another…” line.  BASE → THIS just adds “But more is better”.  So that
> output looks like a sensible and expected combination of those changes:
> it's what I'd do if I had to merge those by hand.  But as a VCS
> developer perhaps my sense of what's expected is skewed ;)
> 
> Given that one side does discard a line, do you still find it odd that
> the merge also discards that line?

I'm not sure now. I think I read it too quickly the first time, but your
explanation about sounds pretty convincing.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Testing/feedback wanted: Using dpkg-mergechangelogs in bzr-builddeb

[James Westby]

On Wed, 6 Jul 2011 17:27:34 +1000, Andrew Bennetts 
 wrote:
>  proposes replacing the
> existing changelog merge logic with dpkg-mergechangelogs' implementation.  
> I've
> done that in  (simply by
> shelling out to it), but it changes the existing output enough that the 
> existing
> tests fail.
> 
> So here's where I'd like help: tell me, is the new output better (or at least 
> no
> worse) for you?
> 
> If so, we can adjust the tests accordingly and land it.  If not, we can try to
> figure out what to do instead.

Hi,

I had three failures:

bzrlib.plugins.builddeb.tests.test_merge_changelog.TestMergeChangelog.test_unsorted

This one is asserting that it will sort the changelog, which I think is
the root of why the bug was filed, so the failure would be expected.

bzrlib.plugins.builddeb.tests.test_merge_changelog.TestMergeChangelog.test_not_valid_changelog

This is testing what happens with an invalid changelog. I'm not sure
what the effect of returning "not_applicable" versus "success" with the
invalid text would be. I don't think it's a particularly important case
though.

bzrlib.plugins.builddeb.tests.test_merge_changelog.TestMergeChangelog.test_3way_conflicted

Getting success back in this case seems wrong.

The text it returns is:

 psuedo-prog (1.1.1-2) unstable; urgency=low

   * New upstream release.
   * Yet another content for 1.1.1-2
   * But more is better

  -- Joe Foo   Thu, 28 Jan 2010 10:45:44 +

and I'm not sure that's correct as it's discarding a line.

If it's the desired behaviour of dpkg-mergechangelog then it may be what
we want, but it seems odd to me.

Thanks,

James

 

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Wondering how to fix up a category of import failure

[James Westby]

On Thu, 23 Jun 2011 10:48:05 +0200, John Arbash Meinel  
wrote:
> I personally like C when reasonable. However reasonable means that
> nobody else has started working on the project (since creating new
> revisions will break their existing changes). Otherwise, I would go for
> B. branch-nick doesn't mean all that much.

I agree that B is likely the best option here.

Thanks,

James



-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: live-build import failure

[James Westby]

On Sat, 11 Jun 2011 13:26:05 +0100, Max Bowsher <_...@maxb.eu> wrote:
> On 11/06/11 00:17, Colin Watson wrote:
> > Hi,
> > 
> > I'm working on a set of modifications to the live-build package, and am
> > almost ready to upload them, but I noticed that lp:ubuntu/live-build
> > doesn't exist.  Could somebody look at the import failure?
> > http://package-import.ubuntu.com/status/live-build.html currently says:
> ...
> >   OSError: [Errno 2] No such file or directory: 
> > '/srv/package-import.canonical.com/new/updates/live-build/tmpY9PcJn/upstream/.bzr/checkout/limbo/new-75'
> 
> 
> I've done some investigation and it appears that the problem is that
> TreeTransform has multiple bugs where a versioned symlink to a directory
> is replaced with an actual directory, containing any children in common
> with those accessible via the former symlink.

Are there any bugs filed against bzr about this that you know of?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: [Launchpad-dev] Removing ubuntu-branches and ubuntu-techboard celebrities

[James Westby]

On Mon, 30 May 2011 17:28:14 -0400, "Francis J. Lacoste" 
 wrote:
> For official package branch, this change would affect James Westby as he's 
> the 
> only member of ubuntu branches that is not part of the technical board.
> 
> Maybe what's needed is to fix 
> https://bugs.launchpad.net/launchpad/+bug/365098 
> and allow anyone who can upload the package to set the official link. The 
> question then becomes, do we need to fix that bug before proceeding (in other 
> word, would there be unwanted fall-outs from restricting this to the current 
> distribution owners.)

Yes, that bug needs to be fixed before removing the celebrity, or some
other way of keeping the importer working needs to be found.

The importer currently runs with my credentials in order to be able to
do all of the things that it needs to do (handily I am a core-dev and a
member of ~ubuntu-branches, which equates to full permission to do
everything that it needs to do.) We want to change this anyway, but
haven't ever got to doing it.

The importer frequently sets official branches (e.g. when a new package
is uploaded to Ubuntu,) and so this needs to continue to work, but I
don't feel strongly about the exact mechanics.

To put it another way the importer needs to have

  * ability to push to every official branch for Debian and Ubuntu (to
keep them up to date)
  * ability to set the official branches for Debian and Ubuntu (to add
them when something new appears, e.g. new package, first security
update for a package to a particular release)

and it uses all of those permissions regularly, so any temporary removal
would disrupt its operation.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Breaking up the importer's huge icommon.py file

[James Westby]

On Sat, 21 May 2011 18:52:20 +0100, Max Bowsher  wrote:
> A huge amount of the UDD importer's interesting code is in one file,
> icommon.py.
>
> I'd like to submit a series of changes to break it up such that only the
> most common bits of code remain there.

Good idea.

> Things I'm thinking of:
> 
> 1) Move code in icommon.py that is uniquely invoked by a single
> executable component of the importer out to that component:

I think I would prefer it if they moved to their own modules...

> e.g.
> class ImportList, class PackageToImport --> import_package.py
> 
> class SubprocessMonitor, class ThreadDriver --> mass_import.py

Especially things like this.

I don't have a strong preference though.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Import layout of Quilt v3 packages

[James Westby]

On Tue, 08 Feb 2011 08:00:25 +, Max Bowsher  wrote:
> Therefore, what about checking in the patched code, without any quilt
> metadata (.pc dir) but with a flag file that triggers bzr-builddeb to
> write out the appropriate metadata whenever a working tree is built for
> such a branch?
> 
> (Writing out the metadata would consist of copying the series file to
> .pc/applied-patches, and reverse-applying each patch in reverse order,
> stashing the resultant modified file in .pc// for
> each patch)

This would work for checkout. What are the implications for merge etc?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Import layout of Quilt v3 packages

[James Westby]

On Fri, 04 Feb 2011 10:22:45 -0600, John Arbash Meinel  
wrote:
> Now that I've described the state as I understand it, here are some
> questions.
> 
> 1) As I understand it, most people are in favor of *not* versioning the
>.pc directory. So that when you do "bzr branch lp:ubuntu/foo" you'll
>get a tree with:
>  a) debian/patches/* still intact
>  b) Those patches applied to the working tree
>  c) no .pc directory

Well, I'm in favour of *not* versioning the .pc directory, however I
don't immediately follow to your a, b and c.

I am convinced that "check out and immediately start hacking" is a
property that we want.

> 2) Doesn't that mean that you have to rebuild the .pc directory right
>after you get the checkout? Wouldn't it be easier to get it from the
>beginning? Or is it just introducing an extra place to get conflicts
>when trying to merge.

Yes, you would have to rebuild the .pc directory right after the
checkout. Yes it would be easier to get it right from the beginning.

>That said, if you did end up merging 2 branches that didn't have .pc
>directories, how would you get the .pc directory rebuilt? (Since
>presumably the patch series need to be combined in some way, and
>modifications to the source tree also need to be replicated into the
>patches themselves.)

There is a similar problem with the current state of affairs, where
merging two branches attempts to merge everything in .pc and doesn't
leave you in a very good state.

> All this may change again if we switch the importer to use looms, since
> then you can do stuff like merge the patches one-by-one up the stack
> until you get to the top, without having to deal with conflicts in .diff
> format.

Exactly.

I think that there may be a middle ground, if we can separate
storage from presentation to the user. I don't know how that would work
without some pretty major changes though. Maybe pursuing looms is the
correct way to go.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Making bzr commit more like debcommit

[James Westby]

On Mon, 24 Jan 2011 10:54:26 -0500, Barry Warsaw  wrote:
> We have 'bzr commit' and we have 'debcommit'.  Currently, the UDD guidelines
> talk about both, but for consistency, I'd like to standardize on recommending
> 'bzr commit'.  One feature that debcommit has:
> 
> DEBCOMMIT(1) DEBCOMMIT(1)
> 
> NAME
>debcommit - commit changes to a package
> 
> ...
> 
> VCS SPECIFIC FEATURES
> ...
>bzr If the changelog entry used for the commit message closes any bugs
>then --fixes options to "bzr commit" will be generated to
>associate the revision and the bugs.
> 
> I know we can just do 'bzr commit --fixes lp:12345', though I often forget to
> do that until after my last commit, so I tend to add --unchanged, which leads
> to an empty revision.
> 
> What do you think about making 'bzr commit' act more like debcommit when
> there's a new debian/changelog entry?

I have always wanted to do this. However, none of the current bzr hook
points allow for this, except for overriding the "commit" command, which
is why I never got to it.

So, +1.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Battle royale: bzr vs. quilt

[James Westby]

On Mon, 20 Dec 2010 07:06:05 -0500, Barry Warsaw  wrote:
> On Dec 19, 2010, at 09:57 PM, James Westby wrote:
> 
> >I meant that it is expected to submit it with the change sitting in a
> >quilt patch and in the tree.
> 
> I found that tricky to accomplish.  I needed a 'quilt pop' to get the quilt
> patch to show up

Was a "quilt refresh" not enough?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Battle royale: bzr vs. quilt

[James Westby]

On Tue, 7 Dec 2010 15:17:43 -0500, Barry Warsaw  wrote:
> >> So now what do I do?  I've got the change sitting in a quilt patch and in 
> >> my
> >> tree.  I clearly can't submit a merge proposal in this state.
> >
> >That is the expected thing to submit as it maintains the above invariant.
> 
> Maybe I misunderstand.  Do you mean you recommend against reverting the
> non-debian directory changes before submitting the merge proposal?  That would
> be counter to how I would work out the patch using apt-get source - I'd end up
> pushing the changes back into quilt and that would revert the in-tree changes,
> right?  Also, if I were reviewing an mp with the changes both in quilt and in
> tree, I'd complain about it.

I meant that it is expected to submit it with the change sitting in a
quilt patch and in the tree.

> (Ideally I think the fact that quilt is there would be hidden from you, and
> you'd just see the in-tree changes, but we've had that discussion before, so
> let's ignore that for now.)

This is why I believe looms are a high priority.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Battle royale: bzr vs. quilt

[James Westby]

On Tue, 7 Dec 2010 11:18:03 -0500, Barry Warsaw  wrote:
> At this point notice that the .pc directory is under version control.  We've
> talked about this before, but it is currently deliberate that the package
> importer includes the .pc directory for quilt in bzr.
> 
> quilt push -a
> 
> I would have expected at this point that a `bzr stat` would show lots of
> modifications to the source tree, but in fact, it shows nothing.  quilt
> definitely reports that all the patches in the series have been applied, but
> if so, where are they?
> 
> quilt pop -a
> 
> This is supposed to un-apply all the patches so that "the source returns to
> the downloaded condition" [http://wiki.debian.org/UsingQuilt].  I would expect
> at this point that a `bzr stat` would show no changes, but in fact it shows
> tons of removes from the .pc directory and modifications to all the source
> tree files.  If I look at `bzr diff`, it does indeed seem like the patches
> have been un-applied.
> 
> So it seems like the source branch already has the effects of `quilt push -a`
> applied.

Correct.

We used the maxim "the branch should contain the same as you get with
apt-get source", which in the case of quilt v3 packages is patches applied.

> So now what do I do?  I've got the change sitting in a quilt patch and in my
> tree.  I clearly can't submit a merge proposal in this state.

That is the expected thing to submit as it maintains the above invariant.

> Does the above make sense?  Any clarifications on what's going on under the
> hood?  Any suggestions to make things with udd and quilt go more smoothly?

We need looms or something to maintain the spirit of the maxim (one of
the motivators of quilt v3 was to give people a tree with patches
applied so that they can just hack, we want to mirror that), while still
being able to produce sensible diffs for Debian.

I think fixing this is urgent, as the current mandated workflow is
opposite to people's aesthetic sense, so they will never be happy with
it, and so won't embrace udd. As the proportion of quilt v3 packages
increase, so will the proportion of people using UDD who will encounter
this and likely dislike what they find.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD survey results

[James Westby]

Hi Martin,

Thanks for running and summarising the survey, very interesting results.

On Thu, 18 Nov 2010 18:23:56 +1100, Martin Pool  wrote:
> * "The patch format from bzr is awkward" - I'm not sure what this
> means; maybe that it is not smart about debdiff stuff

I believe this is that they don't like bundles. They want "git
format-patch/git am" as the way to send patches around.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Please check my thinking on bug 646979

[James Westby]

On Tue, 05 Oct 2010 09:16:36 -0500, John Arbash Meinel  
wrote:
> I've been thinking about it, and I'm pretty confident that what you are
> trying to do is inherently "criss-cross". Specifically consider a
> semi-ideal case:
> 
>  upstream
> 
>   A   release 2.0
>   |
>   B   release 2.1
>   |
>   C   release 3.0
> 
> with debian doing:
> 
>   A
>   |\
>   B d   deb 2.0-1
>   | |
>   C |
>\|
> e   deb 3.0-1
> 
> And ubuntu doing:
> 
>   A
>   |\
>   | d deb 2.0-1
>   | |
>   B u   ubuntu 2.0-1ubuntu1
>\|
> v   ubuntu 2.1-1ubuntu1
> 
> 
> These graphs are, inherently, criss-cross, because you don't have a
> single point of convergence. Both 'ubuntu' and 'debian' make new changes
> vs the upstream, and never does one fully supersede the other. ubuntu is
> constantly pulling from debian, but debian *never* pulls back from ubuntu.

This will be the case for the vast majority of branches for at least the
medium term I think.

> Let's draw it combined:
>  A
>  |\
>  | d
>  | |\
>  B | u
>  |\: |
>  | \ |
>  | :\|
>  C | v
>   \|
>e
> 
> (and that is with the 'ideal' case that C is actually a descendant of B
> and not A, though if it isn't you can usually fake it:
>   A
>   |\
>   B C
>   |/
>   C'
> )

This "faking" is exactly what we are doing.

> Anyway, in the above graph, B and d are competing as common ancestors.
> If you were to strictly use d as the base, then you would get the
> changes from d => e, but that would include the A=>B changes. (d does
> not include B, so those changes would try to be 'replayed')
> 
> If you used B as the base, then you would be trying to merge in the 'd'
> changes a second time.
> 
> Now, what matters is what is explicitly changed by each revision.
> Because if "d - A" is strictly adding debian/, and not touching the
> actual content, then there are likely no conflicts in a per-file sense.
> 
> That is what '--weave' should be able to handle, since it does per-file
> merging.

Would '--weave' perform at least as well if "d - A" doesn't have that
restriction?

> However, if we can't introduce a new revision in the 'debian' line, then
> it is really hard to find a single common base between the import lines.

Thanks for looking at this. I realise my characterisation of "better" is
fuzzy, which doesn't help.

>From our discussions it seems like the biggest improvement we could make
right now would be to improve the message when the user hits this
situation.

Is anyone able to suggest a good phrasing based on the improved
understanding of the problem? We can always extend it in to a help topic
if it isn't particularly terse.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Please check my thinking on bug 646979

[James Westby]

On Tue, 5 Oct 2010 10:50:08 -0400, Barry Warsaw  wrote:
> Won't all the patches Debian (or Ubuntu) adds be in patch system files living
> in debian/?  Of course, the looms<->patchsystem idea kind of blurs that, but
> ultimately the packaging directory should fully contain any downstream changes
> Ubuntu or Debian would add.  (I think. ;)

Firstly, not all packages are handled like that, and many have changes
outside the debian directory.

Secondly, patches inside the debian directory is a wart due to not
having a good VCS to manage the changes. We want to get rid of that in
some sense, not bake it in to the tools when we are working towards such
a system.

As you say looms<->patchsystems blurs the lines, but it would be great
to get to a point where patchsystems are just an export format that you
don't interact with for the most part.

It may be that looms would actually improve the cases we are discussing
anyway, or they may make things much more complex, it's hard to say :-)

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Please check my thinking on bug 646979

[James Westby]

On Mon, 04 Oct 2010 19:20:32 -0500, John Arbash Meinel  
wrote:
> So you preserve the content of exactly D or E, you just generate a new
> node in the graph to supersede the other one, correct?

Yes.

> Say D is the 'winner', then you end up with a patch that reverts
> everything in E.

Yes, except that D should in fact contain all the changes of E, so it's
not like we are completely backing them out, but that's how it appears
in the revision graph.

> In the below graph, the content in H == D, so when generating I, we
> should see D as the common base, and I would then == F (because H - D ==
> NULL, so there is nothing to apply to F to generate I)

Yes.

> Going further with the above example, it really depends on what you
> want. From what you've stated about "whichever one wins", then you sort
> of simplify what you want. You explicitly stated that you are rejecting
> any of the 2.1 changes that weren't in 3.0. I'm having a bit of a hard
> time figuring out what you are trying to preserve in G, versus just
> telling it "and now you are F/I".

We want to preserve any changes vs. E. G has some packaging changes
v.s. E, at least a Debian directory, and we want to preserve them,
conflicting where the other side has made incompatible changes. That's
again rather simplifying things though.

> Probably. Examples can certainly shed light on confusing points.

If you grab r44 of lp:ubuntu/lucid/brltty and r14 of
lp:debian/sid/brltty then you can merge-package them to see an instance
of this issue.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Please check my thinking on bug 646979

[James Westby]

On Mon, 04 Oct 2010 17:01:01 -0500, John Arbash Meinel  
wrote:
> On 9/29/2010 11:23 PM, James Westby wrote:
> > What merge package does is first merge the two upstream revisions
> > together, taking the tree from whichever has the highest version number.
> > 
> >   ---B---F
> >  /  /   /
> > /  / .-D--.
> > \ A-=  H
> >  \`-E-`
> >   \  \
> >C--G
> > 
> > Currently it will then just merge H in to G (the target). This can
> > generate conflicts, which are very, very confusing to users, as it's
> > incredibly hard to explain why they are getting them.
> > 

> Does merging D & E generate conflicts itself? It would seem that if
> merging to G generates conflicts, then you should have gotten a conflict
> in the intermediate stage as well. (offhand the best you can usually
> hope for is more understandable conflicts, unless you have a real
> 'criss-cross' merge and we are selecting a very poor base.)

It is not a real merge.

We create a new revision with D & E as parents, and the contents of the
later of the two (defined in terms of upstream version numbers). So, no,
there is no possibility of conflicts at this stage.

>   A
>  /|\
> E D B
> |\|\|\
> | H F C
>  \ \| |
>   \ I |
>\  |
> \ |
>  \|
>   G

> Now you have a genuine criss-cross. As the lcas are E and B (ancestors
> of both I and G that are not superseded by a more recent ancestor.)

> Just using 3-way merge (vs say --weave) I would expect this to conflict
> more than merging H => G, because of our specific base selection (when
> we find a criss-cross 3-way goes to the next base, which will be A,
> which then will try to merge (I-A) into (G-A).

That's unfortunate.

Is there a way we could use our increased knowledge about the revisions
involved to merge with a strategy that would make this situation better?

Would it be beneficial to have some concrete examples to try out?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Import failures in main

[James Westby]

On Sun, 03 Oct 2010 17:17:59 -0400, James Westby  
wrote:
> > I then turned this in to an almost-live page at
> > 
> >   http://package-import.ubuntu.com/status/main.html
> > 
> > (up to 5 minutes behind the current action)
> 
> Now down to 195 failures.

Now you can see at the bottom of

  http://package-import.ubuntu.com/status/main.html
  http://package-import.ubuntu.com/status/index.html

graphs of the queue size and failure count. They are not very
interesting right now, but should hopefully show some interesting
information after they have had some time to collect data.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Import failures in main

[James Westby]

On Sat, 02 Oct 2010 22:29:57 -0400, James Westby  
wrote:
> With 228 failures and 29 bugs there is quite a skewed distribution, and
> the two bugs causing the most failures are:
> 
>   "Packages failing due to pristine-tar not being able to reconstruct
>   their tarball" - https://bugs.launchpad.net/udd/+bug/653301

I'd forgotten that I had already reported this and Joey fixed the likely
cause of at least most of them. We just need to get the deployed to
jubany now (possibly test beforehand to ensure that most are indeed
fixed, but I think it's highly likely as he was testing with some of our
examples)

> and
> 
>   "Import fails with missing referenced chk root keys" -
>   https://bugs.edge.launchpad.net/udd/+bug/653307

Added some observations to this, but I'm unsure what bzr is telling us
here. Some help with interpreting that would be great.

I've also bumped the bug to critical as it looks like fairly widespread
repo corruption from an unknown source.

> The reason that the first is so high is that something seems to have
> changed recently that means that most of the KDE packages can no longer
> be handled by pristine tar.

I'm guessing the reason was either a switch to .bz2, or a change to the
machine on which the tarballs are generated, based on the Debian bug
logs.

> I then turned this in to an almost-live page at
> 
>   http://package-import.ubuntu.com/status/main.html
> 
> (up to 5 minutes behind the current action)

Now down to 195 failures.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Import failures in main

[James Westby]

Hi,

In order to get a better feel for the effect of import failures, I took
a look at the packages in main.

Currently there are 3292 packages in main. When I started looking there
were 305 failures, which was just below 10%. While going through the
failures I was able to solve around 80 there and then, which takes us to
a current failure rate of ~7%.

I then looked at all the failures and ensured there were bugs filed,
tagging them all "import-failure" and "main", so we can see the current
list at

  https://bugs.launchpad.net/udd/+bugs?field.tag=main

With 228 failures and 29 bugs there is quite a skewed distribution, and
the two bugs causing the most failures are:

  "Packages failing due to pristine-tar not being able to reconstruct
  their tarball" - https://bugs.launchpad.net/udd/+bug/653301

and 

  "Import fails with missing referenced chk root keys" -
  https://bugs.edge.launchpad.net/udd/+bug/653307

totalling ~100 failures in main.

The reason that the first is so high is that something seems to have
changed recently that means that most of the KDE packages can no longer
be handled by pristine tar.

I then turned this in to an almost-live page at

  http://package-import.ubuntu.com/status/main.html

(up to 5 minutes behind the current action)

Thanks,

James



-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Please check my thinking on bug 646979

[James Westby]

Hi,

I'd like someone to check my thinking before I make a change, as I don't
want to introduce data loss or something.

First, some background.

We have a merge-package command, as a simple bzr merge doesn't cut it on
occaision.

Here's why: (apologies to anyone using screen readers or variable width
fonts)

  ---B---F
 /  /   /
/  / .-D
\ A-=
 \`-E
  \  \
   C--G

(Time passes as you go right)

Here A is an "upstream" revision that is an import of a tarball. B and 
packaging revision based on that, and C is another packaging revision in
turn based on that. (Think B==Debian and C==Ubuntu)

Then D and E are independent packagings of two different upstream
release (say Debian jumped to 3.0, and Ubuntu took the point release
2.1). F and G are the new packaging revisions based on merging the old
ones with these new upstreams.

Now if we simply merge all F->G, we try and merge (D, F) in to (C, E,
G), which means we are merging changes from A->D with changes from A->E,
or in other words we are merging two upstream versions together.

Consider the case of a file with the version number in it. The BASE
would be "2.0", one side you change it to "2.1" and the other to "3.0",
but in reality they were sequential changes, we just can't represent
that here.

As a packager you don't care about this (generally), and you assume that
whatever in the latest is fine (you don't try and track at this point
patches in 2.1 that didn't make it in to 3.0, and even if you wanted to
this operation wouldn't just show you that)

What merge package does is first merge the two upstream revisions
together, taking the tree from whichever has the highest version number.

  ---B---F
 /  /   /
/  / .-D--.
\ A-=  H
 \`-E-`
  \  \
   C--G

Currently it will then just merge H in to G (the target). This can
generate conflicts, which are very, very confusing to users, as it's
incredibly hard to explain why they are getting them.

  ---B---F
 /  /   /
/  / .-D--.
\ A-=  H
 \`-E-` \
  \  \   \
   C--G---I

Once we have that merged revision we can merge F->I, which will merge
what we want.

I was thinking about this the other day and realised that uncoditionally
merging H to the target might not be the right thing to do. I think that
it should merge it in to the side that had the highest version. That
should never generate conflicts, as the revision that is being merged
has no changes against the LCA. I think it should generate an
equivalent merge though.

So, back to the graphs, if we this time consider D to be newer, but
still want to merge F->G, would it be ok if we created I on the other
side:

  ---B---FI
 /  /   //
/  / .-D--. /
\ A-=  H
 \`-E-`
  \  \
   C--G

and then merged I->G for our final revision?

I think that it should be "safe" and remove those "odd" conflicts you
get in the intermediate state, instead moving them to the final merge
when they should hopefully make sense.

Can anyone generate a scenario where this would give a worse outcode?
Can anyone in fact generate a scenario where either strategy is the
wrong thing to do?

The more I think about it the more I am confident in the change, but it
certainly doesn't seem like an obviously correct change to me.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Improving merge-upstream

[James Westby]

On Tue, 28 Sep 2010 22:37:38 +0200, Jelmer Vernooij  
wrote:
> +1 This'd be a great feature indeed. It would be nice to implement this
> by using the UpstreamSource API (perhaps by extending it with some
> optional methods). 

I think it needs some tweaks/extension, but I know this is one of the
things you envisaged with that API, so I would like to see that too.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Improving merge-upstream

[James Westby]

On Tue, 28 Sep 2010 21:39:54 +0200, Vincent Ladeuil  
wrote:
> There is a merge proposal that goes in this direction pending review at:
> https://code.edge.launchpad.net/~vila/bzr/323111-orphan-config-option/+merge/35690
> (with pre-requisites).

Great.

> > 4) Remembering the upstream branch.
> 
> > merge-upstream takes an optional upstream branch. Currently there are
> > cases where one person supplies it, but the next doesn't, and that's not
> > ideal, as you get a discontinuity, and there can be extra conflicts in
> > future due to added files.
> 
> Still related to parallel imports ?

It is parallel imports, yes.

> Martin mentioned recently that we could special-case merge to relax the
> conflicts for different file-ids by checking the paths involved. That
> would not solve the parallel import problem but it may help for the
> packaging and upstream branches relationship.

I think so. This is beyond just that problem though, in that it could be
recording a richer ancestry that it will if you forget the upstream
branch.

> > Therefore I would like if if, similar to e.g. bzr merge, if you supply a
> > branch once it is remembered and used again.
> 
> That's clearly a case where some config file should be versioned and
> shared even if it needs to be versioned in a specific way (the reference
> to the upstream branch is a live data that doesn't flow in the same way
> than the upstream code or the packaging data (or does it?)).

It's slightly independent, yes. I would put it in
.bzr-builddeb/default.conf, as that is a good start.

> Would you qualify this upstream branch as a data shared by a set of
> branches but for which updates are never reverted ?

I'm not sure I follow this. I could concoct a situation in which it is
reverted.

> Couldn't you use an ancestry check against the last merged revision ?

That could make sense.

It wouldn't stop partially unrelated things being merged. If you are
merging a release from a stable branch then you don't want to merge
trunk.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Improving merge-upstream

[James Westby]

Hi,

After talking with a few people about it over the last few weeks, and
seeing some difficulties that people face with merge-upstream, I'd like
to start a discussion of what could be done to improve it.

I have some ideas, but please present any others you have.

1) The obvious one, watch file intergration.

We don't have to make you give the URL of the tarball if you have a
watch file. This has been the intent from the first day I wrote
merge-upstream, and the reason why "--version" is a mandatory option
right now, I just haven't got around to it.

I'd be more than happy to help someone implement this. The uscan part
isn't too hard, you just have to run it with a certain combination of
options and parse the resulting xml-like. The only difficulty is
threading it in to the maze of code to deal with all the optional
arguments. I'd love to clean that up and get this feature in.

2) Another obvious one, fixing bugs.

I've just found another "incosistent delta" bug, and there are probably
more. Getting them fixed would make it more reliable.

There may be other classes of bug to fix too.

3) Improved conflicts.

I'm not sure this is really merge-upstream specific, but often people
have difficulty understanding why a conflict happened, especially with
this command it seems.

I know there has been some work on improving the conflict experience,
and I'm not sure if that has had an impact here yet.

4) Remembering the upstream branch.

merge-upstream takes an optional upstream branch. Currently there are
cases where one person supplies it, but the next doesn't, and that's not
ideal, as you get a discontinuity, and there can be extra conflicts in
future due to added files.

Therefore I would like if if, similar to e.g. bzr merge, if you supply a
branch once it is remembered and used again.

However, I don't want to have it merge a completely unrelated revision
of some branch which you never told it to use (but someone else did
previously.)

Therefore I was considering the following:

  bzr merge-upstream --version 1 tarball branch

remembers branch, and then

  bzr merge-upstream --version 2 tarball

will re-use branch. However, it will look for a tag "2" in the branch,
and if found that is the revision it will use. If it is not found it
will error and demand one of -r or --no-branch.

That way it shouldn't merge completely unrelated things.

There's room for future improvements there, to have an "upstream tag
scheme", and I'm not exactly sure how to handle that in a nice way, but
that's why it's a future improvement.


Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Loom thread -> patch file (take 1)

[James Westby]

On Thu, 19 Aug 2010 17:07:29 -0400, Barry Warsaw  wrote:
> I've written up some guidelines on the UDD wiki pages about turning a loom
> into a patch system patch file.  It's rough and I've only really done this
> once so comments are welcome.  James W suggests a bzr command to automate
> this, and I may try to hack something together, because I agree it would be
> great. :)
> 
> https://wiki.ubuntu.com/DistributedDevelopment/Documentation/PatchSystem

Thanks for writing this down.

And yes, a bzr command would be great.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: [braindump] ppa update discontents

[James Westby]

On Fri, 20 Aug 2010 10:53:32 -0400, Barry Warsaw  wrote:
> +1 to that.  I use 'bzr bd -S' all the time and 'bzr bd' occasionally.  It's
> just occurred to me that 'bzr bd --builder=' could probably be used to
> do the build in the proper sbuild chroot, but it would be nice to be able to
> configure that persistently in a Bazaar config file, rather than all the
> memory living in my bash history.

Patches welcome. Currently it's at a rudimentary level that allows
supporting the myriad different ways of building. Improving on this such
that the experience can be incrementally improved for specific ways of
building is certainly something I would like.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: [braindump] ppa update discontents

[James Westby]

On Fri, 20 Aug 2010 18:47:11 +1000, Martin Pool  wrote:
> Along with maxb and garyvdm I've been updating the bzr beta ppa for
> our 2.2 release.  This is not absolutely the most fun thing I've ever
> done; in fact it's pretty tedious.  Since we're supposed to be making
> tools to make packaging more productive and fun we could either find
> some more things here to improve, or perhaps that I'm doing it wrong.
> This mail has more problems than solutions but it's a start.

Thanks, complaints without solutions are useful anyway, so any proposed
solutions are a bonus.

> It's the kind of tedious work that doesn't require much creative input
> but is not very automated, seems to have a bunch of snags that make it
> not trivially automatable, and with long latency, so you can't just
> sit down and do it then know you're done.  It's about 10 minutes of
> actual thinking spread over a couple of days of spade work.

Are there ways that we could chain things for you, so that you could set
them up, and if all went well all steps would be done?

> Most of what we do is just add a rebuild line into the debian
> changelog, push, and upload, but things occasionally go wrong per
> package.  This should be very automatable.

Rebuild line to build for a different distroseries?

> It can be a bit hard to work out which packaging branch is used for
> the version in our ppa.  Launchpad has a concept of series branches
> for We can have a convention for plugins that we package, but it's
> just a convention and because it's evolved over time it's not always
> consistent.  For instance if Gary has uploaded bzr-gtk -
> 0.99.0-1~hardy4 but it's failing, I can get the source for that
> package and try to fix it, but I can't easily work out what branch
> contains that source.  (Well, I can guess it's one of the recently
> updated branches in  but it's
> not great.)

  https://bugs.launchpad.net/launchpad-code/+bug/395200

is one thing that might help that. As it would allow for a package in a
PPA to tell you a list of branches that contain the revision that built
it.

Also, recipes could help here, as they are pointers to branches, and I
think you could go last build->recipe and work from there.

Lastly, Launchpad may want to do something to have "official" branches
for PPAs, like we do for distributions.

> The lag seems to come in mostly through Soyuz delays: there's a delay
> before your package is accepted, then a delay that was running at
> several hours yesterday before it's actually built or rejected.  I do
> realize that there's a lot of demand for Soyuz build machines and that
> people are working on improving scheduling and performance.  But still
> it does mean there's this long period where you have mental
> work-in-progress, and can't wholeheartedly move on to something else.
> If there's a mistake either in your work, or the scripts, you have to
> page in the state you had a while ago.  The long lag is pretty
> unreasonable when bzr has barely any compilation steps and builds in
> well under a minute on a laptop.

You were test-building the packages locally? That's what I do, and means
that I'm not waiting on the builds to know if they worked or not. (They
sometimes fail, but it's rare enough that I assume that they won't.)

> One approach to the lag in soyuz builds is to test build every package
> locally, before uploading.  This seems like a bit of an unnecessary
> kludge, but perhaps it's the most sensible thing for now.

(Should read the email once before replying :-)

Why do you consider it a kludge?

> It seems like we have to keep track of which builds have passed or
> failed by collecting a set of nearly-identical mails.  There's a
> "latest updates" portlet in eg
>  but it only shows
> the last 5 which is not really enough (http://pad.lv/620903).

You know about the "View package details" link?

You're looking at the user-focused page their, clicking on that link
will get you the developer-focused one.

> Perhaps we could standardize a way to do local test builds in a way as
> close as possible to what Launchpad does, including how to get
> bzr-builddeb to use it for a a test build before uploading.

Sounds like a plan.

> Our process at the moment is to upload everything into
>  before then promoting
> it to the regular archive, so that we don't break dependencies between
> packages.  (Uploading a bzr newer than the plugins support might make
> them uninstallable, and unlike the main archive(?) ppas don't
> themselves check for this.)  I think the basic approach of using a
> staging archive is ok.  We ought to be able to script something that
> does both the check and the promotion if it succeeds.

The main archive does not check for this.

> Perhaps some or even a lot of this could be addressed by giving
> bzr-builder the ability to build from a tag into a ppa (or perhaps

Re: Growth of Ubuntu Distributed Development

[James Westby]

On 20 Jul 2010 09:59:01 +0200, "Michael Bienia"  wrote:
> On 2010-07-19 17:59:22 -0400, Francis J. Lacoste wrote:
> Hi,
> 
> > Does that seem consistent with your views of the current usage? What is 
> > blocking more users from jumping on the UDD bandwagon?
> 
> My last attempt (yesterday) to use packaging branches to prepare a
> change (to get it sponsored) stopped early because the branch was
> out-of-date (package: mutt).  I had to fall back to create a debdiff.

Please file bugs against https://bugs.launchpad.net/udd when you hit
this, as it's useful to know when people actually get blocked by it.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Growth of Ubuntu Distributed Development

[James Westby]

On Sat, 24 Jul 2010 10:11:51 +0200, Mathias Gug  wrote:
> We've also talked about bzr/LP recipes.
> 
> I'd like to be able to write a recipe that includes the debian/
> directory from a package in Ubuntu. That would help in bootstrapping a
> recipe as well as checking whether the packaging code still applies to
> the upstream branch.
> Something similar to (I don't remember the exact syntax):
> 
>  lp:puppet/trunk
>  lp:ubuntu/maverick/puppet

Andrew is working on making this possible.

> The recipe above would take the latest upstream trunk and the debian
> packaging from the maverick puppet package to build a package.
> 
> I'd also like to be able to build a specific tag from upstream:
> 
>  lp:puppet/trunk 2.6.0
>  lp:ubuntu/maverick/puppet

That's possible with

  lp:puppet/trunk tag:2.6.0

> In the example above, 2.6.0 is an tag in the upstream git repository
> (which is imported in LP).

Assuming that part all works fine.

> We've also talked about integration with a CI system (hudson and/or
> tarmac?). Whenever a new package is built successfully (ie available in
> a PPA) the CI system automatically starts a test using the PPA. James
> mentioned that some notification system is needed (already available
> somehow?).

We can write code to poll for new packages easily. Is hudson the right
thing for controlling the testing? Can you trigger it to run against a
package whenever we like? Or could we have a plugin so that it watches a
PPA rather than a VCS for things to do?

> I'd also like to be able to create a recipe that automatically builds a
> new package when:
>  1. a new revision appears in the upstream bzr branch
>  2. a new tag appears in the upstream bzr branch
>  3. on a regular basis (daily, hourly).

Without outside scripting only daily or on demand are available right
now. I think all the others make sense, but it's up to the Launchpad
team what they want to support built in, given UI and resource
constraints.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Growth of Ubuntu Distributed Development

[James Westby]

On Wed, 21 Jul 2010 11:07:59 +0200, Martin Pool  wrote:
> We got some more feedback last night from Mathias and Thierry, maybe
> they can add something to what I remember:
> 
> * dpkgv3 tends to mean that the diffs generated by bzr get harder to
> read, but we could handle this and make it look better
> * looms are good; it would be useful to expand packages into looms

I think these are covered by bugs we already have? Maybe an explicit one
for the first part would be good to have.

> * when users submit changes with changes naively committed into the
> working tree we could automatically correct that into the correct
> patch style for the package

A bug for this sounds good.

> * it would be cool to have a bot that notices package merge proposals
> and builds and attaches a binary package for testing, so it's ready to
> go as soon as someone looks at the merge

We can file a bug for this, though it's a project of its own, and not
strictly limited to package branches (a generic "run some tests against
a merge proposal" bot we could configure would be great).

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Growth of Ubuntu Distributed Development

[James Westby]

On Tue, 20 Jul 2010 10:20:03 -0400, Scott Kitterman  
wrote:
> Specifically for merging, I think an equivalent of grab-merge (I'll go so far 
> as to suggest implementing something in grab-merge and then people can grab-
> merge --bzr or something) and an easy way to diff just the non-upstream 
> changes 
> are essential to broader adoption.

Let's work out how we can do this, and then we can look at implementing
it.

I would like it to be an operation done entirely within bzr if possible
(there are already ways to do it with source packages).

What we want to do is compare the old packaged with its upstream
version, and then compare that with the changes from the new upstream to
the new packaged source.

I don't think bzr has any special facilities for interdiffs right now?
In that case we could start by generating those two diffs and piping
them to interdiff. Getting the diffs themselves is relatively
straightforward.

What should the command be? Just an option to diff? diff-upstream?

How much would you want to be able to tweak things beyond the default of
showing the differences between the last two versions in debian/changelog?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Growth of Ubuntu Distributed Development

[James Westby]

On Mon, 19 Jul 2010 17:59:22 -0400, "Francis J. Lacoste" 
 wrote:
> Hi,
> 
> I'm interested in getting an understanding whether the usage of UDD is 
> growing 
> or not.
> 
> According to the stats I get from LP usage, we are seeing more and more 
> package branches being created, but not necessarily many more people using it.

A 50% increase in registrants isn't more people using it?

> My theory is that usage is increasing but among a core group of people. So a 
> few more people try it out, but haven't persisted in making the switch. But 
> the ~15-25 people how are really using it are using it more.
> 
> Does that seem consistent with your views of the current usage? What is 
> blocking more users from jumping on the UDD bandwagon?

This is consistent with my impression of the current usage, with more
people picking it up, but not in huge numbers.

Bear in mind that with a couple of hundred active Ubuntu developers
(core-dev, motu, contributors, etc.) we won't see huge numbers unless we
start having a large impact on the number of people who contribute. I
would like to do that, but unless we can get a majority of existing
contributors happy with using it then I'm not sure we can.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD health check?

[James Westby]

On Tue, 13 Jul 2010 16:44:34 -0400, Barry Warsaw  wrote:
> >Yes, and I'm not supposed to directly edit any of the source outside
> >of debian/, which is why the layout used by svn-buildpackage with only
> >debian/ in VCS resonates with me.

It is true that this is recommended practice. However, this is mainly
due to the fact that while the Debian package format keeps your changes
separate from the upstream source, it is very hard to review and modify
multiple patches that aren't in a patch system.

I was very strongly in favour of this approach until I had better tools
available.

Tools such as looms promise to allow the best of both worlds, managing
the patches to upstream, while still giving you the full source in
version control for things like...

> > Having said that, the full extracted
> >source is incredibly useful when I'm debugging - it's nice to be able
> >to diff a given source file between different package versions which
> >sometimes don't line up in a meaningful way with upstream VCS (because
> >they can be patched in the debian package).

... that.

Being able to "bzr blame" a file and see when a change was introduced,
regardless of whether it is a patch or an upstream change is great. I
think that a lot of people don't see that yet as they either avoid the
type of work where this action is necessary, or they have ingrained
workarounds due to the difficulties and inconsistencies we have now.

> This leads to my last comment.  While you're not *supposed* to touch anything
> outside of debian/, I think that if there were nice tools to convert back and
> forth from dvcs-lingo to patch-system-lingo, we'd have an even more powerful
> development system.

Agreed, and I would love to have this too. Maybe you'd like to chat on
list or in person next week and see if there are some easy things we can
do to move us in that direction?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD health check?

[James Westby]

On Tue, 13 Jul 2010 22:27:52 +0100, Dmitrijs Ledkovs 
 wrote:
> On 13 July 2010 18:23, Elliot Murphy  wrote:
> > On Tue, Jul 13, 2010 at 11:08 AM, Martin Pool  wrote:
> >> I asked Barry how he was finding the UDD experience
> > 
> >> Any other suggestions/wishes/hates?
> >
> > I have found UDD very pleasant for doing merges from debian to ubuntu,
> > and I *love* using merge proposals to ask for changes to be sponsored
> > into Ubuntu. When I want to put a python or erlang package into Ubuntu
> > via debian, I end up needing to use SVN and svn-buildpackage which is
> > a completely different world/workflow from UDD and that difference can
> > be a bit overwhelming at first.
> >
> 
> Why not checkout debian/ dir using bzr-svn and add a local
> bzr-builddeb config to use this checkout in merge-mode? =) YMMV.

Actually, Jelmer made it such that bzr-builddeb will detect this and so
it /should/ do the right thing if you just run "bzr bd" in your SVN
checkout of DPMT.

Bugs welcome if it doesn't, or if there are other things we can do to
make this work easier. I'm all for getting collaboration with Debian to
the same point as we are at with UDD just targetting Ubuntu.

I think one issue is that I only really considered "Ubuntu developer
that just sends patches to Debian" and "Ubuntu developer that can upload
the package to Debian," not "Ubuntu developer that can commit to Debian
SVN."

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD health check?

[James Westby]

On Tue, 13 Jul 2010 13:23:27 -0400, Elliot Murphy  wrote:
> When/where to use the bzr mark-uploaded command when sponsoring
> someone elses work is a bit mysterious to me.

When you upload the package to it's destination (Ubuntu in this case);
you "mark" the package as "uploaded."

> Overall UDD just works for me and generally stays out of my way.

Great!

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: [Launchpad-users] upstream + packaging + looms + lp != happiness

[James Westby]

On Thu, 27 May 2010 16:18:20 -0400, Barry Warsaw  wrote:
> This is the sanest way I can currently think of for organizing the upstream
> and packaging branches for two distros and a PPA.  If it works, it seems like
> a nice way to manage things.

I agree that your sketch looks like a good way to organise things.

> Unfortunately, this doesn't work because round tripping the branch through
> Launchpad throws away all the loom information.  Here's the transcript from
> two different machines ('bzr looms' is an alias for 'bzr show-loom'; the
> 'split' thread is an artifact that will go away).
> 
> limelight% bzr looms
> =>ubuntu
>   debian
>   split
>   trunk
> limelight% bzr record
> Loom recorded.
> limelight% bzr push lp:~barry/computer-janitor/loomified
> Using default stacking branch 
> /~computer-janitor-hackers/computer-janitor/trunk at 
> lp-69637584:///~barry/computer-janitor
> Created new stacked branch referring to 
> /~computer-janitor-hackers/computer-janitor/trunk.
> 
> heresy% bzr branch lp:~barry/computer-janitor/loomified
> Branched 240 revision(s). 
>  
> heresy% cd loomified/
> heresy%
> 
> All the threads disappeared when the branch was pulled to machine heresy.
> Maybe Launchpad doesn't support looms yet?  Maybe the stacking is messing
> things up?  Any other suggestions or comments?

No idea, sorry. Have you filed a bug for tracking purposes?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: diagnosing collisions

[James Westby]

On Wed, 07 Apr 2010 14:18:12 +1000, Robert Collins  
wrote:
> Is there any docs on doing this?
> 
> Many, if not most of the udd bugs are auto generated bug reports like
> https://bugs.edge.launchpad.net/udd/+bug/414095.
> 
> I'd like to analyse and fix these, but currently I'm starting from first
> principles.

Hi,

I wrote up

https://wiki.ubuntu.com/DistributedDevelopment/UnderTheHood/Importer#Collisions
https://wiki.ubuntu.com/DistributedDevelopment/UnderTheHood/Importer/Operational#Investigating%20collisions

which is as much as I can externalise on the technique.

For the example you linked I opened

  http://bazaar.launchpad.net/~ubuntu-branches/debian/sid/ncurses/sid/changes
  
http://bazaar.launchpad.net/~ubuntu-branches/debian/sid/ncurses/sid-200908151535/changes

and found

  Collision for 5.7+2008115-1 sid expecting last revision
  (5.7+20090523-1) to be
  james.wes...@ubuntu.com-20090524151301-uixgxq2zonfov2nr, but it was
  james.wes...@ubuntu.com-20081107210600-mvmjg00uf73ffzuy instead,
  pushing to
  
bzr+ssh://jame...@bazaar.launchpad.net/~ubuntu-branches/debian/sid/ncurses/sid-200908151533

You can see in the lp:debian/sid branch that when it imported that "last
revision" it failed to tag it with a revision in this history of that
branch, tagging some other revision instead. I fixed that bug a couple
of weeks ago, but didn't fix old data.

Let me know if I can provide any more information.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Pipes3.0 maybe Threads3.0

[James Westby]

On Mon, 29 Mar 2010 23:32:19 +0100, Dmitrijs Ledkovs  
wrote:
> When we will have nested trees everything will be fine ;-)

Yes, they are certainly part of the problem I think.

> 1) If lp branches are in the revealed structure we are loosing
> compatability with debian. Right now you can browse Logerhead view and
> ever single debian maintainer will understand what's going on. Even
> hard-core tool-chain maintainers we don't depend on dh at all =)
> 
> 2) Storing in revealed structure will make it harder to move around &
> host elsewhere. Eg. people with bzr without any plugins. (looms &
> pipelines & buildeb are all optional and not shipped with bzr)
> 
> 3) By having revealed state locally, I can rewrite it as much as
> possible until I like what I see ;-)

These are good arguments.

I think it makes it harder to collaborate on work in progress, but
perhaps that's a reasonable trade-off.

You haven't convinced me, but you've at least shifted my opinion :-)

> The content filter was so that you don't have to re-export patches
> after you have edited it in a pipe and came back to the top packaging
> pipe such that you can envoke bzr-builddeb

Right, I like that.

> You can edit debian/patches in regular mode and go between releaved
> <-> colapsed states if you want to switch between editing by hand &
> via pipes.
> 
> In the revealed mode changing the order of patches in the series file
> will instantly make pipes history inconsistent which was just created.
> So no you can't edit debian/patches directly with editor, you have to
> commit to pipe.
>
> > Do you think it would confuse some people in to e.g. losing work as they
> > think they can just alter that patch and be done?
> >
> 
> Good point.
> In revealed state add debian/patches/WARNING-DO-NOT-EDIT with help
> text? Make debian/patches non-writable? Drop people into subshell with
> cool modeline showing pipliene status? Have a notify-osd pop-up to
> reminde them?
> 
> Needs work.

Indeed.
 
> >
> > Right, so you don't want to rewrite?
> >
> 
> At this stage each time you reveal you are creating history and can do
> it locally as many times as you like until you make this plugin work
> ;-)
> When we are ready, we can rewrite, It will work best for 3.0 quilt
> packages cause we want to store the branch in patched state.
> And if we rewrite history we would be better off rewriting on top of
> vcs-imports.
> 
> So my answer is maybe / later.

Sounds reasonable.

> > In all the talk so far the opinion has been that we would just make
> > lp:$distro/* a loom that would provide this without the extra step.
> >
> 
> Ok. Chicken & Egg problem between: looms are used <--> logerhead &
> merge proposals work with looms.

Good point.

> I haven't used looms yet. And pipes are far more stable imho cause
> it's just lightweigth checkouts with UI on top. Plus some parts of
> this spec will be useful even without whole reveal mechanism eg. quilt
> patch series import-export to from pipes of a single commit vs

Yep. pipes work fine for this use case, but don't currently support the
use-case of making lp:ubuntu/* in to the revealed state, which is why
looms have been spoken about more for this task.

> Thanks a lot of comments. I will move to appropriate place such that
> at least this stub ideas are not lost ;-)

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Pipes3.0 maybe Threads3.0

[James Westby]

On Sat, 27 Mar 2010 00:43:26 +, Dmitrijs Ledkovs  
wrote:
> Hello
> 
> How about a spec for integrating 3.0 quilt packages with bzr pipelines
> or threads?

Hi Dmitrijs,

Thanks for working on this, my comments are inline.

> Question, comments and edits are welcome here and on
> https://wiki.ubuntu.com/DmitrijsLedkovs/Pipes3.0
> 
> == Summary ==
> 
> This spec makes bzr & bzr-bd awesome for managing 1.0 or DpkgSrc3.0
> (quilt) packages.
> Because currently it's not easy to manage packages using bzr with 3.0
> (quilt) all patches applied state. And looking at debdiff of the
> previous revisions it's mostly diff-of-diff in the debian/patches/ (I
> can't read that ;-))
> 
> == Assumptions ==
> 
> Spec does not include multiple tarballs format.

Yes, these are tricky to deal with, so I'm happy to defer including them
in this until we have an idea of how to handle them.

> == Design ==
> 
> Imagine bzr:$distro/$package branch implemented as a pipeline. You would have.
> 
>* upstream
>* patch-1
>* ubuntu-patch-1 (changes to patch-1 made for ubuntu)
>* patch-2
>* patch-3
>* debian-packaging
>* ubuntu-packaging (e.g. Maintainer-name, soname bump)
> 
> With package tags on packaging pipes. Such that when you switch to
> that tag, previous pipes are merged (pumped) resulting in your current
> commit and debian/patches/series is replica of pipeline looking
> downward to upstream, and patches are as if you did "bzr pump-patches"
> to make this commit.
> 
> Execpt that our lp:$distro/$package branches are actually 3 merged dsc
> import branches (upstream, debian commits, ubuntu commits) which you
> can access by using tags or point revisions (e.g. 1.1.1):
> 
>* upstream
>* debian-packaging (with patches applied or not)
>* ubuntu-packaging (with patches applied or not)
> 
> My proposal is to locally "reveal" this branch to create a pipeline
> view of it. To work on it and allow to fold back as uncommitted state.
> 
> == Implementation
> 
> So we should add '''bzr reveal-patches''' command which will
> inteligently create a pipeline view of the packag branch like this:
> 
>  1. Shelve current packaging branch somewhere safe similar to
> ~/.bzr/pipes or if using lightweight checkout we are safe.
>  2. Branch off first upstream revision of the to be revealed packaging range
>  3. Make it base of the pipeline.
>  4. Looking at the shelved packaging branch start importing each patch
> as a pipe for equivalent debian revision.
>  5. Make at debian pipe for  "debian/" packaging excluding debian/patches
>  6. For each Ubuntu revision on top of debian
> 1. where possible modify existing patch pipe
> 2. add/remove patch pipes
> 3. and add ubuntu packaging changes on separate pipe above debian
> Do this for all requested package version range
> 
> Each commit should be tagged along the way: Eg. 01_patch-debian-3.2-1
> for a patch pipe, debian-3.2-1 on debian branch and
> ubuntu-3.2-1ubuntu1 for ubuntu branch.
> 
> Note this doesn't re-write history because the real packaging branch
> is shelved safely away and used for reference.
> 
> So far so good, so we have our very nice pipeline view of the package.
> So how would you build a package while on the debian pipe tagged
> 3.2-1?
> 
> To achive this we will use a read/write content filter to show
> debian/patches directory. The content filter will export each
> patch-pipe as quilt series into debian/patches. So that you can invoke
> bzr-bd using current state & generating original tarballs from the
> shelved packaging branch.
> 
> So for example you are MOTU (Master Of The Unseeded) and need to do a
> complex merge with 10 patches. You would grab  lp:ubuntu/$package.
> Reveal it from the base revision. Import new debian revisions from
> lp:debian/$package in revealed state. Do a bzr pump resolving
> conflicts. Do a build, do testing. Fold the revealed state into
> lp:ubuntu/$package which will become a single commit/tag, mark
> uploaded and push to launchad for building.


Interesting approach.

Do you think there is a benefit to providing a command to reveal the
patches, rather than making lp:$distro/* use the revealed structure?

It certainly sounds like you have some good ideas for implementing some
of the tricky bits.

> == Implementation ==
> 
> Modify bzrtools patch command to use quilt for importing patches.
> 
> Provide new command import-quilt in bzr-pipeline plugin, which takes
> quilt series file as argument and creates one pipe per patch. Pipes
> should be named after quilt patch file names. Pipe should store quilt
> header (e.g. DEP-3 headers) in branch config for example.

Good point, the DEP-3 stuff would have to be round-tripped and editable
somehow.

> Provide new command export-quilt in bzr-pipeline plugin, which takes
> output directory as argument and creates quilt series there. This
> should be similar to the current pipe-patches command exept that
> series file must be created and  patches should have he

Re: Tarball inside tarball

[James Westby]

Hi,

Sorry for the delay in responding to this.

On Thu, 21 Jan 2010 00:02:18 +, Dmitrijs Ledkovs  
wrote:
> Main is still using https://code.edge.launchpad.net/~ubuntu-dev for
> their packaging. And that's ok until we iron out all Documentation &
> Infrastructure bugs.
> 
> But some lp:ubuntu/package branches are pure evil =)
> 
> For example 
> https://code.edge.launchpad.net/~ubuntu-branches/ubuntu/lucid/xulrunner-1.9.1/lucid
> is useless for UDD.
> 
> It uses tarball-inside tarball packaging hence the branch is HUGE in
> size and useless for using bzr for example to patch it. This is
> probably a bad example of a package because it is managed as part of
> the ~ubuntu-dev branches in merge mode. But this probably affects a
> few package branches.

Yes, they aren't great for using bzr.

> Will it be possible to identify all of these packages and file bugs
> against them in Ubuntu/Debian (Priority Wishlist?) and make a switch
> to more regular layout?

I can't think of a better way than iterating over all packages and using
a heuristic, can you?

> These requests will be somewhat blocked by the dpkg source 3.0
> transition (support for bz2 compression) & the bzr-bd support for
> those source formats.

Indeed.

The latter should be about solved now.

I have heard other reasons given for using tarball-in-tarball than just
compression, but I don't think it's a good long-term strategy.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Thoughts on 3.0 (quilt) and VCS workflows

[James Westby]

On Tue, 2 Mar 2010 11:25:25 +, Colin Watson  wrote:
> I thought some people here might be interested in the bug I just filed
> on dpkg-dev:
> 
>   http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=572204

Interesting, thanks.

It matches some of my impressions.

Do you know if your hack is required for the lp:ubuntu/* branches, where
we store exactly what dpkg-source -x produces?

I assume this writes a .pc directory and so it would allow you
manipulate the patches directly.

That doesn't get us away from the larger issues though.

> This was independent of the discussion in
> https://lists.ubuntu.com/archives/ubuntu-distributed-devel/2010-February/000508.html
> as I hadn't read that yet, but the second problem I mention in that bug
> is essentially the same thing.

Indeed. I think it might be trickier to solve the merging case, as you
have to reconcile differences in two stacks of patches. They come from
the same root issue though, storing redundant data essentially.

>  I realise the solution I propose isn't
> optimal given further bzr development, but I think it's the cleanest
> method that works right now even if the cost is losing bzr conflict
> resolution (which is annoying, but I'd rather that than have to mangle
> quilt patch files by hand or accidentally mis-refresh a quilt patch to
> include the upstream diff or any of the other things that can go wrong
> when you confuse quilt about its base file contents).  Thoughts?

It is probably the best way, you are right.

I would love to improve this situation, but it requires some rather
large changes to bzr to be able to do this well.

I would be happy to implement an interim solution that didn't prejudice
the long-term solution too much.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Operational changes and documentation

[James Westby]

On Fri, 19 Feb 2010 01:34:35 +, James Westby  
wrote:
> And just as I send this I notice that something has gone screwy and so
> I've stopped the importer. If those people could resist the urge to play
> until I have had a chance to investigate after sleeping :-) (Feel free
> to investigate, but I would advise against restarting the daemon until
> the problem is sorted.)

I forgot to note to the list that I fixed that issue and we are up and
running again.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Operational changes and documentation

[James Westby]

On Fri, 19 Feb 2010 01:28:27 +, James Westby  
wrote:
> Hi,
> 
> Our wonderful sysadmins today helped migrate the package importer to a
> shared environment so that others can help administer it. Unfortunately
> that can only be Canonical employees currently as it requires machine
> access.
> 
> So, the following people can now twiddle with the importer if you need
> something doing, you don't have to go through me:
> 
>   * igc (once he gets added to the group, sorry Ian)
>   * jam
>   * lifeless
>   * poolie
>   * spiv
>   * vila
> 
> Bear with us while we work out how to do it, as some requests will still
> be routed through me while they learn the ins and outs of the system.

And just as I send this I notice that something has gone screwy and so
I've stopped the importer. If those people could resist the urge to play
until I have had a chance to investigate after sleeping :-) (Feel free
to investigate, but I would advise against restarting the daemon until
the problem is sorted.)

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Operational changes and documentation

[James Westby]

Hi,

Our wonderful sysadmins today helped migrate the package importer to a
shared environment so that others can help administer it. Unfortunately
that can only be Canonical employees currently as it requires machine
access.

So, the following people can now twiddle with the importer if you need
something doing, you don't have to go through me:

  * igc (once he gets added to the group, sorry Ian)
  * jam
  * lifeless
  * poolie
  * spiv
  * vila

Bear with us while we work out how to do it, as some requests will still
be routed through me while they learn the ins and outs of the system.

To help them I wrote some docuementation, but it is more widely
applicable, just getting in to the details at the end. You can find it
at

  https://wiki.ubuntu.com/DistributedDevelopment/UnderTheHood

and the links therein.

Help improving this would be appreciated.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[James Westby]

On Thu, 18 Feb 2010 12:54:01 +1100, Robert Collins  
wrote:
> Ah:
> 
> CHKInventoryRepository('lp-hosted:///~ubuntu-branches/debian/sid/camlp5/sid/.bzr/repository',
>  
> fallback_repositories=[CHKInventoryRepository('lp-hosted:///~ubuntu-branches/debian/squeeze/camlp5/squeeze/.bzr/repository')])
>  has no revision james.wes...@ubuntu.com-20091029134422-acd9dzuxa44whcct
> 
> We have a bug open on this - I think we should make it critical.

bug 507040 was the only one that looked similar on a search, is that the
one?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Testing package imports locally

[James Westby]

On Wed, 17 Feb 2010 14:55:31 -0600, John Arbash Meinel  
wrote:
> Well #1, it gives a place to track bugs other than as bugs in
> 'bzr-builddeb'.

> #2 the stacking code causes lp to stack on an unrelated branch
> (bzr-builddeb). ATM it doesn't matter because the branches are small.
> But it does mean it sets up stacking, and then goes ahead and pushes all
> of the ancestry anyway.

> #3 When submitting a merge proposal, it will pick the default 'trunk'
> branch, rather than wanting me to merge into bzr-builddeb's trunk itself.

> If I was picking a place, I'd pick udd, since that seems more a
> 'collection of random stuff', rather than a specific project. For now, I
> guess one place is as good as another.

Done.

Please use lp:udd from now on.

You can join the ~udd team to get write access.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[James Westby]

On Wed, 17 Feb 2010 08:14:26 +1300, Michael Hudson 
 wrote:
> For one reason and another, Tim ran the script in the end, but it looks
> like it's finished now but you might want to rerun your script to check.

Thanks to both of you.

However, the attached didn't seem to "stick". Only 57 branches this time
though.

Thanks,

James

~ubuntu-branches/debian/sid/aiksaurus/sid
~ubuntu-branches/debian/squeeze/anacron/squeeze
~ubuntu-branches/debian/lenny/anacron/lenny
~ubuntu-branches/ubuntu/lucid/anacron/lucid
~ubuntu-branches/ubuntu/karmic/anacron/karmic
~ubuntu-branches/ubuntu/jaunty/anacron/jaunty
~ubuntu-branches/ubuntu/intrepid/anacron/intrepid
~ubuntu-branches/ubuntu/hardy/anacron/hardy
~ubuntu-branches/ubuntu/hardy/anacron/hardy-updates
~ubuntu-branches/ubuntu/hardy/anacron/hardy-proposed
~ubuntu-branches/ubuntu/gutsy/anacron/gutsy
~ubuntu-branches/ubuntu/feisty/anacron/feisty
~ubuntu-branches/ubuntu/edgy/anacron/edgy
~ubuntu-branches/ubuntu/breezy/anacron/breezy
~ubuntu-branches/ubuntu/hoary/anacron/hoary
~ubuntu-branches/debian/sid/antlr/sid
~ubuntu-branches/debian/sid/aspell/sid
~ubuntu-branches/debian/sid/atk1.0/sid
~ubuntu-branches/debian/sid/awstats/sid
~ubuntu-branches/debian/sid/bogofilter/sid
~ubuntu-branches/debian/sid/boost1.38/sid
~ubuntu-branches/debian/sid/bsh/sid
~ubuntu-branches/debian/sid/cairo/sid
~ubuntu-branches/debian/sid/cairomm/sid
~ubuntu-branches/debian/sid/camlp5/sid
~ubuntu-branches/debian/sid/cmake/sid
~ubuntu-branches/debian/sid/dhcp3/sid
~ubuntu-branches/ubuntu/gutsy/dhcp3/gutsy
~ubuntu-branches/debian/sid/directfb/sid
~ubuntu-branches/debian/sid/djvulibre/sid
~ubuntu-branches/debian/sid/docbook-xsl/sid
~ubuntu-branches/debian/sid/dsdo/sid
~ubuntu-branches/debian/sid/elfutils/sid
~ubuntu-branches/debian/sid/emacs22/sid
~ubuntu-branches/debian/sid/epydoc/sid
~ubuntu-branches/debian/sid/espa-nol/sid
~ubuntu-branches/ubuntu/edgy/espa-nol/edgy
~ubuntu-branches/debian/sid/exiv2/sid
~ubuntu-branches/debian/sid/ffmpeg-debian/sid
~ubuntu-branches/ubuntu/intrepid/firefox-3.0/intrepid-updates
~ubuntu-branches/debian/sid/gawk/sid
~ubuntu-branches/debian/sid/gnome-vfs/sid
~ubuntu-branches/debian/sid/gnumeric/sid
~ubuntu-branches/debian/sid/graphviz/sid
~ubuntu-branches/debian/sid/gsl/sid
~ubuntu-branches/debian/sid/gutenprint/sid
~ubuntu-branches/debian/sid/hevea/sid
~ubuntu-branches/debian/lenny/icon/lenny
~ubuntu-branches/debian/sid/igaelic/sid
~ubuntu-branches/ubuntu/lucid/kde-l10n-ta/lucid
~ubuntu-branches/debian/sid/kipi-plugins/sid
~ubuntu-branches/debian/sid/libgdiplus/sid
~ubuntu-branches/debian/sid/libxslt/sid
~ubuntu-branches/debian/sid/moin/sid
~ubuntu-branches/debian/lenny/moin/lenny
~ubuntu-branches/debian/sid/ntfs-3g/sid
~ubuntu-branches/debian/sid/trousers/sid
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Testing package imports locally

[James Westby]

On Wed, 17 Feb 2010 11:30:45 -0600, John Arbash Meinel  
wrote:
> When I tried it on 'gnome-panel' it had this error:
> 54.204  Traceback (most recent call last):
>   File "./import_package.py", line 769, in main
> assert len(versions.plist) > 0
>   File "./import_package.py", line 211, in get_versions
> publications = icommon.lp_call(icommon.call_with_limited_size,
>   File "./import_package.py", line 172, in get_debian_versions
> output, errout = proc.communicate()
>   File "/usr/lib/python2.6/subprocess.py", line 621, in __init__
> errread, errwrite)
>   File "/usr/lib/python2.6/subprocess.py", line 1126, in _execute_child
> raise child_exception
> OSError: [Errno 2] No such file or directory

I don't like this error, not helpful at all for working out what is
wrong, leading me to want to wrap this every time I use it.

> Looking at the code, it seems we also depend on 'madison-lite' being
> installed. Also, I ran into some bugs wrt not installing
> 'trace.enable_default_logging()' which meant it wasn't able to tell me
> what was failing when a plugin failed to load.

I don't follow, but I'll understand when I see the merge proposal.

> I'll submit some small patches that should help this a little bit at least.

Thanks.


James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Testing package imports locally

[James Westby]

On Wed, 17 Feb 2010 11:57:00 -0600, John Arbash Meinel  
wrote:
> By the way, should this be split out into its own LP project? It
> certainly isn't part of the bzr-builddeb codebase.
> (bzr-builddeb-import-scripts ?)

It could be. It could be a new project, or under udd.

Aside from philosphical reasons, what would the benefits be?

I'm happy to do it if there is a justification, but having another place
to check for merge proposals, bug reports, answers etc. would be a cost
I would't want otherwise.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Merging from Debian with patches applied

[James Westby]

On Tue, 16 Feb 2010 16:19:20 -0600, John Arbash Meinel  
wrote:
> To make sure I understand. If you use quilt, it moves the location on
> the patch stack (by reverse applying the patches). Leaving you with a
> working dir that doesn't have the 'newer' patches applied. Then if you
> change stuff and "build", it recomputes the current patch.

Yes, I'm not sure if that's automatic, or whether you are expected to
"refresh" and then "pop" back to the top of the stack.

> I believe this means that you get a single snapshot with all patches,
> and a list of patches that are managed by quilt, right? (What I mean is
> that we don't import the series of patches, just the tip.)

Yes, you get a single commit where the tree is the code with all the
patches applied, and also contains a directory containing those patches.

> Wouldn't it also potentially run into problems with conflicts in the
> patch files themselves?

Indeed, and they are a huge pain to resolve.

> When you say "first patch" you mean the first patch against the
> orig.tar.gz, right? And IIRC the issue is that Ubuntu's orig.tar.gz may
> be a different version that debians.

Yes, but there can be other reasons that cause the patch to require
fuzz.

> Well, trying to integrate a DVCS with a patch workflow is a bit tricky
> regardless. Because you don't know where in history a given patch would
> apply. Consider if 2 patches overlapped in a section. (eg debian
> provides a feature, and Ubuntu provides a bugfix for that feature.)

Yes.

> If we handled the patches as actual commits, then in theory we could
> tell more accurately what part of the updates apply to what patch.
> (Since they would be directly associated.)

Yes.

> Now I guess in both debian and ubuntu all changes vs orig.tar.gz have to
> be represented by a patch. So in theory the 'merge' operation could
> start at the beginning, and merge the 'orig.tar.gz' changes. Then it can
> check to see if the first patch is identical on both sides, and if so,
> skip forward (applying the patch). If they aren't identical, it could
> ask the user do you want to:

> a) apply just patch A and leave patch B in the queue to be applied later
> b) apply just patch B and leave patch A in the queue to be applied later
> c) apply patch A and remove patch B (patch A supersedes B)
> d) apply patch B and remove patch A (patch B supersedes A)
> e) apply both to create a new patch C

> for (e) you may also have conflicts you want to resolve. A different UI
> would be:

> a) apply patch A to WT (and step the queue)
> b) apply patch B to WT (and step the queue)
> c) delete patch A from the queue
> d) delete patch B from the queue
> e) edit WT

> The idea is that you decide which patch gets applied next as you
> regenerate the patch series. This could actually interact fairly well
> with bzr commits / a loom, but for starters it could just be an
> interactive patch resolution.

That could work. However, not every change *has* to be represented as a
quilt patch, but the v3 format enforces that (I'm pretty sure, I haven't
tried to break it).

> I would probably have it start with a line like:

> patch A applies (exactly/with fuzz), patch B applies (exactly/with fuzz)
> patch A and patch B are identical

> Then you would do
> a d

> (apply one and delete the other from the queue)

> If it applied with fuzz, I assume you would regenerate the patch at that
> point.

> If the patches aren't identical, then you probably also want a few
> possible displays:

> 1) show patch A as unidiff
> 2) show patch B as unidiff
> 3) show differences between A & B

> All in all, this seems like a good place for a nice gui, since there is
> enough stuff going on that you'd really like to have a fair amount of
> assistance sorting it all out.

I think that would be nice to have.

> Does quilt support 'merging' 2 patch queues?

Not to my knowledge.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Testing package imports locally

[James Westby]

Some updates since I sent the mail last time.

On Wed, 17 Feb 2010 12:02:48 +1100, Robert Collins  
wrote:
> You can trigger a run with
> 
>   ./import_package.py --no-push 
> 
> (The --no-push ensures that it doesn't try and push to LP if it
>  succeeds.)

There is now also

  --no-existing

to do a fresh import ignoring what is on LP already. I've found that
most of the " is not found in " type errors that suggest
repository corruption don't reproduce on a fresh import, suggesting that
they are bugs fixed in newer bzr, or are caused by the act of pushing to
LP.

There is also

  --check

to run check on the repository as it finishes. However, this just puts
the info in the log as the check_dwim API doesn't tell you if any
problems were detected as far as I could see.

> This will create a number of directories in your cwd. "updates"
> is where the work happens, and where the final branches end up.
> "revids" is where the audit data is stored. "logs" contains a log
> file per-package, you can tail it to watch progress. There are
> a few more of lesser importance.

You may need to create ./logs/diffs and ./logs/merges as well, I don't
think I made it create those if they didn't exist.

> Current failures are at
> 
>   http://package-import.ubuntu.com/failures/.bzr/failures/

As stated elsewhere this is now

  http://package-import.ubuntu.com/status/


Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Merging from Debian with patches applied

[James Westby]

Hi,

There are some interesting edge cases emerging from our experience of
merging the new source package formats from Debian.

In short these formats ship the patches as a quilt series in
./debian/patches, but apply them when the source package is unpacked.
If you then edit something and build it creates a new patch for you on
top. If you want to modify an existing patch then you can use quilt to
move to that patch and make your changes.

When importing these packages we stick to the "bzr branch gets you the
same as apt-get source" mantra and import with the patches applied.

Now when you merge from Debian in to Ubuntu you get bzr's merge
capabilities helping you merge all the changes, which is great, but it
doesn't help you change the contents of the patches that are stored.

The first problem this causes is that you then might not be able to
build the package as the first patch may now only apply with fuzz (which
dpkg checks).

Another issue is that the conflict resolution you do may in fact be
altering one of the patches in the stack and so you would want to be
doing it there, but that's not the obvious thing to do.

This isn't a problem specific to bzr, the scheme chosen by Debian isn't
helpful to derivatives and so any method would have some of these
issues, but we should fix it, even if it's in a bzr-specific way (though
greater applicability would be useful).

I think we need to look at semi-automatically refreshing the stack of
patches, but I'm not entirely sure how to go about this.

Ideas for how we can improve this situation would be welcome.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[James Westby]

On Mon, 15 Feb 2010 19:19:13 +1100, Martin Pool  wrote:
> So, just to be sure, we want to upgrade all the non-2a ones to 2a, on
> Launchpad?  Probably by running the upgrade either on the lp machine
> itself or at least on a nearby machine?

Yep.

Differing formats cause pain, and there's no need for these to be in an
inferior format.

Doing the upgrade from the codehosting machine would be much more
efficient for these three thousand branches.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD as a product?

[James Westby]

On Sat, 13 Feb 2010 07:06:37 +1000, Ian Clatworthy 
 wrote:
> > Would these differ from the
> > "phases" in the overall specification?
> 
> Is there a URL for this?

  https://wiki.ubuntu.com/DistributedDevelopment/Specification

(linked from https://wiki.ubuntu.com/DistributedDevelopment)

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD as a product?

[James Westby]

On Thu, 11 Feb 2010 17:59:14 +1000, Ian Clatworthy 
 wrote:
> UDD now has an active mailing list, a Launchpad project and a bug/task
> list. Does it make sense to begin thinking about UDD as a product? Would
> it be valuable to talk about UDD x.y vs x.z?

By x.y and x.z do you mean versions? Would these differ from the
"phases" in the overall specification?

> Code wise, I guess the "product" is a mix of LP features, Bazaar
> features and Bazaar plugins. OTOH, those things come together to form a
> system.

Yeah, we usually associate products with a codebase.

> There's a huge amount of wisdom being imparted each week on this list.
> Perhaps we should turn some of the threads into tutorials (either in a
> wiki or bzr branch) or a "UDD Hackers Guide" say. Is it too early for that?

I think that would be very valuable. I am behind on the documentation
and would love for others to contribute things as they learn them.

Are you proposing specific changes in the way we work, or just a shift
in thinking?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD @ Portland

[James Westby]

On Thu, 11 Feb 2010 13:33:27 +1100, Martin Pool  wrote:
> I'd like to let looms progress, but not (unless james or others feel
> differently) add them into the dependency chain for getting UDD going.

No, we don't have to add it to the chain to get it going, but I think
it's one ingredient of having a great system.

>  iow people should be able to try them on particular branches, without
> mandating them for all package branches, and (perhaps?) without
> requiring everyone working on that package to use them.

I think a gradual migration path is something to aim for. What we want
is consistency of interaction. I don't want to have to work out what is
going on in the packaging branch before I can start work on it. Allowing
"branch; hack; build; push" regardless of what's going on and allowing
others to delve more deeply is one way, another would be to have "bzr
add-patch" or something that prepared the tree for working, I'm sure
there are more.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD @ Portland

[James Westby]

On Thu, 11 Feb 2010 13:18:30 +1100, Robert Collins 
 wrote:
> James Westby and I had some time together in Portland to talk about UDD
> stuff.

Yes, it was good to have the time, thanks for coming and for sending
this mail.

> Firstly though, a couple of overview points:
>  - the udd project has 200 bugs on it. While many of these are
> 'collision' reports many are not. The collision reports are currently
> overly noisy, so please ignore them for now. However, the other bugs are
> open season for people to fix - and every bug fixed there will
> streamline things for people using bzr to package in Ubuntu.

They are now reported as merge proposals if it's an Ubuntu branch, so
that should stop the list growing too much.

>  - Measurement error: the hottest 100 has a fairly high error rate for
> package imports at the moment, OTOH its looking at precisely the
> packages most likely to fail. James says that overall 96% or so of
> packages import successfully.

They are most likely to fail as they will tend to be larger, uploaded
more often etc: making them more likely to trigger bugs.

> Ok, onto the fun stuff. While long term we want a Launchpad control
> panel for the package importer, James thinks its reasonable that folk
> helping with the operation of it should be able to directly kick it off
> - so he has filed an RT ticket to get more access to that machine.
> James, whats the RT ticket number? Opening this up will let us rerun
> imports more promptly that appear to have had only spurious failures.

#37368

Feel free to drop me an email with the names of packages you think
should be retried in the meantime. I'll do it the next time I read my
email. It's no good for debugging issues, but it's not the best way to
do it even when you can do it straightaway. Remember that you can run
exactly the same code locally and so make use of bzr, pdb, etc. to
investigate.

> Bugs with the importer can and should be debugged on peoples development
> environment - there is an earlier mail from December documenting how to
> do this. We should put that in the Wiki I think think.

Good idea.

> The collisions that are reported as bugs can be divided into three broad
> groups:
>  - impossible (a collision in debian: at least at the moment, we don't
> expect people uploading to Debian packaging-branches. Well, *generally
> speaking* we don't expect this). (Nb: I do it for stuff I maintain in
> Debian :)
>  - spurious (its not a collision, and a bug caused it)
>  - genuine (it is a collision and it should be a merge proposal)
> 
> We have a few collision specific tasks:
>  - James is rapidly making new collisions be filed as merge proposals
> (unless they are in Debian imports)

Done, but with some slight issues due to the LP API and other
things. They are being filed as merge proposals now.

>  - we need to write a script to analyse the nearly 200 collisions in the
> bug tracker to highlight the debian imports (must be bugs, might be
> fixed), and convert the ubuntu ones to merge proposals.
>  - We should delete the stale branches for collisions that we decide are
> bogus. Membership in the magic group ubuntu-branches is needed for that,
> and that group needs to be kept locked down (as it is equivalent to
> upload rights to the archive). So - lets make a list somewhere if you
> determine a branch isn't needed, and ping James or anyone on the tech
> board to delete such a branch).

I'd say comment in the bug report for it. It has all the info I need and
I can do it the next time I read mail.

> We looked at the workflow involved in packaging, and I'm very happy that
> James has seen the light and will be implementing an 'import-upstream'
> command to import and make a tarball micro-branch but not do the debian
> metadata updates. This will be useful for looms, where the two steps
> occur on different threads.

It's currently spelt "bzr dh_make", "import-upstream" would be "bzr
dh_make --bzr-only". When we get a workflow going with looms we can look
at how we it fit in there. I didn't want to have "import-upstream"
straight away as I didn't want confusion arising from the fact that you
can run it in a packaging branch and so delete all the packaging.

> Finally we looked at Looms with mathiaz who is hoping to get the MySQL
> packages in Looms for both Debian and Ubuntu. We identified some rough
> spots and a missing command (import-upstream) but it seems doable, if
> not /nice/ today. After that we talked about a sparser loom merge graph.
> 
> The basic idea I have is that while the stack seems essential to
> providing a simple UI, all the merge commits make a lot of noise. So if
> we only do a merge commit when a conflict has happened, and ot

Re: import failures

[James Westby]

On Thu, 04 Feb 2010 14:57:25 +1300, Michael Hudson 
 wrote:
> James Westby wrote:
> > On Wed, 06 Jan 2010 09:41:17 +1300, Michael Hudson 
> >  wrote:
> >> James Westby wrote:
> >>
> >>>> Is it possible to get a query of old ones, and just run a bulk-update of
> >>>> them?
> >>> I have the list of packages, and mwhudson was going to query for the
> >>> list of branches based on that, and then request server-side upgrade I
> >>> believe.
> >> Well, I've managed to completely drop the ball on this :(
> > 
> > No problem. I could have done much of it myself.
> > 
> >> Can you send me the list of packages again?
> > 
> > Attached.
> 
> Once again, I've not done anything here... can you send me an updated
> list?  Some of the ones from that list are in 2a format and some not, so
> if you have an up-to-date list it'll make things a bit easier for me.

Sorry, forgot to respond to this.

Some of them have been upgraded. If it's easier for me to do an info
against all of them and filter out those not in 2a then I can do so.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel