Re: New overview page for the Bazaar importer
On Fri, 15 Jan 2010 10:50:39 -0500, "Francis J. Lacoste" wrote: > Looking at the signature today, I only see the first one as network related: > 2 packages failed to many times to retry with key > launchpadlib.errors.HTTPError::main:get_versions:iterate_collection:get_collection_slice:get:_request > Being discussed in https://bugs.edge.launchpad.net/soyuz/+bug/513491 It's not network related, but is often a transient error on the LP side. The soyuz team have been helping to reduce the number of timeouts we get, and it is much better now. "too many times to retry" meant that they were auto-retried several times, but they hit the limit and so are awaiting investigation in case they are not transient errors. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: New overview page for the Bazaar importer
On Thu, 14 Jan 2010 16:58:29 -0600, John Arbash Meinel
wrote:
> So ignoring the 'latest 100' failures,
That's there to see if there are new issues occuring. It was more useful
before auto-retry was added for some LP failures.
> the top 2 failures are:
> 182 packages failed with key
> AssertionError::main:find_unimported_versions
>
> Which when poking at one looks like:
> AssertionError: abiword 1.0.2+cvs.2002.06.05-1woody2 debian woody is
> marked but not imported
>
> Poking around I see debian woody, debian sid, ubuntu warty as failing on
> those. This certainly sounds like just bad configuration.
Let me explain the parts of this.
When the importer sees a version for the first time it will record a
tuple of (distro, suite, version, revid, testamant sha) for audit
purposes. It refers to this as "marking."
It also has checks to ensure that the tuple matches the data every time
it considers the package. In this case it is complaining as it appears
like the revision has been deleted from the branch, which would be bad.
When this was first implemented it wasn't transactional. This meant that
even if the import failed it was marked, so when you retried it it would
fail. That's fixed, but the old data wasn't purged entirely. This is a
case that never worked.
A little while ago I accidentally retried a bunch of packages I
shouldn't have as they weren't problems which were fixed, including
abiword. They failed in this manner, but it is masking the real error.
It needs to be checked to see if it was in the case that never worked
and if so do a full retry to re-instate the original error. I don't want
to do that wholesale without checking though.
> Number 2 is
> 89 packages failed with key
> AssertionError::main:import_package:import_package:extract
>
> And all the ones I poked at were:
> File
> "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
> line 1802, in extract
> "Can't handle non gz tarballs yet"
> AssertionError: Can't handle non gz tarballs yet
Colin asked me to increase the priority of this today.
> The next 3 failures (34 + 34 + 32 = 100, so it should be #2?), are all
> UnicodeDecodeError.
> The first set are all 'author.decode', the next is path issues (probably
> non-ascii paths in the dataset). The next three are all
> "find_extra_authors" which is breaking down at "change.decode('UTF-8')".
For find_extra_authors we should catch the errors and skip I guess.
For paths, I'm not sure what the best solution is.
For author.decode, I'm not sure if we should skip, perhaps add a
translation from byte-string->unicode string for known problematic byte
strings?
> Next is 21 diverged branches...
I figured out what the issue is here and I am considering what the best
way to fix it is.
Basically the collision code forgets to set overwrite=True when
pushing. This has highlighted that the collision code is a little too
strict about how it calcualtes collisions though, which is what needs to
be considered.
> Next is 16 different serializations (need to be upgraded?) ...
Yes.
> And then a fairly long tail.
> All of this doesn't really look like stuff to do on the bzr side.
> Decoding the changelogs is something to investigate, supporting non .gz
> tarballs, and figuring out what to do when something wants to import a
> distro version that is "marked but not imported" (which I don't claim to
> understand).
Thanks for looking in to it, I hope my explanations are useful.
Thanks,
James
--
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: New overview page for the Bazaar importer
On Mon, 01 Feb 2010 20:10:15 +, Max Bowsher wrote: > On 08/01/10 00:52, James Westby wrote: > > Hi, > > > > After a few days spent doing some infrastructure work I'm now about to > > present a more coherent interface to the current status of the bzr > > importer. You can find it at > > > >http://package-import.ubuntu.com/failures/.bzr/failures/index.html > > This seems to have gone away. Where does it live now? Yes, sorry, the sysadmins accidentally obscured it by cleaning up some redirections as they didn't realise it was there. It does mean we can have sensible URLs now though :-) http://package-import.ubuntu.com/status/ I've added an index.html at the top level as well now, which should make it easier to find. I also added http://package-import.ubuntu.com/merges/ http://package-import.ubuntu.com/diffs/ which, while the presentation is really poor, aim to be replacements for patches.ubuntu.com and merges.ubuntu.com. I don't think they are working correctly yet, suggestions for how to imporove it (or even patches) welcome. James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: New overview page for the Bazaar importer
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Max Bowsher wrote: > On 08/01/10 00:52, James Westby wrote: >> Hi, >> >> After a few days spent doing some infrastructure work I'm now about to >> present a more coherent interface to the current status of the bzr >> importer. You can find it at >> >>http://package-import.ubuntu.com/failures/.bzr/failures/index.html > > This seems to have gone away. Where does it live now? > > Thanks, > Max. > So there is still: http://package-import.ubuntu.com/failures/ Which at least gives the individual failure listing, even if it isn't a pretty html page. We used it last week when we were working on the hottest100 stuff. John =:-> -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktnS/gACgkQJdeBCYSNAAOA4gCggozahvbJ2pQm2qAY9H90oy5Q Jf0AnRVQIcq4FO7DMwVN35hyUWi7bJ09 =jdS5 -END PGP SIGNATURE- -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: New overview page for the Bazaar importer
On 08/01/10 00:52, James Westby wrote: > Hi, > > After a few days spent doing some infrastructure work I'm now about to > present a more coherent interface to the current status of the bzr > importer. You can find it at > >http://package-import.ubuntu.com/failures/.bzr/failures/index.html This seems to have gone away. Where does it live now? Thanks, Max. -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
