Re: Import failures in main

[James Westby]

On Sun, 03 Oct 2010 17:17:59 -0400, James Westby  
wrote:
> > I then turned this in to an almost-live page at
> > 
> >   http://package-import.ubuntu.com/status/main.html
> > 
> > (up to 5 minutes behind the current action)
> 
> Now down to 195 failures.

Now you can see at the bottom of

  http://package-import.ubuntu.com/status/main.html
  http://package-import.ubuntu.com/status/index.html

graphs of the queue size and failure count. They are not very
interesting right now, but should hopefully show some interesting
information after they have had some time to collect data.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Import failures in main

[James Westby]

On Sat, 02 Oct 2010 22:29:57 -0400, James Westby  
wrote:
> With 228 failures and 29 bugs there is quite a skewed distribution, and
> the two bugs causing the most failures are:
> 
>   "Packages failing due to pristine-tar not being able to reconstruct
>   their tarball" - https://bugs.launchpad.net/udd/+bug/653301

I'd forgotten that I had already reported this and Joey fixed the likely
cause of at least most of them. We just need to get the deployed to
jubany now (possibly test beforehand to ensure that most are indeed
fixed, but I think it's highly likely as he was testing with some of our
examples)

> and
> 
>   "Import fails with missing referenced chk root keys" -
>   https://bugs.edge.launchpad.net/udd/+bug/653307

Added some observations to this, but I'm unsure what bzr is telling us
here. Some help with interpreting that would be great.

I've also bumped the bug to critical as it looks like fairly widespread
repo corruption from an unknown source.

> The reason that the first is so high is that something seems to have
> changed recently that means that most of the KDE packages can no longer
> be handled by pristine tar.

I'm guessing the reason was either a switch to .bz2, or a change to the
machine on which the tarballs are generated, based on the Debian bug
logs.

> I then turned this in to an almost-live page at
> 
>   http://package-import.ubuntu.com/status/main.html
> 
> (up to 5 minutes behind the current action)

Now down to 195 failures.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[James Westby]

On Thu, 18 Feb 2010 12:54:01 +1100, Robert Collins  
wrote:
> Ah:
> 
> CHKInventoryRepository('lp-hosted:///~ubuntu-branches/debian/sid/camlp5/sid/.bzr/repository',
>  
> fallback_repositories=[CHKInventoryRepository('lp-hosted:///~ubuntu-branches/debian/squeeze/camlp5/squeeze/.bzr/repository')])
>  has no revision james.wes...@ubuntu.com-20091029134422-acd9dzuxa44whcct
> 
> We have a bug open on this - I think we should make it critical.

bug 507040 was the only one that looked similar on a search, is that the
one?

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[Robert Collins]

On Thu, 2010-02-18 at 14:46 +1300, Michael Hudson wrote:
> Robert Collins wrote:
> > On Thu, 2010-02-18 at 13:22 +1300, Michael Hudson wrote:
> >> James Westby wrote:
> >>> On Wed, 17 Feb 2010 08:14:26 +1300, Michael Hudson 
> >>>  wrote:
>  For one reason and another, Tim ran the script in the end, but it looks
>  like it's finished now but you might want to rerun your script to check.
> >>> Thanks to both of you.
> >>>
> >>> However, the attached didn't seem to "stick". Only 57 branches this time
> >>> though.
> >> So a bit of poking has got almost all of these up to 2a.  There are 5
> >> that seem to be broken:
> > 
> > Details please.
> 
> Clicking the links will get you as many details as I have.

Ah:

CHKInventoryRepository('lp-hosted:///~ubuntu-branches/debian/sid/camlp5/sid/.bzr/repository',
 
fallback_repositories=[CHKInventoryRepository('lp-hosted:///~ubuntu-branches/debian/squeeze/camlp5/squeeze/.bzr/repository')])
 has no revision james.wes...@ubuntu.com-20091029134422-acd9dzuxa44whcct

We have a bug open on this - I think we should make it critical.

-Rob


signature.asc
Description: This is a digitally signed message part
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[Michael Hudson]

Robert Collins wrote:
> On Thu, 2010-02-18 at 13:22 +1300, Michael Hudson wrote:
>> James Westby wrote:
>>> On Wed, 17 Feb 2010 08:14:26 +1300, Michael Hudson 
>>>  wrote:
 For one reason and another, Tim ran the script in the end, but it looks
 like it's finished now but you might want to rerun your script to check.
>>> Thanks to both of you.
>>>
>>> However, the attached didn't seem to "stick". Only 57 branches this time
>>> though.
>> So a bit of poking has got almost all of these up to 2a.  There are 5
>> that seem to be broken:
> 
> Details please.

Clicking the links will get you as many details as I have.

Cheers,
mwh

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[Robert Collins]

On Thu, 2010-02-18 at 13:22 +1300, Michael Hudson wrote:
> James Westby wrote:
> > On Wed, 17 Feb 2010 08:14:26 +1300, Michael Hudson 
> >  wrote:
> >> For one reason and another, Tim ran the script in the end, but it looks
> >> like it's finished now but you might want to rerun your script to check.
> > 
> > Thanks to both of you.
> > 
> > However, the attached didn't seem to "stick". Only 57 branches this time
> > though.
> 
> So a bit of poking has got almost all of these up to 2a.  There are 5
> that seem to be broken:

Details please.

-Rob


signature.asc
Description: This is a digitally signed message part
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[Michael Hudson]

James Westby wrote:
> On Wed, 17 Feb 2010 08:14:26 +1300, Michael Hudson 
>  wrote:
>> For one reason and another, Tim ran the script in the end, but it looks
>> like it's finished now but you might want to rerun your script to check.
> 
> Thanks to both of you.
> 
> However, the attached didn't seem to "stick". Only 57 branches this time
> though.

So a bit of poking has got almost all of these up to 2a.  There are 5
that seem to be broken:

https://code.edge.launchpad.net/~ubuntu-branches/debian/sid/camlp5/sid
https://code.edge.launchpad.net/~ubuntu-branches/debian/sid/directfb/sid
https://code.edge.launchpad.net/~ubuntu-branches/debian/sid/djvulibre/sid
https://code.edge.launchpad.net/~ubuntu-branches/debian/sid/elfutils/sid
https://code.edge.launchpad.net/~ubuntu-branches/debian/lenny/moin/lenny

I don't know what's happened to these.  It's possible they were broken
somehow before the upgrade I guess...

Not sure what the next step for these is.

Cheers,
mwh

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[James Westby]

On Wed, 17 Feb 2010 08:14:26 +1300, Michael Hudson 
 wrote:
> For one reason and another, Tim ran the script in the end, but it looks
> like it's finished now but you might want to rerun your script to check.

Thanks to both of you.

However, the attached didn't seem to "stick". Only 57 branches this time
though.

Thanks,

James

~ubuntu-branches/debian/sid/aiksaurus/sid
~ubuntu-branches/debian/squeeze/anacron/squeeze
~ubuntu-branches/debian/lenny/anacron/lenny
~ubuntu-branches/ubuntu/lucid/anacron/lucid
~ubuntu-branches/ubuntu/karmic/anacron/karmic
~ubuntu-branches/ubuntu/jaunty/anacron/jaunty
~ubuntu-branches/ubuntu/intrepid/anacron/intrepid
~ubuntu-branches/ubuntu/hardy/anacron/hardy
~ubuntu-branches/ubuntu/hardy/anacron/hardy-updates
~ubuntu-branches/ubuntu/hardy/anacron/hardy-proposed
~ubuntu-branches/ubuntu/gutsy/anacron/gutsy
~ubuntu-branches/ubuntu/feisty/anacron/feisty
~ubuntu-branches/ubuntu/edgy/anacron/edgy
~ubuntu-branches/ubuntu/breezy/anacron/breezy
~ubuntu-branches/ubuntu/hoary/anacron/hoary
~ubuntu-branches/debian/sid/antlr/sid
~ubuntu-branches/debian/sid/aspell/sid
~ubuntu-branches/debian/sid/atk1.0/sid
~ubuntu-branches/debian/sid/awstats/sid
~ubuntu-branches/debian/sid/bogofilter/sid
~ubuntu-branches/debian/sid/boost1.38/sid
~ubuntu-branches/debian/sid/bsh/sid
~ubuntu-branches/debian/sid/cairo/sid
~ubuntu-branches/debian/sid/cairomm/sid
~ubuntu-branches/debian/sid/camlp5/sid
~ubuntu-branches/debian/sid/cmake/sid
~ubuntu-branches/debian/sid/dhcp3/sid
~ubuntu-branches/ubuntu/gutsy/dhcp3/gutsy
~ubuntu-branches/debian/sid/directfb/sid
~ubuntu-branches/debian/sid/djvulibre/sid
~ubuntu-branches/debian/sid/docbook-xsl/sid
~ubuntu-branches/debian/sid/dsdo/sid
~ubuntu-branches/debian/sid/elfutils/sid
~ubuntu-branches/debian/sid/emacs22/sid
~ubuntu-branches/debian/sid/epydoc/sid
~ubuntu-branches/debian/sid/espa-nol/sid
~ubuntu-branches/ubuntu/edgy/espa-nol/edgy
~ubuntu-branches/debian/sid/exiv2/sid
~ubuntu-branches/debian/sid/ffmpeg-debian/sid
~ubuntu-branches/ubuntu/intrepid/firefox-3.0/intrepid-updates
~ubuntu-branches/debian/sid/gawk/sid
~ubuntu-branches/debian/sid/gnome-vfs/sid
~ubuntu-branches/debian/sid/gnumeric/sid
~ubuntu-branches/debian/sid/graphviz/sid
~ubuntu-branches/debian/sid/gsl/sid
~ubuntu-branches/debian/sid/gutenprint/sid
~ubuntu-branches/debian/sid/hevea/sid
~ubuntu-branches/debian/lenny/icon/lenny
~ubuntu-branches/debian/sid/igaelic/sid
~ubuntu-branches/ubuntu/lucid/kde-l10n-ta/lucid
~ubuntu-branches/debian/sid/kipi-plugins/sid
~ubuntu-branches/debian/sid/libgdiplus/sid
~ubuntu-branches/debian/sid/libxslt/sid
~ubuntu-branches/debian/sid/moin/sid
~ubuntu-branches/debian/lenny/moin/lenny
~ubuntu-branches/debian/sid/ntfs-3g/sid
~ubuntu-branches/debian/sid/trousers/sid
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[Michael Hudson]

Michael Hudson wrote:
> James Westby wrote:
>> On Mon, 15 Feb 2010 19:19:13 +1100, Martin Pool  wrote:
>>> So, just to be sure, we want to upgrade all the non-2a ones to 2a, on
>>> Launchpad?  Probably by running the upgrade either on the lp machine
>>> itself or at least on a nearby machine?
>> Yep.
>>
>> Differing formats cause pain, and there's no need for these to be in an
>> inferior format.
>>
>> Doing the upgrade from the codehosting machine would be much more
>> efficient for these three thousand branches.
> 
> I'm upgrading the branches using a stupid shell script on devpad now.

For one reason and another, Tim ran the script in the end, but it looks
like it's finished now but you might want to rerun your script to check.

Cheers,
mwh

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[Michael Hudson]

James Westby wrote:
> On Mon, 15 Feb 2010 19:19:13 +1100, Martin Pool  wrote:
>> So, just to be sure, we want to upgrade all the non-2a ones to 2a, on
>> Launchpad?  Probably by running the upgrade either on the lp machine
>> itself or at least on a nearby machine?
> 
> Yep.
> 
> Differing formats cause pain, and there's no need for these to be in an
> inferior format.
> 
> Doing the upgrade from the codehosting machine would be much more
> efficient for these three thousand branches.

I'm upgrading the branches using a stupid shell script on devpad now.

Cheers,
mwh

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[James Westby]

On Mon, 15 Feb 2010 19:19:13 +1100, Martin Pool  wrote:
> So, just to be sure, we want to upgrade all the non-2a ones to 2a, on
> Launchpad?  Probably by running the upgrade either on the lp machine
> itself or at least on a nearby machine?

Yep.

Differing formats cause pain, and there's no need for these to be in an
inferior format.

Doing the upgrade from the codehosting machine would be much more
efficient for these three thousand branches.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[Martin Pool]

On 13 February 2010 04:09, James Westby  wrote:
> On Thu, 11 Feb 2010 12:49:22 +1100, Robert Collins 
>  wrote:
>> On Wed, 2010-02-10 at 21:46 +, James Westby wrote:
>> >
>> > Some of them have been upgraded. If it's easier for me to do an info
>> > against all of them and filter out those not in 2a then I can do so.
>>
>> I think thats easiest.
>
> For some value of easy that involves writing a python script and running
> it for four hours to extract information from the LP database :-)
>
> Attached is a raw list of branch API url and repository format string,
> it can be massaged in to a list of branches to upgrade using your
> preferred text manipulation tools.
>
> 3336 branches by my reckoning.

So, just to be sure, we want to upgrade all the non-2a ones to 2a, on
Launchpad?  Probably by running the upgrade either on the lp machine
itself or at least on a nearby machine?

-- 
Martin 

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[Robert Collins]

On Wed, 2010-02-10 at 21:46 +, James Westby wrote:
> 
> Some of them have been upgraded. If it's easier for me to do an info
> against all of them and filter out those not in 2a then I can do so. 

I think thats easiest.


signature.asc
Description: This is a digitally signed message part
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[James Westby]

On Thu, 04 Feb 2010 14:57:25 +1300, Michael Hudson 
 wrote:
> James Westby wrote:
> > On Wed, 06 Jan 2010 09:41:17 +1300, Michael Hudson 
> >  wrote:
> >> James Westby wrote:
> >>
>  Is it possible to get a query of old ones, and just run a bulk-update of
>  them?
> >>> I have the list of packages, and mwhudson was going to query for the
> >>> list of branches based on that, and then request server-side upgrade I
> >>> believe.
> >> Well, I've managed to completely drop the ball on this :(
> > 
> > No problem. I could have done much of it myself.
> > 
> >> Can you send me the list of packages again?
> > 
> > Attached.
> 
> Once again, I've not done anything here... can you send me an updated
> list?  Some of the ones from that list are in 2a format and some not, so
> if you have an up-to-date list it'll make things a bit easier for me.

Sorry, forgot to respond to this.

Some of them have been upgraded. If it's easier for me to do an info
against all of them and filter out those not in 2a then I can do so.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[Michael Hudson]

James Westby wrote:
> On Wed, 06 Jan 2010 09:41:17 +1300, Michael Hudson 
>  wrote:
>> James Westby wrote:
>>
 Is it possible to get a query of old ones, and just run a bulk-update of
 them?
>>> I have the list of packages, and mwhudson was going to query for the
>>> list of branches based on that, and then request server-side upgrade I
>>> believe.
>> Well, I've managed to completely drop the ball on this :(
> 
> No problem. I could have done much of it myself.
> 
>> Can you send me the list of packages again?
> 
> Attached.

Once again, I've not done anything here... can you send me an updated
list?  Some of the ones from that list are in 2a format and some not, so
if you have an up-to-date list it'll make things a bit easier for me.

Cheers,
mwh

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[James Westby]

On Wed, 06 Jan 2010 09:41:17 +1300, Michael Hudson 
 wrote:
> James Westby wrote:
> 
> >> Is it possible to get a query of old ones, and just run a bulk-update of
> >> them?
> > 
> > I have the list of packages, and mwhudson was going to query for the
> > list of branches based on that, and then request server-side upgrade I
> > believe.
> 
> Well, I've managed to completely drop the ball on this :(

No problem. I could have done much of it myself.

> Can you send me the list of packages again?

Attached.

Thanks,

James


aalib
acct
acl
acpica-unix
adduser
adns
aiksaurus
alien
anacron
analog
anthy
antlr
apex
app-install-data-partner
app-install-data-ubuntu
apparmor
appconfig
apport
apr
apr-util
apt-xapian-index
apturl
asciidoc
asio
aspell
aspell-en
aspell-tl
at
at-spi
atk1.0
atomix
attr
auctex
audiofile
auth-client-config
authbind
autoconf-nonfree
autoconf2.59
autofs
autogen
automake1.10
automoc
autotools-dev
awstats
ayaspell-dic
b43-fwcutter
babl
backuppc
bacula
bbdb
bc
beecrypt
beep
bf-utf
bgoffice
binfmt-support
bittornado
blt
bluez
bluez-gnome
bogl
bogofilter
bonnie++
boost1.38
br.ispell
bridge-utils
brltty
bsd-finger
bsd-mailx
bsdmainutils
bsh
bterm-unifont
byacc
byobu
bzip2
ca-certificates
cairo
cairomm
camlp5
ccache
cdbs
cdebconf-entropy
cdebconf-keystep
cdebconf-terminal
cdparanoia
cfitsio3
check
checkbox
checksecurity
chkrootkit
clamav
cli-common
cln
clock-setup
cloop
clucene-core
cmake
commons-daemon
commons-pool
compiz-fusion-bcop
compizconfig-backend-gconf
computer-janitor
consolekit
contact-lookup-applet
corosync
cppunit
crash
cryptsetup
culmus
cvsps
cwidget
cyrus-sasl2
db-defaults
dbconfig-common
dbs
dcraw
dctrl-tools
debhelper
debian-goodies
debianutils
debootstrap
dejagnu
desktop-effects-kde
device-tree-compiler
devicekit
devicekit-disks
devio
devscripts
dh-buildinfo
dh-ocaml
dhcp3
dia
dict-gcide
dict-jargon
dict-moby-thesaurus
dict-nr
dict-ns
dict-ss
dict-st
dict-tn
dict-ts
dict-ve
dict-xh
dictclient
dictd
dictdlib
dictionaries-common
diff-doc
diffstat
diffutils
digikam
directfb
djvulibre
dmake
dmz-cursor-theme
dnstracer
doc-base
docbook
docbook-dsssl
docbook-to-man
docbook-xml
docbook-xsl
docbook-xsl-doc
dosfstools
dpatch
dpsyco
drac
dsdo
dutch
dvd+rw-tools
dvipdfmx
dvipng
ecosconfig-imx
edubuntu-artwork
edubuntu-meta
efi-reader
efibootmgr
egenix-mx-base
eigen2
elfutils
elilo
emacs22
emacsen-common
enca
enchant
eo-spell
epydoc
espa-nol
ethtool
evolution-indicator
example-content
exim4
exiv2
expat
exuberant-ctags
facile
fakeroot
fast-user-switch-applet
fbset
ffmpeg-debian
file
findutils
finish-install
firefox-3.0
flac
flash-kernel
fltk1.1
flute-openoffice.org
fontforge
fortune-mod
freecdb
freeglut
friendly-recovery
fuse
gartoon
gawk
gawk-doc
gconfmm2.6
gegl
genext2fs
genromfs
gfxboot
ggz-server
giflib
git-core
gle
gmetadom
gnome-control-center
gnome-disk-utility
gnome-vfs
gnome-vfsmm2.6
gnu-efi
gnumeric
gob2
goffice
gperf
graphviz
gsfonts-x11
gsl
gtk-sharp2
gtkmm2.4
gtkspell
guile-1.8
gutenprint
hevea
hfsplus
hkgerman
html2text
human-theme
hwdata
icon
icu
id3lib3.8.3
ifenslave-2.6
igaelic
ilmbase
imhangul
indicator-applet
installation-guide
ipsec-tools
iptables
ispell-et
ispell-gl
jadetex
jakarta-log4j
jfsutils
jigit
jlex
jline
john
kde-l10n-ca
kde-l10n-cs
kde-l10n-ta
kde-l10n-tg
kde-l10n-th
kde-l10n-tr
kde-l10n-uk
kipi-plugins
konq-plugins
language-pack-ar-base
language-pack-ast-base
language-pack-da
language-pack-de
language-pack-dz
language-pack-el-base
language-pack-fil-base
language-pack-gnome-bg-base
language-pack-gnome-bs-base
language-pack-gnome-cs-base
language-pack-gnome-csb
language-pack-gnome-el
language-pack-gnome-en
language-pack-gnome-eo
language-pack-gnome-es
language-pack-gnome-et
language-pack-gnome-eu
language-pack-gnome-fa
language-pack-gnome-fi
language-pack-gnome-fo
language-pack-gnome-fr
language-pack-gnome-fr-base
language-pack-gnome-fy
language-pack-gnome-ga-base
language-pack-gnome-he-base
language-pack-gnome-it-base
language-pack-gnome-ka-base
language-pack-gnome-ko-base
language-pack-gnome-kw-base
language-pack-gnome-pap-base
language-pack-gnome-shs-base
language-pack-gnome-ti-base
language-pack-gnome-wa
language-pack-gnome-wo
language-pack-hr-base
language-pack-kde-an-base
language-pack-kde-cy
language-pack-kde-ga
language-pack-kde-gd
language-pack-kde-gl
language-pack-kde-gu
language-pack-kde-ia
language-pack-kde-id
language-pack-kde-is
language-pack-kde-it
language-pack-kde-ku-base
language-pack-kde-om-base
language-pack-kde-shs-base
language-pack-kde-so-base
language-pack-kde-ug
language-pack-kde-ur
language-pack-kde-uz
language-pack-lo-base
language-pack-oc-base
language-pack-ro
language-pack-ru
language-pack-te
language-pack-tg
language-pack-xh-base
language-support-et
language-support-fi
language-support-fonts-am
language-support-fonts-ar
language-support-fr
language-support-input-ar
language-support-input-gu
language-support-input-ml
language-support-ja
language-support-mnc
language-support-se
language-support-translations-fa
language-support-translatio

Re: import failures

[James Westby]

On Tue, 05 Jan 2010 18:03:48 +, James Westby  
wrote:
> I'll run a local test of this packages to see if it is associated with
> the data.

It is not. Either the bug has been fixed, or it was something about that
particular import that triggered it. Anyway, I can't provide a simple
testcase currently.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[Michael Hudson]

James Westby wrote:

>> Is it possible to get a query of old ones, and just run a bulk-update of
>> them?
> 
> I have the list of packages, and mwhudson was going to query for the
> list of branches based on that, and then request server-side upgrade I
> believe.

Well, I've managed to completely drop the ball on this :(

Can you send me the list of packages again?

Cheers,
mwh

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[James Westby]

On Tue, 05 Jan 2010 11:18:59 -0600, John Arbash Meinel  
wrote:
> ^- 'unsupported Unicode code range' sounds funny, but it may just be
> that they have latin-1 chars in what should otherwise be a UTF-8 doc. Is
> changelog *defined* as UTF-8? Or is it just '8-bit, put whatever feels
> good to you' in there?

It is UTF-8 now, but wasn't in the past, so some will be different
encodings when we are importing old packages.

> My guess is that you are handing us 8-bit paths, and inside bzrlib all
> *paths* are supposed to be Unicode. And if you hand us an 8-bit string,
> and we up-cast it to Unicode, then we fail because the upcast is
> generally done via ascii.
> 
> So I would at least take a first look at the 'import_archive' code, and
> make sure it is trying to work in Unicode paths, rather than 8-bit strings.

I'll check that, but I don't think it's doing anything odd here.

> Could this be related to the overloading of whatever machine that also
> happened? Meaning running this stuff is hammering on a machine hard
> enough that it times out occassionally? (Swapping, etc?)

It's possible. It was my understanding that the API is served by the
same appservers as the webapp, and so my requests shouldn't be a large
percentage of the requests they are getting. It's really hard to debug
this without someone on the LP side to look at it: these aren't errors
that we get OOPS numbers in the response for.

> An author field that is non-ascii and not utf-8. There is always the:
> 
> def decode_as_best_you_can(s):
>   try:
> return s.decode('utf-8')
>   except UnicodeDecodeError:
> return s.decode('latin-1')

Worth a go. This isn't critical data anyway, so graceful handling would
be good if the data can't be decoded.

I squashed some more bugs in this area after I sent the message
though. I had some confusion over which level was doing the decoding, so
there should be less issues like this now.


> Is it possible to get a query of old ones, and just run a bulk-update of
> them?

I have the list of packages, and mwhudson was going to query for the
list of branches based on that, and then request server-side upgrade I
believe.

> Happens if you commit exactly the same data 2 times, or if you try to
> autopack a single file. We've fixed a few of them, but having
> reproducible data here would help.

I'll run a local test of this packages to see if it is associated with
the data.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: import failures

[John Arbash Meinel]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1


...
> On Fri Dec 11 03:35:01 + 2009 james.westby wrote:
>> 639 packages failed
>>
>> 94 repeated reasons:
>>
>> 61 packages failed with reason

...

>> "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/util.py", 
>> line
>> 358, in find_extra_authors
>>   match = extra_author_re.match(change.decode("utf-8"))
>> File "/usr/lib/python2.5/encodings/utf_8.py", line 16, in decode
>>   return codecs.utf_8_decode(input, errors, True)
>>   UnicodeDecodeError: 'utf8' codec can't decode bytes in position 9-14:
>> unsupported Unicode code range
> 
> First of a set which is probably non-utf8 data in changelogs. There may be 
> hacks
> we can do for this. Don't discount the possibility that it is faulty encoding
> handling though.
> 

^- 'unsupported Unicode code range' sounds funny, but it may just be
that they have latin-1 chars in what should otherwise be a UTF-8 doc. Is
changelog *defined* as UTF-8? Or is it just '8-bit, put whatever feels
good to you' in there?


>> 43 packages failed with reason
...

>> "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
>> line 159, in import_dir
>>   import_archive(tree, dir_file, file_ids_from=file_ids_from)
>> File
>> "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
>> line 234, in import_archive
>>   trans_id = tt.trans_id_tree_path(relative_path)
>> File "/usr/lib/python2.5/site-packages/bzrlib/transform.py", line 241, in
>> trans_id_tree_path
>>   path = self.canonical_path(path)
>> File "/usr/lib/python2.5/site-packages/bzrlib/transform.py", line 1282, 
>> in
>> canonical_path
>>   abs = self._tree.abspath(path)
>> File "/usr/lib/python2.5/site-packages/bzrlib/workingtree.py", line 394, 
>> in
>> abspath
>>   return pathjoin(self.basedir, filename)
>> File "/usr/lib/python2.5/posixpath.py", line 65, in join
>>   path += '/' + b
>>   UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 25:
>> ordinal not in range(128)
> 
> This is usually that there is a filename from a different encoding in the
> package. We may not be able to get around this. I imagine some are utf-8
> though, so it may be a bug that it is trying to decode in ascii.
> 

My guess is that you are handing us 8-bit paths, and inside bzrlib all
*paths* are supposed to be Unicode. And if you hand us an 8-bit string,
and we up-cast it to Unicode, then we fail because the upcast is
generally done via ascii.

So I would at least take a first look at the 'import_archive' code, and
make sure it is trying to work in Unicode paths, rather than 8-bit strings.


>> 36 packages failed with reason
>> 'launchpadlib.errors.HTTPError::main:get_versions:lp_call:__call__:_requ
>> est':
> 
...

>> File "/usr/lib/python2.5/site-packages/launchpadlib/_browser.py", line 
>> 211,
>> in _request
>>   raise HTTPError(response, content)
>>   launchpadlib.errors.HTTPError: HTTP Error 503: Service Unavailable
> 
> Launchpad doesn't like me. These 36 happened in the few hours I
> was working on this task.
> 

Could this be related to the overloading of whatever machine that also
happened? Meaning running this stuff is hammering on a machine hard
enough that it times out occassionally? (Swapping, etc?)



...

> 
>> 30 packages failed with reason
>> 'UnicodeDecodeError::main:import_package:import_package:_do_import_packa
>> ge:import_upstream:decode':
>>  
>> /srv/package-import.canonical.com/new/scripts/python-debian/debian_bundle/change
>> log.py:274: UserWarning: Unexpected line while looking for next heading of 
>> EOF:
>>   vim:ai:et:sts=2:sw=2:tw=78:
>> warnings.warn(message)
>>   Traceback (most recent call last):
>> File "./import_package.py", line 884, in 
>>   sys.exit(main(args[0]))
>> File "./import_package.py", line 849, in main
>>   import_package(temp_dir, package, version, distro, release, pocket,
>> package_url, possible_transports=possible_transports)
>> File "./import_package.py", line 532, in import_package
>>   use_time_from_changelog=True)
>> File
>> "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
>> line 1555, in import_package
>>   timestamp=timestamp, author=author)
>> File
>> "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
>> line 1434, in _do_import_package
>>   timestamp=timestamp, author=author)
>> File
>> "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py",
>> line 1155, in import_upstream
>>   revprops['authors'] = author.decode("utf-8")
>> File "/usr/lib/python2.5/encodings/utf_8.py", line 16, in decode
>>   return codecs.utf_8_decode(input, errors, True)
>>   UnicodeDecodeError: 'utf8' codec can't decode bytes in position 10-12:
>> invalid data
> 
> Changelog data again perhaps.

An author field that is non-ascii and not utf-8. There is always the:

def de