Bug#388708: apt: Upgrading to this version makes status file unparseable

[Manoj Srivastava]

Package: apt
Version: 0.6.46
Severity: grave

#_> aptitude -u
 ... upgrade ...
#_> aptitude -vvv upgrade
Reading package lists... Error!
E: Unable to parse package file /var/lib/dpkg/status (1)
E: The package lists or status file could not be parsed or opened.
Reading package lists... Error!
E: Unable to parse package file /var/lib/dpkg/status (1)
E: The package lists or status file could not be parsed or opened.
[1]22606 exit 255   aptitude -vvv upgrade

Tested this on two machines. Downgrading to 0.6.45 fixes the
 bug. Look below for a sequence of action:

#_> dpkg -i apt_0.6.46_i386.deb
(Reading database ... 333632 files and directories currently installed.)
Preparing to replace apt 0.6.45 (using apt_0.6.46_i386.deb) ...
Unpacking replacement apt ...
Setting up apt (0.6.46) ...

#_> apt-cache policy apt
E: Unable to parse package file /var/lib/dpkg/status (1)
[1]24419 exit 100   apt-cache policy apt
#_> dpkg -i apt_0.6.45_i386.deb
dpkg - warning: downgrading apt from 0.6.46 to 0.6.45.
(Reading database ... 333635 files and directories currently installed.)
Preparing to replace apt 0.6.46 (using apt_0.6.45_i386.deb) ...
Unpacking replacement apt ...
Setting up apt (0.6.45) ...

#_> apt-cache policy apt
apt:
  Installed: 0.6.45
  Candidate: 0.6.46
  Version table:
 0.6.46 0
990 http://glaurung.internal.golden-gryphon.com sid/main Packages
990 http://ftp.debian.org sid/main Packages
 0.6.45exp2 0
  1 http://ftp.debian.org experimental/main Packages
 *** 0.6.45 0
100 /var/lib/dpkg/status

-- System Information:
Debian Release: testing/unstable
  APT prefers unstable
  APT policy: (990, 'unstable'), (500, 'stable'), (1, 'experimental')
Architecture: i386 (i686)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.17.13-mh5-skas3-v9-pre9
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL 
set to en_US.UTF-8)


manoj



strace.log
Description: strace log

Versions of packages apt depends on:
ii  libc6   2.3.6.ds1-4  GNU C Library: Shared libraries
ii  libgcc1 1:4.2-20060709-1 GCC support library
ii  libstdc++6  4.1.1-13 The GNU Standard C++ Library v3

Versions of packages apt recommends:
ii  debian-archive-keyring2006.01.18 GnuPG archive keys of the Debian a

-- no debconf information

-- 
Sex is like snow... You never know how many inches you're going to get
or how long it will last.
Manoj Srivastava <[EMAIL PROTECTED]>
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C

Bug#388708: apt: Upgrading to this version makes status file unparseable

[Mark Hedges]

Package: apt
Followup-For: Bug #388708


I also encountered this problem and thank you Manoj for explaining how
to fix it.  I was totally lost.  Because apt didn't work, users have to
download the deb of 0.6.45 directly from the repository and install it
with dpkg.  It's a gnarly fix. 

Seems like a test that a basic `apt-get update` works would be a good
benchmark for allowing a core package like apt to pass from unstable
to testing.  I know testing is testing and all, and a few times I've
had to reformat upon an upgrade breakage, but luckily this one was
fixable without reinstallation.

Mark

-- Package-specific info:

-- apt-config dump --

APT "";
APT::Architecture "i386";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Periodic "";
APT::Periodic::Update-Package-Lists "0";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
APT::Archives "";
APT::Archives::MaxAge "30";
APT::Archives::MinAge "2";
APT::Archives::MaxSize "500";
APT::Default-Release "testing";
APT::Cache-Limit "16777216";
Dir "/";
Dir::State "var/lib/apt/";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::userstatus "status.user";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt/";
Dir::Cache::archives "archives/";
Dir::Cache::srcpkgcache "srcpkgcache.bin";
Dir::Cache::pkgcache "pkgcache.bin";
Dir::Etc "etc/apt/";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::sourceparts "sources.list.d";
Dir::Etc::vendorlist "vendors.list";
Dir::Etc::vendorparts "vendors.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::dpkg "/usr/bin/dpkg";
DPkg "";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
DPkg::Post-Invoke "";
DPkg::Post-Invoke:: "if [ -d /var/lib/update-notifier ]; then  touch 
/var/lib/update-notifier/dpkg-run-stamp; fi";
Acquire "";
AcquireProxy "false";

-- (no /etc/apt/preferences present) --


-- /etc/apt/sources.list --

# deb http://ftp.us.debian.org/debian/ stable main non-free contrib
# deb http://ftp.us.debian.org/debian/ testing main non-free contrib
# deb http://ftp.us.debian.org/debian/ unstable main non-free

# deb http://ike.egr.msu.edu/debian/ stable main non-free contrib
# deb http://ike.egr.msu.edu/debian/ testing main non-free contrib
# deb http://ike.egr.msu.edu/debian/ unstable main non-free

# deb http://debian-mirror.mirror.umn.edu/debian/ stable main non-free contrib
# deb http://debian-mirror.mirror.umn.edu/debian/ testing main non-free contrib
# deb http://debian-mirror.mirror.umn.edu/debian/ unstable main non-free

# deb http://mirrors1.kernel.org/debian/ stable main non-free contrib
# deb http://mirrors1.kernel.org/debian/ testing main non-free contrib
# deb http://mirrors1.kernel.org/debian/ unstable main non-free

  deb http://archive.progeny.com/debian/ stable main non-free contrib
  deb http://archive.progeny.com/debian/ testing main non-free contrib
  deb http://archive.progeny.com/debian/ unstable main non-free

# deb http://debian.mirror.frontiernet.net/debian stable main non-free contrib
# deb http://debian.mirror.frontiernet.net/debian testing main non-free contrib
# deb http://debian.mirror.frontiernet.net/debian unstable main non-free contrib

# deb http://security.debian.org/ stable/updates main contrib non-free
# deb http://security.debian.org/ testing/updates main contrib non-free

  deb http://klecker.debian.org/debian-security stable/updates main contrib 
non-free
  deb http://klecker.debian.org/debian-security testing/updates main contrib 
non-free



# deb http://mirrors.ibiblio.org/pub/mirrors/blackdown/debian testing non-free
# deb http://mirrors.ibiblio.org/pub/mirrors/blackdown/debian stable non-free

# deb http://ftp.tux.org/pub/java/debian testing non-free
# deb http://ftp.tux.org/pub/java/debian stable non-free


# deb http://people.debian.org/~srivasta/ packages/
# deb-src http://people.debian.org/~srivasta/ packages/

-- System Information:
Debian Release: testing/unstable
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable'), (500, 'stable')
Architecture: i386 (i686)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.17-hedges
Locale: LANG=en_US, LC_CTYPE=en_US (charmap=ISO-8859-1)

Versions of packages apt depends on:
ii  libc62.3.6.ds1-4 GNU C Library: Shared libraries
ii  libgcc1  1:4.1.1-13  GCC support library
ii  libstdc++6   4.1.1-13The GNU Standard C++ Library v3

Versions of packages apt recommends:
ii  debian-archive-keyring2006.01.18 GnuPG archive keys of the Debian a

-- no debconf information



This email message is for the sole use of the intended recipient(s) and
may contain privileged information. Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the inte

Bug#388708: apt: Upgrading to this version makes status file unparseable

[Michael Vogt]

On Fri, Sep 22, 2006 at 12:49:22AM -0500, Manoj Srivastava wrote:
> Package: apt
> Version: 0.6.46
> Severity: grave

Thanks for your bugreport.
 
> #_> aptitude -u
>  ... upgrade ...
> #_> aptitude -vvv upgrade
> Reading package lists... Error!
> E: Unable to parse package file /var/lib/dpkg/status (1)
> E: The package lists or status file could not be parsed or opened.
> Reading package lists... Error!
> E: Unable to parse package file /var/lib/dpkg/status (1)
> E: The package lists or status file could not be parsed or opened.
> [1]22606 exit 255   aptitude -vvv upgrade
> 
> Tested this on two machines. Downgrading to 0.6.45 fixes the
>  bug. Look below for a sequence of action:
[..]
 
I can't reproduce this bug here. Could you please put a gziped copy of
your /var/lib/dpkg/status file somewhere to download? Or send it to me
directly? 

I would also be interessted if this code here works and if it dies, at
what package:
---8<--
#!/usr/bin/env python
import apt_pkg

tagfile = apt_pkg.ParseTagFile(open("/var/lib/dpkg/status","r"));

while tagfile.Step():
   print tagfile.Section.get("Package");
---8<--


Thanks,
 Michael



-- 
Linux is not The Answer. Yes is the answer. Linux is The Question. - Neo


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Bug#388708: apt: Upgrading to this version makes status file unparseable

[Michael Vogt]

[..]
On Fri, Sep 22, 2006 at 12:49:22AM -0500, Manoj Srivastava wrote:
> #_> aptitude -u
>  ... upgrade ...
> #_> aptitude -vvv upgrade
> Reading package lists... Error!
> E: Unable to parse package file /var/lib/dpkg/status (1)
> E: The package lists or status file could not be parsed or opened.
> Reading package lists... Error!
> E: Unable to parse package file /var/lib/dpkg/status (1)
> E: The package lists or status file could not be parsed or opened.
[..]

I was able to reproduce this now (thanks for your help Manoj!). It
turned out that the buffer in the tagfile code was too small
(again!). The attached patch should fix the issue by allowing the
tagfile buffer to grow dynamically. Testing is very welcome.

Cheers,
 Michael

-- 
Linux is not The Answer. Yes is the answer. Linux is The Question. - Neo
=== modified file 'apt-pkg/tagfile.cc'
--- apt-pkg/tagfile.cc  2006-09-15 16:24:47 +
+++ apt-pkg/tagfile.cc  2006-09-30 21:34:13 +
@@ -59,19 +59,52 @@
delete [] Buffer;
 }
/*}}}*/
+// TagFile::Resize - Resize the internal buffer
/*{{{*/
+// -
+/* Resize the internal buffer (double it in size). Fail if a maximum size
+ * size is reached.
+ */
+bool pkgTagFile::Resize()
+{
+   char *tmp;
+   unsigned long EndSize = End - Start;
+
+   // fail is the buffer grows too big
+   if(Size > 1024*1024+1)
+  return false;
+
+   // get new buffer and use it
+   tmp = new char[2*Size];
+   memcpy(tmp, Buffer, Size);
+   Size = Size*2;
+   delete [] Buffer;
+   Buffer = tmp;
+
+   // update the start/end pointers to the new buffer
+   Start = Buffer;
+   End = Start + EndSize;
+   return true;
+}
+
 // TagFile::Step - Advance to the next section /*{{{*/
 // -
-/* If the Section Scanner fails we refill the buffer and try again. */
+/* If the Section Scanner fails we refill the buffer and try again. 
+ * If that fails too, double the buffer size and try again until a
+ * maximum buffer is reached.
+ */
 bool pkgTagFile::Step(pkgTagSection &Tag)
 {
-   if (Tag.Scan(Start,End - Start) == false)
+   while (Tag.Scan(Start,End - Start) == false)
{
   if (Fill() == false)
 return false;
   
-  if (Tag.Scan(Start,End - Start) == false)
+  if(Tag.Scan(Start,End - Start))
+break;
+
+  if (Resize() == false)
 return _error->Error(_("Unable to parse package file %s (1)"),
- Fd.Name().c_str());
+Fd.Name().c_str());
}
Start += Tag.size();
iOffset += Tag.size();

=== modified file 'apt-pkg/tagfile.h'
--- apt-pkg/tagfile.h   2006-09-30 20:50:41 +
+++ apt-pkg/tagfile.h   2006-09-30 21:35:25 +
@@ -77,6 +77,7 @@
unsigned long Size;

bool Fill();
+   bool Resize();

public:
 
 

Bug#388708: apt: Upgrading to this version makes status file unparseable

[Jeroen van Wolffelaar]

On Sun, Oct 01, 2006 at 12:04:48AM +0200, Michael Vogt wrote:
> --- apt-pkg/tagfile.h 2006-09-30 20:50:41 +
> +++ apt-pkg/tagfile.h 2006-09-30 21:35:25 +
> @@ -77,6 +77,7 @@
> unsigned long Size;
> 
> bool Fill();
> +   bool Resize();

This would (unnecessarily) change the ABI of libapt, requiring a shlibs
versioned bump.  Fill() neither should've been part of the ABI (it makes
no sense to use it externally), but it is at the moment.

This symbol should instead be placed in a "private:" section of the class
(without any explicit definition, symbols are public).

Otherwise, the patch logic is as far as I can see, correct.

--Jeroen

-- 
Jeroen van Wolffelaar
[EMAIL PROTECTED] (also for Jabber & MSN; ICQ: 33944357)
http://Jeroen.A-Eskwadraat.nl


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Bug#388708: apt: Upgrading to this version makes status file unparseable

[Michael Vogt]

On Sun, Oct 01, 2006 at 06:31:36PM +0200, Jeroen van Wolffelaar wrote:
> On Sun, Oct 01, 2006 at 12:04:48AM +0200, Michael Vogt wrote:
> > --- apt-pkg/tagfile.h   2006-09-30 20:50:41 +
> > +++ apt-pkg/tagfile.h   2006-09-30 21:35:25 +
> > @@ -77,6 +77,7 @@
> > unsigned long Size;
> > 
> > bool Fill();
> > +   bool Resize();
> 
> This would (unnecessarily) change the ABI of libapt, requiring a shlibs
> versioned bump.  Fill() neither should've been part of the ABI (it makes
> no sense to use it externally), but it is at the moment.
> 
> This symbol should instead be placed in a "private:" section of the class
> (without any explicit definition, symbols are public).

For c++ the default access (if nothing else is specified) is
"private:".

According to
http://developer.kde.org/documentation/other/binarycompatibility.html
it is ok to add new non-virtual functions and won't break the ABI.
 
> Otherwise, the patch logic is as far as I can see, correct.

Thanks for reviewing it!

Cheers,
 Michael

-- 
Linux is not The Answer. Yes is the answer. Linux is The Question. - Neo


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]