Re: apt: Saves some downloaded packages under truncated filenames

2013-09-19 Thread Cyril Brulebois 
Cyril Brulebois  (2013-09-19):
> Tracked down to: pkgAcqArchive::pkgAcqArchive(), where Parse.FileName()
> returns: "pool/main/c/c"
> 
> Looking further, through debindexfile.cc then debrecords.cc, it
> looks like debRecordParser::FileName() returns a faulty (truncated)
> string.
> 
> Root cause is probably in pkgTagSection::Find(), which I'll try to
> look at later on.

It actually finds what it can, which is faulty because the “Fast scanner
for RFC-822 type header information” (tagfile.cc) failed to read the
stanza properly.

I added some traces to pkgTagSection::Scan(), dumping std::string(Start,Stop)
after both TrimRecord() calls, and got that:
| >>> trimmed record (1) <<<
| Package: cdebconf-newt-terminal
| Source: cdebconf-terminal
| Version: 0.22
| Installed-Size: 43
| Maintainer: Debian Install System Team 
| Architecture: amd64
| Provides: cdebconf-terminal
| Depends: cdebconf-newt-udeb (>= 0.146), libc6-udeb (>= 2.17), libnewt0.52
| Description: cdebconf newt plugin to provide a clean terminal
| Description-md5: 4109a053022081b573d864d84d6eb16d
| Section: debian-installer
| Priority: extra
| 
| >>> eof <<<
| >>> trimmed record (2) <<<
| Package: cdebconf-newt-terminal
| Source: cdebconf-terminal
| Version: 0.22
| Installed-Size: 43
| Maintainer: Debian Install System Team 
| Architecture: amd64
| Provides: cdebconf-terminal
| Depends: cdebconf-newt-udeb (>= 0.146), libc6-udeb (>= 2.17), libnewt0.52
| Description: cdebconf newt plugin to provide a clean terminal
| Description-md5: 4109a053022081b573d864d84d6eb16d
| Section: debian-installer
| Priority: extra
| Filename: pool/main/c/c
| 
| 
| >>> eof <<<

I guess something should read a bit more to get the full stanza,
otherwise, some fields (Filename here) end up getting truncated.

Good luck fixing the scanner. :-)

Mraw,
KiBi.


-- 
To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130919124844.gd15...@mraw.org

Re: apt: Saves some downloaded packages under truncated filenames

2013-09-19 Thread Cyril Brulebois 
Cyril Brulebois  (2013-09-19):
> Since Owner->DestFile is used both for creating a Message and for
> printing URIs, I suspect that's the one going bad. Looking at
> apt-private/private-install.cc's InstallPackages(), it appears
> the Fetcher is created by the PackageManager, and one then gets the
> files out of there. I suspect this is what wants getting looked at.

Tracked down to: pkgAcqArchive::pkgAcqArchive(), where Parse.FileName()
returns: "pool/main/c/c"

Looking further, through debindexfile.cc then debrecords.cc, it
looks like debRecordParser::FileName() returns a faulty (truncated)
string.

Root cause is probably in pkgTagSection::Find(), which I'll try to
look at later on.

> As for reproducing the issue:
> |   debcheckout debian-installer foo
> |   cd foo/build
> |   export DEB_HOST_ARCH=$(dpkg-architecture -qDEB_HOST_ARCH)
> |   ./util/get-packages udeb acpi-modules-3.10-3-amd64-di alsa-base-udeb 
> alsa-utils-udeb anna archdetect bogl-bterm-udeb brltty-udeb busybox-udeb 
> cdebconf-gtk-terminal cdebconf-gtk-udeb cdebconf-newt-terminal 
> cdebconf-newt-udeb cdebconf-priority cdebconf-text-udeb cdebconf-udeb 
> choose-mirror choose-mirror-bin console-setup-linux-fonts-udeb 
> console-setup-pc-ekmap console-setup-udeb core-modules-3.10-3-amd64-di 
> crc-modules-3.10-3-amd64-di crypto-modules-3.10-3-amd64-di 
> debian-archive-keyring-udeb di-utils di-utils-reboot di-utils-shell 
> di-utils-terminfo download-installer env-preseed espeak-data-udeb 
> espeakup-udeb ethdetect event-modules-3.10-3-amd64-di 
> fat-modules-3.10-3-amd64-di fb-modules-3.10-3-amd64-di file-preseed 
> firewire-core-modules-3.10-3-amd64-di fontconfig-udeb fonts-farsiweb-udeb 
> fonts-khmeros-udeb fonts-lao-udeb fonts-lklug-sinhala-udeb fonts-mlym-udeb 
> fonts-sil-abyssinica-udeb fonts-sil-padauk-udeb fonts-taml-udeb 
> fonts-telu-udeb fonts-thai-tlwg-udeb fonts-tibetan-machine-udeb 
> fonts-ukij-uyghur-udeb
> 
> (I suspect one can truncate the package list some more, but that's
> for another day; if you're on another arch, try this command
> instead, without the export: make rebuild_netboot)

Thanks to a tiny shell loop, I managed to reduce the needed package
list to:
  cdebconf-newt-terminal cdebconf-gtk-terminal

Mraw,
KiBi.


--
To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130919103229.gc15...@mraw.org

Re: apt: Saves some downloaded packages under truncated filenames

2013-09-18 Thread Cyril Brulebois 
Cyril Brulebois  (2013-09-19):
> My log said:
> | Get:140 http://ftp.fr.debian.org/debian/ unstable/main/debian-installer 
> cdebconf-newt-terminal amd64 0.22 [4,538 B]   
>  
> […]
> | Needed cdebconf-newt-terminal not found (looked in 
> apt.udeb/cache/archives/, debugudebs/)
> 
> and no such file there indeed, but some strangely-named files:
> | c

I used a breakpoint in flNotDir to detect when said udeb was being
handled. Caller was pkgAcqArchive::Done(), which extracts the filename
from the Message it receives:
| // Grab the output filename
| string FileName = LookupTag(Message,"Filename");

Showing all of Message (using gdb's “set print elements 0”):
| (gdb) p Message
| $24 = {
|   static npos = , 
|   _M_dataplus = {
| > = {
|   <__gnu_cxx::new_allocator> = {}, }, 
| members of std::basic_string, 
std::allocator >::_Alloc_hider: 
| _M_p = 0x186b348 "201 URI Done\nURI: 
http://ftp.fr.debian.org/debian/pool/main/c/cdebconf-terminal/cdebconf-newt-terminal_0.22_amd64.udeb\nFilename:
 
/home/kibi/debian-installer/installer/build/apt.udeb/cache/archives/partial/c\nSize:
 4538\nLast-Modified: Fri, 06 Sep 2013 05:42:23 GMT\nMD5-Hash: 
20db6152fce5081fcbf49c7c08f21246\nMD5Sum-Hash: 
20db6152fce5081fcbf49c7c08f21246\nSHA1-Hash: 
fa2a40f777a2f48b9634866bc780fb059e60b2fe\nSHA256-Hash: 
c4d99ef27285f0c9090005313165627e56e0972e687af7e68c2b1d1538e2ae09\nSHA512-Hash: 
046dc9b0dbe08fd1ec54301714a452c70abb847b262a94fc9f468fff7259a542849b759e71f974ae3a878f4b04db42bf6e600bfd2090bc40eba0806a9b4e9a8c"
|   }
| }

So it appears the message is corrupted?

Now looking into the http method (ISTR ftp led to the same results),
adding a trivial clog call in there, I'm getting:
| Filename in http method: 
/home/kibi/debian-installer/installer/build/apt.udeb/cache/archives/partial/c

so it was actually set way before that, as expected DestFile.

Trying to apt-get install --print-uris, that's indeed sufficient
to exhibit the issue, no need to download/clean/playagain.

Since Owner->DestFile is used both for creating a Message and for
printing URIs, I suspect that's the one going bad. Looking at
apt-private/private-install.cc's InstallPackages(), it appears
the Fetcher is created by the PackageManager, and one then gets the
files out of there. I suspect this is what wants getting looked at.

As for reproducing the issue:
|   debcheckout debian-installer foo
|   cd foo/build
|   export DEB_HOST_ARCH=$(dpkg-architecture -qDEB_HOST_ARCH)
|   ./util/get-packages udeb acpi-modules-3.10-3-amd64-di alsa-base-udeb 
alsa-utils-udeb anna archdetect bogl-bterm-udeb brltty-udeb busybox-udeb 
cdebconf-gtk-terminal cdebconf-gtk-udeb cdebconf-newt-terminal 
cdebconf-newt-udeb cdebconf-priority cdebconf-text-udeb cdebconf-udeb 
choose-mirror choose-mirror-bin console-setup-linux-fonts-udeb 
console-setup-pc-ekmap console-setup-udeb core-modules-3.10-3-amd64-di 
crc-modules-3.10-3-amd64-di crypto-modules-3.10-3-amd64-di 
debian-archive-keyring-udeb di-utils di-utils-reboot di-utils-shell 
di-utils-terminfo download-installer env-preseed espeak-data-udeb espeakup-udeb 
ethdetect event-modules-3.10-3-amd64-di fat-modules-3.10-3-amd64-di 
fb-modules-3.10-3-amd64-di file-preseed firewire-core-modules-3.10-3-amd64-di 
fontconfig-udeb fonts-farsiweb-udeb fonts-khmeros-udeb fonts-lao-udeb 
fonts-lklug-sinhala-udeb fonts-mlym-udeb fonts-sil-abyssinica-udeb 
fonts-sil-padauk-udeb fonts-taml-udeb fonts-telu-udeb fonts-thai-tlwg-udeb 
fonts-tibetan-machine-udeb fonts-ukij-uyghur-udeb

(I suspect one can truncate the package list some more, but that's
for another day; if you're on another arch, try this command
instead, without the export: make rebuild_netboot)

Mraw,
KiBi.


signature.asc
Description: Digital signature