Bug#657029: apt-get update does not understand content negotiation

2012-01-30 Thread Matthijs Kooijman 
Hi folks,

 I also tried to prevent the negatiation from happening by passing
 Accept-Encoding: identity in the HTTP request using wget, but that
 didn't help

I just realized that this is not about Accept-Encoding, but the
Accept: header. If apt would send an Accept: text/plain header, the
mirror replies with 406 Not Acceptable instead.

matthijs@grubby:~$  wget -O /dev/null --header='Accept: text/plain' 
--server-response 
http://ftp.ch.debian.org/mirror/debian/dists/wheezy/main/i18n/Translation-en
--2012-01-27 22:50:32--  
http://ftp.ch.debian.org/mirror/debian/dists/wheezy/main/i18n/Translation-en
Resolving ftp.ch.debian.org (ftp.ch.debian.org)... 129.132.86.210
Connecting to ftp.ch.debian.org (ftp.ch.debian.org)|129.132.86.210|:80... 
connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 406 Not Acceptable
  Date: Fri, 27 Jan 2012 21:50:32 GMT
  Server: Apache/2.2.16 (Debian)
  Alternates: {Translation-en.bz2 1 {type application/x-bzip2} {length 
3700790}}
  Vary: negotiate
  TCN: list
  Content-Length: 504
  Keep-Alive: timeout=15, max=100
  Connection: Keep-Alive
  Content-Type: text/html; charset=iso-8859-1
2012-01-27 22:50:32 ERROR 406: Not Acceptable.

Note sure what sending an Accept header does to other mirrors, though. It seems
Apache ignores it normally (happily returning application/x-bzip2 when you
Accept: text/plain only, if you use a .bz2 url). Only when multiviews are
enabled and the url does not have an extension, Apache looks at the Accept
header (Disclaimer: I haven't looked into Apache documentation and did a short
test with a sample size of 1, so please verify this). In any case, sending the
mime type you except in the Accept header should just work for compliant HTTP
servers. Alternatively, just sending the Accept headers for the uncompressed
case could make sense too.

Gr.

Matthijs


signature.asc
Description: Digital signature

Bug#657029: apt-get update does not understand content negotiation

2012-01-26 Thread Matthijs Kooijman 
Hi Steve,

 I had to copy the bzip2 deb package from a working host and install to  
 get apt to work.
Seems you point out a relevant issue here: Existing wheezy and sid hosts
that do not have bzip2, for whatever reason, now have their apt broken
without the option of fixing them through apt directly.

Perhaps a short-term solution to consider here is to push uncompressed
versions of the Translation files to the mirrors? I suppose that'll
cause the offending mirrors to just return that file instead of the
bz2 version?

This will increase the storage size required, but only back up to the
level of last week (when these Translations were included in the main
Package file, which is also offered uncompressed, right?).

Gr.

Matthijs


signature.asc
Description: Digital signature

Bug#657029: apt-get update does not understand content negotiation

2012-01-25 Thread Matthijs Kooijman 
Hi folks,

I'm seeing a similar problem on a chroot that doesn't have bzip2,
combined with a mirror that only offers the bzip2 version (with and
without .bz2 extension).

I can reproduce this on my regular system by adding this sources.list
line,

deb http://ftp.ch.debian.org/debian/ wheezy main

uninstalling the bzip2 pacakge,

sudo dpkg -r --force-depends bzip2

and then running apt-get update:

matthijs@grubby:~$ sudo apt-get update
[sudo] password for matthijs: 
Hit http://ftp.ch.debian.org wheezy InRelease
Hit http://ftp.ch.debian.org wheezy/main amd64 Packages/DiffIndex
Hit http://ftp.ch.debian.org wheezy/main TranslationIndex
Get:1 http://ftp.ch.debian.org wheezy/main Translation-en [3699 kB]
Fetched 1 B in 1s (1 B/s)
W: Failed to fetch

copy:/var/lib/apt/lists/partial/ftp.ch.debian.org_debian_dists_wheezy_main_i18n_Translation-en
Encountered a section with no Package: header

E: Some index files failed to download. They have been ignored, or old
ones used instead.

This downloads the following file:

matthijs@grubby:~$ wget -O /dev/null --server-response 
http://ftp.ch.debian.org/mirror/debian/dists/wheezy/main/i18n/Translation-en
--2012-01-25 18:42:48--  
http://ftp.ch.debian.org/mirror/debian/dists/wheezy/main/i18n/Translation-en
Resolving ftp.ch.debian.org (ftp.ch.debian.org)... 129.132.86.210
Connecting to ftp.ch.debian.org (ftp.ch.debian.org)|129.132.86.210|:80... 
connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Wed, 25 Jan 2012 17:42:48 GMT
  Server: Apache/2.2.16 (Debian)
  Content-Location: Translation-en.bz2
  Vary: negotiate
  TCN: choice
  Last-Modified: Wed, 25 Jan 2012 14:02:21 GMT
  ETag: 3c6c19d-387179-4b75ab657f540;4b75ab657f540
  Accept-Ranges: bytes
  Content-Length: 3699065
  Keep-Alive: timeout=15, max=100
  Connection: Keep-Alive
  Content-Type: application/x-bzip2
Length: 3699065 (3.5M) [application/x-bzip2]

as suggested before, this response really returns the .bz2 version, which apt
does not currently handle.

I'm not really sure how apt should handle this: By the time it gets to try the
uncompressed version, it has already tried all supported compressions, so it
probably can't handle the file, even if it would try to find out the type. I
can imagine apt looking at the Content-Type, and simply rejecting the
file if it's not what it expects, but that sounds a bit harsh to me.
Also, it could cause problems with mirrors that don't set the correct
Content-Type, but I guess those mirrors need fixing...

Alternatively, apt could just drop anything that has a Content-Location
in its response, but that sounds even harsher...

I also tried to prevent the negatiation from happening by passing
Accept-Encoding: identity in the HTTP request using wget, but that
didn't help (which makes sense, since Accept-Encoding is about
negotiating Content-Encoding, not Content-Type). However, I'm not sure
if you can actually negotiate over the Content-Type, since the server is
really just sending a different resource instead...

Anyway, just my observations,

Gr.

Matthijs


signature.asc
Description: Digital signature

Bug#657029: apt-get update does not understand content negotiation

2012-01-25 Thread Steve Allison 
Thank you! I've been trying to understand why some machines were failing 
apt-get and not others.


I've been using mirror, http://ukdebian.mirror.anlx.net/debian/ for wheezy

I had to copy the bzip2 deb package from a working host and install to 
get apt to work.


===
99% [9 Translation-en uncompressed 3,700 kB]E: Unable to parse package 
file 
/var/lib/apt/lists/partial/wheezy.apt:3142_debian_dists_wheezy_main_i18n_Translation-en.decomp 
(1)

Fetched 3,988 kB in 3s (1,298 kB/s)
W: Failed to fetch 
copy:/var/lib/apt/lists/partial/wheezy.apt:3142_debian_dists_wheezy_main_i18n_Translation-en 



W: Failed to fetch 
copy:/var/lib/apt/lists/partial/wheezy.apt:3142_debian_dists_wheezy_non-free_i18n_Translation-en  
Encountered a section with no Package: header


E: Some index files failed to download. They have been ignored, or old 
ones used instead.




--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#657029: apt-get update does not understand content negotiation

2012-01-23 Thread Andreas Beckmann 
Package: apt
Version: 0.8.15.9
Severity: normal

Hi,

I just noticed the following failure while running apt-get update in a
fresh, clean sid chroot:

  E: Unable to parse package file 
/var/lib/apt/lists/partial/ftp2.de.debian.org_debian_dists_sid_main_i18n_Translation-en.decomp
 (1)
  W: Failed to fetch 
copy:/var/lib/apt/lists/partial/ftp2.de.debian.org_debian_dists_sid_main_i18n_Translation-en
  E: Some index files failed to download. They have been ignored, or old ones 
used instead.

This can be reproduced with
  sudo piuparts -d sid -m http://ftp2.de.debian.org/debian -a hello

Some testing shows that this is a mirror specific problem. Other mirrors
work fine. The only difference I could find is how a request for
Translation-en(.bz2) is answered by the mirrors:


wget http://ftp.de.debian.org/debian/dists/sid/main/i18n/Translation-en
  = 404 Not Found
wget http://ftp.de.debian.org/debian/dists/sid/main/i18n/Translation-en.bz2
  = 200 OK Length: 3857620 (3.7M) [application/x-bzip2]
wget http://ftp2.de.debian.org/debian/dists/sid/main/i18n/Translation-en
  = 200 OK Length: 3857620 (3.7M) [application/x-bzip2]
wget http://ftp2.de.debian.org/debian/dists/sid/main/i18n/Translation-en.bz2
  = 200 OK Length: 3857620 (3.7M) [application/octet-stream]

wget -S http://ftp2.de.debian.org/debian/dists/sid/main/i18n/Translation-en
--2012-01-23 16:26:21--  
http://ftp2.de.debian.org/debian/dists/sid/main/i18n/Translation-en
Resolving ftp2.de.debian.org... 137.226.34.42
Connecting to ftp2.de.debian.org|137.226.34.42|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/1.0.0
  Date: Mon, 23 Jan 2012 15:26:21 GMT
  Content-Type: application/x-bzip2
  Connection: keep-alive
  Content-Location: Translation-en.bz2
  Vary: negotiate
  TCN: choice
  Last-Modified: Mon, 23 Jan 2012 14:01:21 GMT
  ETag: 19103b3-3adcd4-4b7327715ae40;4b7327715ae40
  Accept-Ranges: bytes
  Content-Length: 3857620
Length: 3857620 (3.7M) [application/x-bzip2]
Saving to: `Translation-en.5'

100%[=...=] 3,857,620   1.05M/s   in 3.6s

2012-01-23 16:26:25 (1.03 MB/s) - `Translation-en.5' saved [3857620/3857620]


This looks like apt-get receives a compressed file for the request for
Translation-en but does ignore the Content-Type and treats it as a text
file which it fails do parse ...

This issues seems to be uncovered only by the description-less package
indices as announced yesterday:
http://lists.debian.org/debian-devel-announce/2012/01/msg4.html


Andreas


Full transcript of the apt-get update command:

1m25.5s DEBUG: Starting command: ['chroot', '/tmp/tmpZOvwZN', 'eatmydata', 
'apt-get', 'update']
1m29.0s DUMP:
  Get:1 http://ftp2.de.debian.org sid InRelease [190 kB]
  Get:2 http://ftp2.de.debian.org sid/main amd64 Packages/DiffIndex [2038 B]
  Get:3 http://ftp2.de.debian.org sid/contrib amd64 Packages/DiffIndex [2023 B]
  Get:4 http://ftp2.de.debian.org sid/non-free amd64 Packages/DiffIndex [2023 B]
  Ign http://ftp2.de.debian.org sid/contrib TranslationIndex
  Get:5 http://ftp2.de.debian.org sid/main TranslationIndex [2154 B]
  Ign http://ftp2.de.debian.org sid/non-free TranslationIndex
  Get:6 http://ftp2.de.debian.org sid/contrib Translation-en [36.8 kB]
  Get:7 http://ftp2.de.debian.org sid/main Translation-en [3858 kB]
  Get:8 http://ftp2.de.debian.org sid/non-free Translation-en [69.5 kB]
  E: Unable to parse package file 
/var/lib/apt/lists/partial/ftp2.de.debian.org_debian_dists_sid_main_i18n_Translation-en.decomp
 (1)
  Fetched 4162 kB in 3s (1209 kB/s)
  W: Failed to fetch 
copy:/var/lib/apt/lists/partial/ftp2.de.debian.org_debian_dists_sid_main_i18n_Translation-en

  E: Some index files failed to download. They have been ignored, or old ones 
used instead.
1m29.0s ERROR: Command failed (status=100): ['chroot', '/tmp/tmpZOvwZN', 
'eatmydata', 'apt-get', 'update']



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#657029: apt-get update does not understand content negotiation

2012-01-23 Thread Ansgar Burchardt 
reassign 657029 mirrors
thanks

Andreas Beckmann deb...@abeckmann.de writes:
 wget http://ftp.de.debian.org/debian/dists/sid/main/i18n/Translation-en
   = 404 Not Found
 wget http://ftp.de.debian.org/debian/dists/sid/main/i18n/Translation-en.bz2
   = 200 OK Length: 3857620 (3.7M) [application/x-bzip2]
 wget http://ftp2.de.debian.org/debian/dists/sid/main/i18n/Translation-en
   = 200 OK Length: 3857620 (3.7M) [application/x-bzip2]
 wget http://ftp2.de.debian.org/debian/dists/sid/main/i18n/Translation-en.bz2
   = 200 OK Length: 3857620 (3.7M) [application/octet-stream]

 wget -S http://ftp2.de.debian.org/debian/dists/sid/main/i18n/Translation-en
 --2012-01-23 16:26:21--  
 http://ftp2.de.debian.org/debian/dists/sid/main/i18n/Translation-en
 Resolving ftp2.de.debian.org... 137.226.34.42
 Connecting to ftp2.de.debian.org|137.226.34.42|:80... connected.
 HTTP request sent, awaiting response...
   HTTP/1.1 200 OK
   Server: nginx/1.0.0
   Date: Mon, 23 Jan 2012 15:26:21 GMT
   Content-Type: application/x-bzip2
   Connection: keep-alive
   Content-Location: Translation-en.bz2
   Vary: negotiate
   TCN: choice
   Last-Modified: Mon, 23 Jan 2012 14:01:21 GMT
   ETag: 19103b3-3adcd4-4b7327715ae40;4b7327715ae40
   Accept-Ranges: bytes
   Content-Length: 3857620
 Length: 3857620 (3.7M) [application/x-bzip2]
 Saving to: `Translation-en.5'

I don't know how content negotiation in HTTP works, but it looks wrong
to just send the compressed file when requesting the uncompressed
version.  I think for transparent compression there needs to be a
'Content-Encoding' field somewhere.

As it does not happen with all mirrors, but only with ftp2.de.d.o (and
others?), I'm inclined to blame the mirror configuration.

   Get:7 http://ftp2.de.debian.org sid/main Translation-en [3858 kB]
   Get:8 http://ftp2.de.debian.org sid/non-free Translation-en [69.5 kB]
   E: Unable to parse package file 
 /var/lib/apt/lists/partial/ftp2.de.debian.org_debian_dists_sid_main_i18n_Translation-en.decomp
  (1)
   Fetched 4162 kB in 3s (1209 kB/s)
   W: Failed to fetch 
 copy:/var/lib/apt/lists/partial/ftp2.de.debian.org_debian_dists_sid_main_i18n_Translation-en

Regards,
Ansgar



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#657029: apt-get update does not understand content negotiation

2012-01-23 Thread Simon Paillard 
On Mon, Jan 23, 2012 at 06:18:06PM +0100, Ansgar Burchardt wrote:
 Andreas Beckmann deb...@abeckmann.de writes:
  wget http://ftp.de.debian.org/debian/dists/sid/main/i18n/Translation-en
= 404 Not Found
  wget http://ftp.de.debian.org/debian/dists/sid/main/i18n/Translation-en.bz2
= 200 OK Length: 3857620 (3.7M) [application/x-bzip2]
  wget http://ftp2.de.debian.org/debian/dists/sid/main/i18n/Translation-en
= 200 OK Length: 3857620 (3.7M) [application/x-bzip2]
  wget http://ftp2.de.debian.org/debian/dists/sid/main/i18n/Translation-en.bz2
= 200 OK Length: 3857620 (3.7M) [application/octet-stream]
 
  wget -S http://ftp2.de.debian.org/debian/dists/sid/main/i18n/Translation-en
[..]
Content-Type: application/x-bzip2
[..]
Content-Location: Translation-en.bz2
[..] 
 I don't know how content negotiation in HTTP works, but it looks wrong
 to just send the compressed file when requesting the uncompressed
 version.  I think for transparent compression there needs to be a
 'Content-Encoding' field somewhere.

It's expected behaviour for some content negotiation setup, and it's about
filename and not 'compressed or not', specially MultiViews in Apache.

 As it does not happen with all mirrors, but only with ftp2.de.d.o (and
 others?), I'm inclined to blame the mirror configuration.

A HTTP client for Translation-en, and the daemon tells in Content-Location that
the file sent is Translation-en.bz2, there is no problem (as long as the client
fully support HTTP).

And I am sure more mirrors than only ftp2.de.d.o may have Multiviews enabled.
(ftp2.de.d.o admin noticed if he wants to turn this off).
 
Get:7 http://ftp2.de.debian.org sid/main Translation-en [3858 kB]
Get:8 http://ftp2.de.debian.org sid/non-free Translation-en [69.5 kB]
E: Unable to parse package file 
  /var/lib/apt/lists/partial/ftp2.de.debian.org_debian_dists_sid_main_i18n_Translation-en.decomp
   (1)
Fetched 4162 kB in 3s (1209 kB/s)
W: Failed to fetch 
  copy:/var/lib/apt/lists/partial/ftp2.de.debian.org_debian_dists_sid_main_i18n_Translation-en

I cannot reproduce the impact on apt that in a sid chroot using apt 0.8.15.9
deb http://ftp2.de.debian.org/debian/ sid main

Do you have some specific apt configuration ?

Any way, IMO it's up to apt to be robust to that.

Regards.

-- 
Simon Paillard



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org