[jira] Commented: (WAGON-264) compressed tarball download problems
[ http://jira.codehaus.org/browse/WAGON-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=181523#action_181523 ] John Casey commented on WAGON-264: -- The wagon-maven-plugin will have to adjust (and be re-released) to handle httpConfiguration, I would expect. That is, unless it deals with the WagonManager component from maven-artifact-manager, which in turn handles Wagon instances that are configured using entries in the settings.xml...which IMO is probably a good practice for it to do... > compressed tarball download problems > > > Key: WAGON-264 > URL: http://jira.codehaus.org/browse/WAGON-264 > Project: Maven Wagon > Issue Type: Improvement > Components: wagon-http-lightweight >Affects Versions: 1.0-beta-2, 1.0-beta-3, 1.0-beta-4, 1.0-beta-5 >Reporter: Lee Thompson >Assignee: John Casey > Fix For: 1.0-beta-6 > > > HTTP wagon will uncompress tarballs it downloads, sometimes badly. > Downloading openssl results in an unpressed download > $ mvn -X wagon:download-single -Dwagon.fromFile=openssl-0.9.8k.tar.gz > -Dwagon.url=http://www.openssl.org/source -Dwagon.toDir=./ > $ tar tzf openssl-0.9.8k.tar.gz > gzip: stdin: not in gzip format > tar: Child returned status 1 > tar: Error exit delayed from previous errors > $ tar tf openssl-0.9.8k.tar.gz | more > openssl-0.9.8k/apps/ > openssl-0.9.8k/apps/app_rand.c > Downloading expat results in a corrupted file > mvn -X wagon:download-single -Dwagon.fromFile=expat-1.98.8.tar.gz > -Dwagon.url=http://prdownloads.sourceforge.net/expat -Dwagon.toDir=./ > $ tar tf expat-1.98.8.tar.gz > tar: This does not look like a tar archive > tar: Skipping to next header > tar: Read 1943 bytes from expat-1.98.8.tar.gz > tar: Error exit delayed from previous errors -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WAGON-264) compressed tarball download problems
[ http://jira.codehaus.org/browse/WAGON-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=180402#action_180402 ] Lee Thompson commented on WAGON-264: I have a question. In the use case that opened the issue using wagon-maven-plugin to retrieve off of openssl which isn't a maven repo, how would you inject the httpConfiguration into wagon-maven-plugin? > compressed tarball download problems > > > Key: WAGON-264 > URL: http://jira.codehaus.org/browse/WAGON-264 > Project: Maven Wagon > Issue Type: Improvement > Components: wagon-http-lightweight >Affects Versions: 1.0-beta-2, 1.0-beta-3, 1.0-beta-4, 1.0-beta-5 >Reporter: Lee Thompson >Assignee: John Casey > Fix For: 1.0 > > > HTTP wagon will uncompress tarballs it downloads, sometimes badly. > Downloading openssl results in an unpressed download > $ mvn -X wagon:download-single -Dwagon.fromFile=openssl-0.9.8k.tar.gz > -Dwagon.url=http://www.openssl.org/source -Dwagon.toDir=./ > $ tar tzf openssl-0.9.8k.tar.gz > gzip: stdin: not in gzip format > tar: Child returned status 1 > tar: Error exit delayed from previous errors > $ tar tf openssl-0.9.8k.tar.gz | more > openssl-0.9.8k/apps/ > openssl-0.9.8k/apps/app_rand.c > Downloading expat results in a corrupted file > mvn -X wagon:download-single -Dwagon.fromFile=expat-1.98.8.tar.gz > -Dwagon.url=http://prdownloads.sourceforge.net/expat -Dwagon.toDir=./ > $ tar tf expat-1.98.8.tar.gz > tar: This does not look like a tar archive > tar: Skipping to next header > tar: Read 1943 bytes from expat-1.98.8.tar.gz > tar: Error exit delayed from previous errors -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WAGON-264) compressed tarball download problems
[ http://jira.codehaus.org/browse/WAGON-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=175191#action_175191 ] Lee Thompson commented on WAGON-264: I'm reading rfc2616 and indeed, I agree with you that the openssl webserver is misconfigured deviating from HTTP spec. The webserver in this case transmitted Content-Type of gzip with no Content-Encoding but declared in the reply Content-Encoding of gzip. Unfortunately, it looks to me like the openssl config is following the Apache documentation to the letter http://httpd.apache.org/docs/2.2/mod/mod_mime.html#addencoding specifically --> To make this long story short, you should always use x-gzip and x-compress for these two specific encodings. The Apache docs are probably incorrect (or confusing at best) leading to this issue. I'd have to go back to the point that wget, and curl (and I just tested firefox) will all download from these sites and get the correct file. I'd also go back to my point that performance is worse when you ask for GZIP encoding for filetypes that are already compressed. The only typical maven uncompressed file types are the pom XML files and the SHA1 and MD5SUM hash files and these artifacts are so small that the benefit is negligible. While wagon might not have a bug in regards to rfc2616, it does have an issue dealing with sites who read the Apache configuration documentation. Wagon is the only http client I know of that can't download tarballs from sourceforge and openssl. (Lots of sourceforge mirrors have this config problem). The internet is a cooperative media and strict compliance is often compromised when interoperability concerns are brought in. Please make wagon defend itself from badly configured webservers. > compressed tarball download problems > > > Key: WAGON-264 > URL: http://jira.codehaus.org/browse/WAGON-264 > Project: Maven Wagon > Issue Type: Bug > Components: wagon-http-lightweight >Affects Versions: 1.0-beta-2, 1.0-beta-3, 1.0-beta-4, 1.0-beta-5 >Reporter: Lee Thompson >Assignee: John Casey > Fix For: 1.0-beta-5 > > > HTTP wagon will uncompress tarballs it downloads, sometimes badly. > Downloading openssl results in an unpressed download > $ mvn -X wagon:download-single -Dwagon.fromFile=openssl-0.9.8k.tar.gz > -Dwagon.url=http://www.openssl.org/source -Dwagon.toDir=./ > $ tar tzf openssl-0.9.8k.tar.gz > gzip: stdin: not in gzip format > tar: Child returned status 1 > tar: Error exit delayed from previous errors > $ tar tf openssl-0.9.8k.tar.gz | more > openssl-0.9.8k/apps/ > openssl-0.9.8k/apps/app_rand.c > Downloading expat results in a corrupted file > mvn -X wagon:download-single -Dwagon.fromFile=expat-1.98.8.tar.gz > -Dwagon.url=http://prdownloads.sourceforge.net/expat -Dwagon.toDir=./ > $ tar tf expat-1.98.8.tar.gz > tar: This does not look like a tar archive > tar: Skipping to next header > tar: Read 1943 bytes from expat-1.98.8.tar.gz > tar: Error exit delayed from previous errors -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WAGON-264) compressed tarball download problems
[
http://jira.codehaus.org/browse/WAGON-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=175089#action_175089
]
John Casey commented on WAGON-264:
--
Sorry, the output for the expat download didn't seem correct, so I retried with
a URL I got from browsing to the expat project:
{noformat}
Betelgeuse:wagon-plugin-tests jdcasey$ curl -X HEAD -i
http://softlayer.dl.sourceforge.net/sourceforge/expat/expat-1.95.8.tar.gz
HTTP/1.1 200 OK
Date: Mon, 04 May 2009 20:42:45 GMT
Server: Apache
Last-Modified: Sat, 24 Jul 2004 05:36:30 GMT
ETag: "3136801c-4db8d-22419380"
Accept-Ranges: bytes
Content-Length: 318349
Connection: close
Content-Type: application/x-gzip
{noformat}
> compressed tarball download problems
>
>
> Key: WAGON-264
> URL: http://jira.codehaus.org/browse/WAGON-264
> Project: Maven Wagon
> Issue Type: Bug
> Components: wagon-http-lightweight
>Affects Versions: 1.0-beta-2, 1.0-beta-3, 1.0-beta-4, 1.0-beta-5
>Reporter: Lee Thompson
>
> HTTP wagon will uncompress tarballs it downloads, sometimes badly.
> Downloading openssl results in an unpressed download
> $ mvn -X wagon:download-single -Dwagon.fromFile=openssl-0.9.8k.tar.gz
> -Dwagon.url=http://www.openssl.org/source -Dwagon.toDir=./
> $ tar tzf openssl-0.9.8k.tar.gz
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> tar: Error exit delayed from previous errors
> $ tar tf openssl-0.9.8k.tar.gz | more
> openssl-0.9.8k/apps/
> openssl-0.9.8k/apps/app_rand.c
> Downloading expat results in a corrupted file
> mvn -X wagon:download-single -Dwagon.fromFile=expat-1.98.8.tar.gz
> -Dwagon.url=http://prdownloads.sourceforge.net/expat -Dwagon.toDir=./
> $ tar tf expat-1.98.8.tar.gz
> tar: This does not look like a tar archive
> tar: Skipping to next header
> tar: Read 1943 bytes from expat-1.98.8.tar.gz
> tar: Error exit delayed from previous errors
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WAGON-264) compressed tarball download problems
[
http://jira.codehaus.org/browse/WAGON-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=175087#action_175087
]
John Casey commented on WAGON-264:
--
This looks like a configuration issue on the openssl.org website. The first,
openssl, is using content-encoding 'x-gzip' and appears not to actually be
re-compressing the file, while the other two websites use no content-encoding.
I'm guessing they messed up something similar to this:
{noformat}
AddType x-gzip .gz
{noformat}
Here's the output from hitting three different servers with curl:
{noformat}
Betelgeuse:wagon-plugin-tests jdcasey$ curl -X HEAD -i
http://www.openssl.org/source/openssl-0.9.8k.tar.gz
HTTP/1.1 200 OK
Date: Mon, 04 May 2009 17:40:48 GMT
Server: Apache/1.3.33 (OpenPKG/2.5)
Last-Modified: Wed, 25 Mar 2009 12:21:40 GMT
ETag: "f0c99a-3ac7e3-49ca21d4"
Accept-Ranges: bytes
Content-Length: 3852259
Content-Type: application/x-tar
Content-Encoding: x-gzip
^C
Betelgeuse:wagon-plugin-tests jdcasey$
Betelgeuse:wagon-plugin-tests jdcasey$
Betelgeuse:wagon-plugin-tests jdcasey$ curl -X HEAD -i
http://prdownloads.sourceforge.net/expat/expat-1.98.8.tar.gz
HTTP/1.1 301 Moved Permanently
X-Powered-By: PHP/5.2.8
Location: http://sourceforge.net/projects/expat/files
Content-type: text/html
Date: Mon, 04 May 2009 20:30:07 GMT
Server: lighttpd/1.4.19
Betelgeuse:wagon-plugin-tests jdcasey$ curl -X HEAD -i
http://repo2.maven.org/maven2/org/apache/maven/apache-maven/2.0.10/apache-maven-2.0.10-bin.tar.gz
HTTP/1.1 200 OK
Server: nginx/0.7.19
Date: Mon, 04 May 2009 20:31:13 GMT
Content-Type: application/octet-stream
Content-Length: 2101451
Last-Modified: Tue, 10 Feb 2009 02:58:01 GMT
Connection: keep-alive
Accept-Ranges: bytes
{noformat}
> compressed tarball download problems
>
>
> Key: WAGON-264
> URL: http://jira.codehaus.org/browse/WAGON-264
> Project: Maven Wagon
> Issue Type: Bug
> Components: wagon-http-lightweight
>Affects Versions: 1.0-beta-2, 1.0-beta-3, 1.0-beta-4, 1.0-beta-5
>Reporter: Lee Thompson
>
> HTTP wagon will uncompress tarballs it downloads, sometimes badly.
> Downloading openssl results in an unpressed download
> $ mvn -X wagon:download-single -Dwagon.fromFile=openssl-0.9.8k.tar.gz
> -Dwagon.url=http://www.openssl.org/source -Dwagon.toDir=./
> $ tar tzf openssl-0.9.8k.tar.gz
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> tar: Error exit delayed from previous errors
> $ tar tf openssl-0.9.8k.tar.gz | more
> openssl-0.9.8k/apps/
> openssl-0.9.8k/apps/app_rand.c
> Downloading expat results in a corrupted file
> mvn -X wagon:download-single -Dwagon.fromFile=expat-1.98.8.tar.gz
> -Dwagon.url=http://prdownloads.sourceforge.net/expat -Dwagon.toDir=./
> $ tar tf expat-1.98.8.tar.gz
> tar: This does not look like a tar archive
> tar: Skipping to next header
> tar: Read 1943 bytes from expat-1.98.8.tar.gz
> tar: Error exit delayed from previous errors
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WAGON-264) compressed tarball download problems
[ http://jira.codehaus.org/browse/WAGON-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=174980#action_174980 ] Lee Thompson commented on WAGON-264: Looks like the client can not see the bypass behavior of mod_gzip. If you run this little groovy program, it will dump the request headers... URL url = new URL ( "http://prdownloads.sourceforge.net/expat/expat-1.98.8.tar.gz"; ); HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection(); urlConnection.setRequestProperty( "Accept-Encoding", "gzip" ); int responseCode = urlConnection.getResponseCode(); println "Response code is " + responseCode Map> hdrFields = urlConnection.getHeaderFields() println hdrFields output... Response code is 200 [X-Powered-By:[PHP/5.2.8], Set-Cookie:[PHPSESSID=7f4eb0ee669a6363aa7c1e80b4980b0e; expires=Sunday, 17-May-09 13:23:42 GMT; path=/], X-SFX-Webhead:[sfs-web-7], null:[HTTP/1.1 200 OK], X-SFX-Revision:[release_20090417], Date:[Sun, 03 May 2009 13:23:42 GMT], Pragma:[no-cache], Content-type:[text/html], Cache-Control:[no-store, no-cache, must-revalidate, post-check=0, pre-check=0], Content-Encoding:[gzip], Expires:[Thu, 19 Nov 1981 08:52:00 GMT], Server:[lighttpd/1.4.19], Transfer-Encoding:[chunked], Vary:[Accept-Encoding, User-Agent]] Suggest a "getCompressedFileExtension()" method to Resource.java which checks the file extenstion for common internet compressed file types like jar, zip, gz, tgz, bz2 and if getCompressedFileExtension() is true, don't as for the gzip encoding and don't check for it either. This check would be needed fro http and https. > compressed tarball download problems > > > Key: WAGON-264 > URL: http://jira.codehaus.org/browse/WAGON-264 > Project: Maven Wagon > Issue Type: Bug > Components: wagon-http-lightweight >Affects Versions: 1.0-beta-2, 1.0-beta-3, 1.0-beta-4, 1.0-beta-5 >Reporter: Lee Thompson > > HTTP wagon will uncompress tarballs it downloads, sometimes badly. > Downloading openssl results in an unpressed download > $ mvn -X wagon:download-single -Dwagon.fromFile=openssl-0.9.8k.tar.gz > -Dwagon.url=http://www.openssl.org/source -Dwagon.toDir=./ > $ tar tzf openssl-0.9.8k.tar.gz > gzip: stdin: not in gzip format > tar: Child returned status 1 > tar: Error exit delayed from previous errors > $ tar tf openssl-0.9.8k.tar.gz | more > openssl-0.9.8k/apps/ > openssl-0.9.8k/apps/app_rand.c > Downloading expat results in a corrupted file > mvn -X wagon:download-single -Dwagon.fromFile=expat-1.98.8.tar.gz > -Dwagon.url=http://prdownloads.sourceforge.net/expat -Dwagon.toDir=./ > $ tar tf expat-1.98.8.tar.gz > tar: This does not look like a tar archive > tar: Skipping to next header > tar: Read 1943 bytes from expat-1.98.8.tar.gz > tar: Error exit delayed from previous errors -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WAGON-264) compressed tarball download problems
[ http://jira.codehaus.org/browse/WAGON-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=174975#action_174975 ] Lee Thompson commented on WAGON-264: I believe the problem is the wagon-http-lightweight is ignoring the DECLINED:FEXT_GZ status code returned by mod_gzip. When this error code is seen, mod_gzip passed on dynamic compresson since the request is for an already compressed gzip file. Wagon requesting mod_gzip on ZIP and JAR files doesn't seem to trigger this behavior. It would be an optimization to not ask for compression on JAR and ZIP files as these files are compressed but mod_gzip doesn't seem to know that. Double compressing consumes time on both the client and server with no decrease in download payload (download sometimes increases somewhat) > compressed tarball download problems > > > Key: WAGON-264 > URL: http://jira.codehaus.org/browse/WAGON-264 > Project: Maven Wagon > Issue Type: Bug > Components: wagon-http-lightweight >Affects Versions: 1.0-beta-2, 1.0-beta-3, 1.0-beta-4, 1.0-beta-5 >Reporter: Lee Thompson > > HTTP wagon will uncompress tarballs it downloads, sometimes badly. > Downloading openssl results in an unpressed download > $ mvn -X wagon:download-single -Dwagon.fromFile=openssl-0.9.8k.tar.gz > -Dwagon.url=http://www.openssl.org/source -Dwagon.toDir=./ > $ tar tzf openssl-0.9.8k.tar.gz > gzip: stdin: not in gzip format > tar: Child returned status 1 > tar: Error exit delayed from previous errors > $ tar tf openssl-0.9.8k.tar.gz | more > openssl-0.9.8k/apps/ > openssl-0.9.8k/apps/app_rand.c > Downloading expat results in a corrupted file > mvn -X wagon:download-single -Dwagon.fromFile=expat-1.98.8.tar.gz > -Dwagon.url=http://prdownloads.sourceforge.net/expat -Dwagon.toDir=./ > $ tar tf expat-1.98.8.tar.gz > tar: This does not look like a tar archive > tar: Skipping to next header > tar: Read 1943 bytes from expat-1.98.8.tar.gz > tar: Error exit delayed from previous errors -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
