Downloads from maven repository fail when using transparent HTTP proxies
------------------------------------------------------------------------
Key: IVY-1194
URL: https://issues.apache.org/jira/browse/IVY-1194
Project: Ivy
Issue Type: Bug
Components: Maven Compatibility
Affects Versions: 2.1.0
Environment: Windows XP, Java 1.5.0.16, java 1.6.0.20
Reporter: Constantine Plotnikov
Priority: Blocker
The package without dependencies cannot fetch dependencies and samples in the
package with dependencies do not work.
Fetching dependencies fails with the following log:
{quote}
Buildfile: build.xml
init-ivy:
retrieve-all:
[ivy:retrieve] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ ::
[ivy:retrieve] :: loading settings :: url =
jar:file:/C:/utils/apache/apache-ivy-2.1.0-wo/ivy.jar!/org/apache/ivy/core/settings/ivysettings.xml
[ivy:retrieve] :: resolving dependencies :: org.apache.ivy#ivy;2.1.0
[ivy:retrieve] confs: [core, httpclient, oro, vfs, sftp, standalone, ant,
default, test, source]
[ivy:retrieve] found commons-httpclient#commons-httpclient;3.0 in public
[ivy:retrieve] found commons-codec#commons-codec;1.2 in public
[ivy:retrieve] found oro#oro;2.0.8 in public
[ivy:retrieve] found commons-vfs#commons-vfs;1.0 in public
[ivy:retrieve] found com.jcraft#jsch;0.1.31 in public
[ivy:retrieve] found ant#ant;1.6.2 in public
[ivy:retrieve] found ant#ant-nodeps;1.6.2 in public
[ivy:retrieve] found ant#ant-trax;1.6.2 in public
[ivy:retrieve] found junit#junit;3.8.2 in public
[ivy:retrieve] found commons-lang#commons-lang;2.4 in public
[ivy:retrieve] found org.apache.ant#ant-testutil;1.7.0 in public
[ivy:retrieve] found ant#ant-launcher;1.6.2 in public
[ivy:retrieve] found xerces#xercesImpl;2.6.2 in public
[ivy:retrieve] found xerces#xmlParserAPIs;2.6.2 in public
[ivy:retrieve] downloading
http://repo1.maven.org/maven2/junit/junit/3.8.2/junit-3.8.2.jar ...
[ivy:retrieve] ............ (117kB)
[ivy:retrieve] .. (0kB)
[ivy:retrieve] :: resolution report :: resolve 1250ms :: artifacts dl 250ms
[ivy:retrieve] :: evicted modules:
[ivy:retrieve] commons-logging#commons-logging;1.0.3 by
[commons-logging#commons-logging;1.0.4] in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| core | 0 | 0 | 0 | 0 || 0 | 0 |
| httpclient | 3 | 0 | 0 | 0 || 2 | 0 |
| oro | 1 | 0 | 0 | 0 || 1 | 0 |
| vfs | 2 | 0 | 0 | 0 || 1 | 0 |
| sftp | 1 | 0 | 0 | 0 || 1 | 0 |
| standalone | 0 | 0 | 0 | 0 || 0 | 0 |
| ant | 1 | 0 | 0 | 0 || 1 | 0 |
| default | 10 | 0 | 0 | 1 || 8 | 0 |
| test | 6 | 0 | 0 | 0 || 6 | 0 |
| source | 0 | 0 | 0 | 0 || 0 | 0 |
---------------------------------------------------------------------
[ivy:retrieve] :: problems summary ::
[ivy:retrieve] :::: WARNINGS
[ivy:retrieve] problem while downloading module descriptor:
http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.3/commons-logging-1.0.3.pom:
invalid sha1: expected=<
[ivy:retrieve] г computed=88c58ea4a562116ab15fb76c9097ee1f25cc750b (62ms)
[ivy:retrieve] module not found: commons-logging#commons-logging;1.0.3
[ivy:retrieve] ==== local: tried
[ivy:retrieve] C:\Documents and
Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.3/ivys/ivy.xml
[ivy:retrieve] -- artifact
commons-logging#commons-logging;1.0.3!commons-logging.jar:
[ivy:retrieve] C:\Documents and
Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.3/jars/commons-logging.jar
[ivy:retrieve] ==== shared: tried
[ivy:retrieve] C:\Documents and
Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.3/ivys/ivy.xml
[ivy:retrieve] -- artifact
commons-logging#commons-logging;1.0.3!commons-logging.jar:
[ivy:retrieve] C:\Documents and
Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.3/jars/commons-logging.jar
[ivy:retrieve] ==== public: tried
[ivy:retrieve]
http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.3/commons-logging-1.0.3.pom
[ivy:retrieve] problem while downloading module descriptor:
http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.pom:
invalid sha1: expected=<
[ivy:retrieve] module not found: commons-logging#commons-logging;1.0.4
[ivy:retrieve] ==== local: tried
[ivy:retrieve] C:\Documents and
Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.4/ivys/ivy.xml
[ivy:retrieve] -- artifact
commons-logging#commons-logging;1.0.4!commons-logging.jar:
[ivy:retrieve] C:\Documents and
Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.4/jars/commons-logging.jar
[ivy:retrieve] ==== shared: tried
[ivy:retrieve] C:\Documents and
Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.4/ivys/ivy.xml
[ivy:retrieve] -- artifact
commons-logging#commons-logging;1.0.4!commons-logging.jar:
[ivy:retrieve] C:\Documents and
Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.4/jars/commons-logging.jar
[ivy:retrieve] ==== public: tried
[ivy:retrieve]
http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.pom
[ivy:retrieve] [FAILED ] junit#junit;3.8.2!junit.jar: invalid
sha1: expected=<
[ivy:retrieve] [FAILED ] junit#junit;3.8.2!junit.jar: (0ms)
[ivy:retrieve] ==== shared: tried
[ivy:retrieve] C:\Documents and
Settings\Constantin.Plotnikov\.ivy2/shared/junit/junit/3.8.2/jars/junit.jar
[ivy:retrieve] ==== public: tried
[ivy:retrieve]
http://repo1.maven.org/maven2/junit/junit/3.8.2/junit-3.8.2.jar
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: UNRESOLVED DEPENDENCIES ::
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: commons-logging#commons-logging;1.0.3: not found
[ivy:retrieve] :: commons-logging#commons-logging;1.0.4: not found
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve]
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: FAILED DOWNLOADS ::
[ivy:retrieve] :: ^ see resolution messages for details ^ ::
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: junit#junit;3.8.2!junit.jar
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve]
[ivy:retrieve]
[ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
{quote}
The sample in package with dependencies fails with the following log:
{quote}
Buildfile: build.xml
download-ivy:
[mkdir] Created dir: C:\utils\apache\apache-ivy-2.1.0\src\example\go-ivy\ivy
[echo] installing ivy...
[get] Getting:
http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
[get] To: C:\utils\apache\apache-ivy-2.1.0\src\example\go-ivy\ivy\ivy.jar
install-ivy:
[taskdef] java.util.zip.ZipException: error in opening zip file
[taskdef] at java.util.zip.ZipFile.open(Native Method)
[taskdef] at java.util.zip.ZipFile.<init>(ZipFile.java:203)
[taskdef] at java.util.zip.ZipFile.<init>(ZipFile.java:234)
[taskdef] at
org.apache.tools.ant.AntClassLoader.getResourceURL(AntClassLoader.java:919)
[taskdef] at
org.apache.tools.ant.AntClassLoader$ResourceEnumeration.findNextResource(AntClassLoader.java:126)
[taskdef] at
org.apache.tools.ant.AntClassLoader$ResourceEnumeration.<init>(AntClassLoader.java:88)
[taskdef] at
org.apache.tools.ant.AntClassLoader.findResources(AntClassLoader.java:869)
[taskdef] at java.lang.ClassLoader.getResources(ClassLoader.java:1015)
[taskdef] at
org.apache.tools.ant.taskdefs.Definer.resourceToURLs(Definer.java:267)
[taskdef] at
org.apache.tools.ant.taskdefs.Definer.execute(Definer.java:211)
[taskdef] at
org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275)
[taskdef] at org.apache.tools.ant.Task.perform(Task.java:364)
[taskdef] at org.apache.tools.ant.Target.execute(Target.java:341)
[taskdef] at org.apache.tools.ant.Target.performTasks(Target.java:369)
[taskdef] at
org.apache.tools.ant.Project.executeSortedTargets(Project.java:1216)
[taskdef] at org.apache.tools.ant.Project.executeTarget(Project.java:1185)
[taskdef] at
org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:40)
[taskdef] at
org.apache.tools.ant.Project.executeTargets(Project.java:1068)
[taskdef] at org.apache.tools.ant.Main.runBuild(Main.java:668)
[taskdef] at org.apache.tools.ant.Main.startAnt(Main.java:187)
[taskdef] at org.apache.tools.ant.launch.Launcher.run(Launcher.java:246)
[taskdef] at org.apache.tools.ant.launch.Launcher.main(Launcher.java:67)
[taskdef] Could not load definitions from resource
org/apache/ivy/ant/antlib.xml. It could not be found.
generate-src:
go:
[echo] using ivy to resolve commons-lang 2.1...
BUILD FAILED
C:\utils\apache\apache-ivy-2.1.0\src\example\go-ivy\build.xml:85: Could not
create task or type of type: antlib:org.apache.ivy.ant:cachepath.
Ant could not find the task or a class this task relies upon.
This is common and has a number of causes; the usual
solutions are to read the manual pages then download and
install needed JAR files, or fix the build file:
- You have misspelt 'antlib:org.apache.ivy.ant:cachepath'.
Fix: check your spelling.
- The task needs an external JAR file to execute
and this is not found at the right place in the classpath.
Fix: check the documentation for dependencies.
Fix: declare the task.
- The task is an Ant optional task and the JAR file and/or libraries
implementing the functionality were not found at the time you
yourself built your installation of Ant from the Ant sources.
Fix: Look in the ANT_HOME/lib for the 'ant-' JAR corresponding to the
task and make sure it contains more than merely a META-INF/MANIFEST.MF.
If all it contains is the manifest, then rebuild Ant with the needed
libraries present in ${ant.home}/lib/optional/ , or alternatively,
download a pre-built release version from apache.org
- The build file was written for a later version of Ant
Fix: upgrade to at least the latest release version of Ant
- The task is not an Ant core or optional task
and needs to be declared using <taskdef>.
- You are attempting to use a task defined using
<presetdef> or <macrodef> but have spelt wrong or not
defined it at the point of use
Remember that for JAR files to be visible to Ant tasks implemented
in ANT_HOME/lib, the files must be in the same directory or on the
classpath
Please neither file bug reports on this problem, nor email the
Ant mailing lists, until all of these causes have been explored,
as this is not an Ant bug.
Total time: 1 second
{quote}
The reason for the bug is the squid. When request comes and it has a cached
content with gzip content encoding, it return a content with encoded content,
event if no encoding was specified in the request or "Accept-Encoding:
identity" was specified. The below is a sample log that demonstrates the
problem:
{quote}
DEBUG output created by Wget 1.11.4 on cygwin.
--2010-06-02 12:41:55--
http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
Resolving repo1.maven.org... 38.97.124.18
Caching repo1.maven.org => 38.97.124.18
Connecting to repo1.maven.org|38.97.124.18|:80... connected.
Created socket 4.
Releasing 0x006d9100 (new refcount 1).
---request begin---
GET /maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar HTTP/1.0
User-Agent: Wget/1.11.4
Accept: */*
Host: repo1.maven.org
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.0 200 OK
Server: nginx/0.7.62
Date: Tue, 01 Jun 2010 13:09:36 GMT
Content-Type: application/java-archive
Last-Modified: Mon, 12 Oct 2009 21:19:02 GMT
Content-Encoding: gzip
X-Cache: MISS from squid_havp_node1
X-Cache-Lookup: MISS from squid_havp_node1:3128
Age: 70339
Content-Length: 834513
X-Cache: HIT from squid_havp_node1
X-Cache-Lookup: HIT from squid_havp_node1:3128
Connection: keep-alive
---response end---
HTTP/1.0 200 OK
Server: nginx/0.7.62
Date: Tue, 01 Jun 2010 13:09:36 GMT
Content-Type: application/java-archive
Last-Modified: Mon, 12 Oct 2009 21:19:02 GMT
Content-Encoding: gzip
X-Cache: MISS from squid_havp_node1
X-Cache-Lookup: MISS from squid_havp_node1:3128
Age: 70339
Content-Length: 834513
X-Cache: HIT from squid_havp_node1
X-Cache-Lookup: HIT from squid_havp_node1:3128
Connection: keep-alive
Registered socket 4 for persistent reuse.
Length: 834513 (815K) [application/java-archive]
Saving to: `ivy-2.1.0.jar'
0K .......... .......... .......... .......... .......... 6% 8.49M 0s
50K .......... .......... .......... .......... .......... 12% 12.0M 0s
100K .......... .......... .......... .......... .......... 18% 7.26M 0s
150K .......... .......... .......... .......... .......... 24% 9.28M 0s
200K .......... .......... .......... .......... .......... 30% 5.33M 0s
250K .......... .......... .......... .......... .......... 36% 9.31M 0s
300K .......... .......... .......... .......... .......... 42% 5.26M 0s
350K .......... .......... .......... .......... .......... 49% 9.15M 0s
400K .......... .......... .......... .......... .......... 55% 7.27M 0s
450K .......... .......... .......... .......... .......... 61% 7.42M 0s
500K .......... .......... .......... .......... .......... 67% 9.21M 0s
550K .......... .......... .......... .......... .......... 73% 6.11M 0s
600K .......... .......... .......... .......... .......... 79% 9.50M 0s
650K .......... .......... .......... .......... .......... 85% 8.84M 0s
700K .......... .......... .......... .......... .......... 92% 9.54M 0s
750K .......... .......... .......... .......... .......... 98% 6.04M 0s
800K .......... .... 100% 16.2K=1.0s
2010-06-02 12:41:56 (795 KB/s) - `ivy-2.1.0.jar' saved [834513/834513]
{quote}
In our environment, a bunch of files from repo1.maven.org are cached in the
squid. So there were a lot of conflicts with content encoding.
The possible workarounds:
1. Check returned content encoding from http client, if it is not identity,
reissue the request with disabling the caches.
2. Ungzip files fetched with gzip content encoding.
3. Support gzip content encoding in common HTTP.
3. Use local maven proxies
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
