[
https://issues.apache.org/jira/browse/IVY-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875099#action_12875099
]
Constantine Plotnikov commented on IVY-1194:
--------------------------------------------
It might be squid related. When there caching is disabled, the following
request/response sequence happens:
{code}
---request begin---
GET /maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar HTTP/1.0
User-Agent: Mozilla/Test
Accept: */*
Host: repo1.maven.org
Connection: Keep-Alive
Accept-Encoding: gzip
Cache-Control: no-cache
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.0 200 OK
Server: nginx/0.7.62
Date: Thu, 03 Jun 2010 13:20:31 GMT
Content-Type: application/java-archive
Last-Modified: Mon, 12 Oct 2009 21:19:02 GMT
Content-Encoding: gzip
X-Cache: MISS from squid_havp_node1
X-Cache-Lookup: HIT from squid_havp_node1:3128
X-Cache: MISS from squid_havp_node1
X-Cache-Lookup: HIT from squid_havp_node1:3128
Connection: close
---response end---
{code}
As you can see, the "Content-Length:" header is not specified (and thus cannot
be checked). But squid cache hit, the content length is specified (and possibly
checked).
{code}
---request begin---
GET /maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar HTTP/1.0
User-Agent: Mozilla/Test
Accept: */*
Host: repo1.maven.org
Connection: Keep-Alive
Accept-Encoding: identity
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.0 200 OK
Server: nginx/0.7.62
Date: Thu, 03 Jun 2010 13:20:31 GMT
Content-Type: application/java-archive
Last-Modified: Mon, 12 Oct 2009 21:19:02 GMT
Content-Encoding: gzip
X-Cache: MISS from squid_havp_node1
X-Cache-Lookup: HIT from squid_havp_node1:3128
Age: 111
Content-Length: 834513
X-Cache: HIT from squid_havp_node1
X-Cache-Lookup: HIT from squid_havp_node1:3128
Connection: keep-alive
---response end---
{code}
So the possibly the problem surface only when squid has a complete file in the
cache. The fix might be ignoring "Content-Length" header for responses that
have "Content-Encoding" headers.
> Downloads from maven repository fail when using transparent HTTP proxies
> ------------------------------------------------------------------------
>
> Key: IVY-1194
> URL: https://issues.apache.org/jira/browse/IVY-1194
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.1.0
> Environment: Windows XP, Java 1.5.0.16, java 1.6.0.20
> Reporter: Constantine Plotnikov
> Assignee: Maarten Coene
> Priority: Blocker
> Fix For: 2.2.0-RC1
>
>
> The package without dependencies cannot fetch dependencies and samples in the
> package with dependencies do not work.
> Fetching dependencies fails with the following log:
> {quote}
> Buildfile: build.xml
> init-ivy:
> retrieve-all:
> [ivy:retrieve] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ ::
> [ivy:retrieve] :: loading settings :: url =
> jar:file:/C:/utils/apache/apache-ivy-2.1.0-wo/ivy.jar!/org/apache/ivy/core/settings/ivysettings.xml
> [ivy:retrieve] :: resolving dependencies :: org.apache.ivy#ivy;2.1.0
> [ivy:retrieve] confs: [core, httpclient, oro, vfs, sftp, standalone,
> ant, default, test, source]
> [ivy:retrieve] found commons-httpclient#commons-httpclient;3.0 in
> public
> [ivy:retrieve] found commons-codec#commons-codec;1.2 in public
> [ivy:retrieve] found oro#oro;2.0.8 in public
> [ivy:retrieve] found commons-vfs#commons-vfs;1.0 in public
> [ivy:retrieve] found com.jcraft#jsch;0.1.31 in public
> [ivy:retrieve] found ant#ant;1.6.2 in public
> [ivy:retrieve] found ant#ant-nodeps;1.6.2 in public
> [ivy:retrieve] found ant#ant-trax;1.6.2 in public
> [ivy:retrieve] found junit#junit;3.8.2 in public
> [ivy:retrieve] found commons-lang#commons-lang;2.4 in public
> [ivy:retrieve] found org.apache.ant#ant-testutil;1.7.0 in public
> [ivy:retrieve] found ant#ant-launcher;1.6.2 in public
> [ivy:retrieve] found xerces#xercesImpl;2.6.2 in public
> [ivy:retrieve] found xerces#xmlParserAPIs;2.6.2 in public
> [ivy:retrieve] downloading
> http://repo1.maven.org/maven2/junit/junit/3.8.2/junit-3.8.2.jar ...
> [ivy:retrieve] ............ (117kB)
> [ivy:retrieve] .. (0kB)
> [ivy:retrieve] :: resolution report :: resolve 1250ms :: artifacts dl 250ms
> [ivy:retrieve] :: evicted modules:
> [ivy:retrieve] commons-logging#commons-logging;1.0.3 by
> [commons-logging#commons-logging;1.0.4] in [default]
> ---------------------------------------------------------------------
> | | modules || artifacts |
> | conf | number| search|dwnlded|evicted|| number|dwnlded|
> ---------------------------------------------------------------------
> | core | 0 | 0 | 0 | 0 || 0 | 0 |
> | httpclient | 3 | 0 | 0 | 0 || 2 | 0 |
> | oro | 1 | 0 | 0 | 0 || 1 | 0 |
> | vfs | 2 | 0 | 0 | 0 || 1 | 0 |
> | sftp | 1 | 0 | 0 | 0 || 1 | 0 |
> | standalone | 0 | 0 | 0 | 0 || 0 | 0 |
> | ant | 1 | 0 | 0 | 0 || 1 | 0 |
> | default | 10 | 0 | 0 | 1 || 8 | 0 |
> | test | 6 | 0 | 0 | 0 || 6 | 0 |
> | source | 0 | 0 | 0 | 0 || 0 | 0 |
> ---------------------------------------------------------------------
> [ivy:retrieve] :: problems summary ::
> [ivy:retrieve] :::: WARNINGS
> [ivy:retrieve] problem while downloading module descriptor:
> http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.3/commons-logging-1.0.3.pom:
> invalid sha1: expected=<
> [ivy:retrieve] г computed=88c58ea4a562116ab15fb76c9097ee1f25cc750b (62ms)
> [ivy:retrieve] module not found:
> commons-logging#commons-logging;1.0.3
> [ivy:retrieve] ==== local: tried
> [ivy:retrieve] C:\Documents and
> Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.3/ivys/ivy.xml
> [ivy:retrieve] -- artifact
> commons-logging#commons-logging;1.0.3!commons-logging.jar:
> [ivy:retrieve] C:\Documents and
> Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.3/jars/commons-logging.jar
> [ivy:retrieve] ==== shared: tried
> [ivy:retrieve] C:\Documents and
> Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.3/ivys/ivy.xml
> [ivy:retrieve] -- artifact
> commons-logging#commons-logging;1.0.3!commons-logging.jar:
> [ivy:retrieve] C:\Documents and
> Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.3/jars/commons-logging.jar
> [ivy:retrieve] ==== public: tried
> [ivy:retrieve]
> http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.3/commons-logging-1.0.3.pom
> [ivy:retrieve] problem while downloading module descriptor:
> http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.pom:
> invalid sha1: expected=<
> [ivy:retrieve] module not found:
> commons-logging#commons-logging;1.0.4
> [ivy:retrieve] ==== local: tried
> [ivy:retrieve] C:\Documents and
> Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.4/ivys/ivy.xml
> [ivy:retrieve] -- artifact
> commons-logging#commons-logging;1.0.4!commons-logging.jar:
> [ivy:retrieve] C:\Documents and
> Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.4/jars/commons-logging.jar
> [ivy:retrieve] ==== shared: tried
> [ivy:retrieve] C:\Documents and
> Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.4/ivys/ivy.xml
> [ivy:retrieve] -- artifact
> commons-logging#commons-logging;1.0.4!commons-logging.jar:
> [ivy:retrieve] C:\Documents and
> Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.4/jars/commons-logging.jar
> [ivy:retrieve] ==== public: tried
> [ivy:retrieve]
> http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.pom
> [ivy:retrieve] [FAILED ] junit#junit;3.8.2!junit.jar:
> invalid sha1: expected=<
> [ivy:retrieve] [FAILED ] junit#junit;3.8.2!junit.jar:
> (0ms)
> [ivy:retrieve] ==== shared: tried
> [ivy:retrieve] C:\Documents and
> Settings\Constantin.Plotnikov\.ivy2/shared/junit/junit/3.8.2/jars/junit.jar
> [ivy:retrieve] ==== public: tried
> [ivy:retrieve]
> http://repo1.maven.org/maven2/junit/junit/3.8.2/junit-3.8.2.jar
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve] :: UNRESOLVED DEPENDENCIES ::
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve] :: commons-logging#commons-logging;1.0.3: not
> found
> [ivy:retrieve] :: commons-logging#commons-logging;1.0.4: not
> found
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve]
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve] :: FAILED DOWNLOADS ::
> [ivy:retrieve] :: ^ see resolution messages for details ^ ::
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve] :: junit#junit;3.8.2!junit.jar
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve]
> [ivy:retrieve]
> [ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
> {quote}
> The sample in package with dependencies fails with the following log:
> {quote}
> Buildfile: build.xml
> download-ivy:
> [mkdir] Created dir:
> C:\utils\apache\apache-ivy-2.1.0\src\example\go-ivy\ivy
> [echo] installing ivy...
> [get] Getting:
> http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
> [get] To:
> C:\utils\apache\apache-ivy-2.1.0\src\example\go-ivy\ivy\ivy.jar
> install-ivy:
> [taskdef] java.util.zip.ZipException: error in opening zip file
> [taskdef] at java.util.zip.ZipFile.open(Native Method)
> [taskdef] at java.util.zip.ZipFile.<init>(ZipFile.java:203)
> [taskdef] at java.util.zip.ZipFile.<init>(ZipFile.java:234)
> [taskdef] at
> org.apache.tools.ant.AntClassLoader.getResourceURL(AntClassLoader.java:919)
> [taskdef] at
> org.apache.tools.ant.AntClassLoader$ResourceEnumeration.findNextResource(AntClassLoader.java:126)
> [taskdef] at
> org.apache.tools.ant.AntClassLoader$ResourceEnumeration.<init>(AntClassLoader.java:88)
> [taskdef] at
> org.apache.tools.ant.AntClassLoader.findResources(AntClassLoader.java:869)
> [taskdef] at java.lang.ClassLoader.getResources(ClassLoader.java:1015)
> [taskdef] at
> org.apache.tools.ant.taskdefs.Definer.resourceToURLs(Definer.java:267)
> [taskdef] at
> org.apache.tools.ant.taskdefs.Definer.execute(Definer.java:211)
> [taskdef] at
> org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275)
> [taskdef] at org.apache.tools.ant.Task.perform(Task.java:364)
> [taskdef] at org.apache.tools.ant.Target.execute(Target.java:341)
> [taskdef] at org.apache.tools.ant.Target.performTasks(Target.java:369)
> [taskdef] at
> org.apache.tools.ant.Project.executeSortedTargets(Project.java:1216)
> [taskdef] at org.apache.tools.ant.Project.executeTarget(Project.java:1185)
> [taskdef] at
> org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:40)
> [taskdef] at
> org.apache.tools.ant.Project.executeTargets(Project.java:1068)
> [taskdef] at org.apache.tools.ant.Main.runBuild(Main.java:668)
> [taskdef] at org.apache.tools.ant.Main.startAnt(Main.java:187)
> [taskdef] at org.apache.tools.ant.launch.Launcher.run(Launcher.java:246)
> [taskdef] at org.apache.tools.ant.launch.Launcher.main(Launcher.java:67)
> [taskdef] Could not load definitions from resource
> org/apache/ivy/ant/antlib.xml. It could not be found.
> generate-src:
> go:
> [echo] using ivy to resolve commons-lang 2.1...
> BUILD FAILED
> C:\utils\apache\apache-ivy-2.1.0\src\example\go-ivy\build.xml:85: Could not
> create task or type of type: antlib:org.apache.ivy.ant:cachepath.
> Ant could not find the task or a class this task relies upon.
> This is common and has a number of causes; the usual
> solutions are to read the manual pages then download and
> install needed JAR files, or fix the build file:
> - You have misspelt 'antlib:org.apache.ivy.ant:cachepath'.
> Fix: check your spelling.
> - The task needs an external JAR file to execute
> and this is not found at the right place in the classpath.
> Fix: check the documentation for dependencies.
> Fix: declare the task.
> - The task is an Ant optional task and the JAR file and/or libraries
> implementing the functionality were not found at the time you
> yourself built your installation of Ant from the Ant sources.
> Fix: Look in the ANT_HOME/lib for the 'ant-' JAR corresponding to the
> task and make sure it contains more than merely a META-INF/MANIFEST.MF.
> If all it contains is the manifest, then rebuild Ant with the needed
> libraries present in ${ant.home}/lib/optional/ , or alternatively,
> download a pre-built release version from apache.org
> - The build file was written for a later version of Ant
> Fix: upgrade to at least the latest release version of Ant
> - The task is not an Ant core or optional task
> and needs to be declared using <taskdef>.
> - You are attempting to use a task defined using
> <presetdef> or <macrodef> but have spelt wrong or not
> defined it at the point of use
> Remember that for JAR files to be visible to Ant tasks implemented
> in ANT_HOME/lib, the files must be in the same directory or on the
> classpath
> Please neither file bug reports on this problem, nor email the
> Ant mailing lists, until all of these causes have been explored,
> as this is not an Ant bug.
> Total time: 1 second
> {quote}
> The reason for the bug is the squid. When request comes and it has a cached
> content with gzip content encoding, it return a content with encoded content,
> event if no encoding was specified in the request or "Accept-Encoding:
> identity" was specified. The below is a sample log that demonstrates the
> problem:
> {quote}
> DEBUG output created by Wget 1.11.4 on cygwin.
> --2010-06-02 12:41:55--
> http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
> Resolving repo1.maven.org... 38.97.124.18
> Caching repo1.maven.org => 38.97.124.18
> Connecting to repo1.maven.org|38.97.124.18|:80... connected.
> Created socket 4.
> Releasing 0x006d9100 (new refcount 1).
> ---request begin---
> GET /maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar HTTP/1.0
> User-Agent: Wget/1.11.4
> Accept: */*
> Host: repo1.maven.org
> Connection: Keep-Alive
> ---request end---
> HTTP request sent, awaiting response...
> ---response begin---
> HTTP/1.0 200 OK
> Server: nginx/0.7.62
> Date: Tue, 01 Jun 2010 13:09:36 GMT
> Content-Type: application/java-archive
> Last-Modified: Mon, 12 Oct 2009 21:19:02 GMT
> Content-Encoding: gzip
> X-Cache: MISS from squid_havp_node1
> X-Cache-Lookup: MISS from squid_havp_node1:3128
> Age: 70339
> Content-Length: 834513
> X-Cache: HIT from squid_havp_node1
> X-Cache-Lookup: HIT from squid_havp_node1:3128
> Connection: keep-alive
> ---response end---
> HTTP/1.0 200 OK
> Server: nginx/0.7.62
> Date: Tue, 01 Jun 2010 13:09:36 GMT
> Content-Type: application/java-archive
> Last-Modified: Mon, 12 Oct 2009 21:19:02 GMT
> Content-Encoding: gzip
> X-Cache: MISS from squid_havp_node1
> X-Cache-Lookup: MISS from squid_havp_node1:3128
> Age: 70339
> Content-Length: 834513
> X-Cache: HIT from squid_havp_node1
> X-Cache-Lookup: HIT from squid_havp_node1:3128
> Connection: keep-alive
> Registered socket 4 for persistent reuse.
> Length: 834513 (815K) [application/java-archive]
> Saving to: `ivy-2.1.0.jar'
> 0K .......... .......... .......... .......... .......... 6% 8.49M 0s
> 50K .......... .......... .......... .......... .......... 12% 12.0M 0s
> 100K .......... .......... .......... .......... .......... 18% 7.26M 0s
> 150K .......... .......... .......... .......... .......... 24% 9.28M 0s
> 200K .......... .......... .......... .......... .......... 30% 5.33M 0s
> 250K .......... .......... .......... .......... .......... 36% 9.31M 0s
> 300K .......... .......... .......... .......... .......... 42% 5.26M 0s
> 350K .......... .......... .......... .......... .......... 49% 9.15M 0s
> 400K .......... .......... .......... .......... .......... 55% 7.27M 0s
> 450K .......... .......... .......... .......... .......... 61% 7.42M 0s
> 500K .......... .......... .......... .......... .......... 67% 9.21M 0s
> 550K .......... .......... .......... .......... .......... 73% 6.11M 0s
> 600K .......... .......... .......... .......... .......... 79% 9.50M 0s
> 650K .......... .......... .......... .......... .......... 85% 8.84M 0s
> 700K .......... .......... .......... .......... .......... 92% 9.54M 0s
> 750K .......... .......... .......... .......... .......... 98% 6.04M 0s
> 800K .......... .... 100% 16.2K=1.0s
> 2010-06-02 12:41:56 (795 KB/s) - `ivy-2.1.0.jar' saved [834513/834513]
> {quote}
> In our environment, a bunch of files from repo1.maven.org are cached in the
> squid. So there were a lot of conflicts with content encoding.
> The possible workarounds:
> 1. Check returned content encoding from http client, if it is not identity,
> reissue the request with disabling the caches.
> 2. Ungzip files fetched with gzip content encoding.
> 3. Support gzip content encoding in common HTTP.
> 3. Use local maven proxies
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
