[ https://issues.apache.org/jira/browse/IVY-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875099#action_12875099 ]
Constantine Plotnikov commented on IVY-1194: -------------------------------------------- It might be squid related. When there caching is disabled, the following request/response sequence happens: {code} ---request begin--- GET /maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar HTTP/1.0 User-Agent: Mozilla/Test Accept: */* Host: repo1.maven.org Connection: Keep-Alive Accept-Encoding: gzip Cache-Control: no-cache ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.0 200 OK Server: nginx/0.7.62 Date: Thu, 03 Jun 2010 13:20:31 GMT Content-Type: application/java-archive Last-Modified: Mon, 12 Oct 2009 21:19:02 GMT Content-Encoding: gzip X-Cache: MISS from squid_havp_node1 X-Cache-Lookup: HIT from squid_havp_node1:3128 X-Cache: MISS from squid_havp_node1 X-Cache-Lookup: HIT from squid_havp_node1:3128 Connection: close ---response end--- {code} As you can see, the "Content-Length:" header is not specified (and thus cannot be checked). But squid cache hit, the content length is specified (and possibly checked). {code} ---request begin--- GET /maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar HTTP/1.0 User-Agent: Mozilla/Test Accept: */* Host: repo1.maven.org Connection: Keep-Alive Accept-Encoding: identity ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.0 200 OK Server: nginx/0.7.62 Date: Thu, 03 Jun 2010 13:20:31 GMT Content-Type: application/java-archive Last-Modified: Mon, 12 Oct 2009 21:19:02 GMT Content-Encoding: gzip X-Cache: MISS from squid_havp_node1 X-Cache-Lookup: HIT from squid_havp_node1:3128 Age: 111 Content-Length: 834513 X-Cache: HIT from squid_havp_node1 X-Cache-Lookup: HIT from squid_havp_node1:3128 Connection: keep-alive ---response end--- {code} So the possibly the problem surface only when squid has a complete file in the cache. The fix might be ignoring "Content-Length" header for responses that have "Content-Encoding" headers. > Downloads from maven repository fail when using transparent HTTP proxies > ------------------------------------------------------------------------ > > Key: IVY-1194 > URL: https://issues.apache.org/jira/browse/IVY-1194 > Project: Ivy > Issue Type: Bug > Components: Core > Affects Versions: 2.1.0 > Environment: Windows XP, Java 1.5.0.16, java 1.6.0.20 > Reporter: Constantine Plotnikov > Assignee: Maarten Coene > Priority: Blocker > Fix For: 2.2.0-RC1 > > > The package without dependencies cannot fetch dependencies and samples in the > package with dependencies do not work. > Fetching dependencies fails with the following log: > {quote} > Buildfile: build.xml > init-ivy: > retrieve-all: > [ivy:retrieve] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ :: > [ivy:retrieve] :: loading settings :: url = > jar:file:/C:/utils/apache/apache-ivy-2.1.0-wo/ivy.jar!/org/apache/ivy/core/settings/ivysettings.xml > [ivy:retrieve] :: resolving dependencies :: org.apache.ivy#ivy;2.1.0 > [ivy:retrieve] confs: [core, httpclient, oro, vfs, sftp, standalone, > ant, default, test, source] > [ivy:retrieve] found commons-httpclient#commons-httpclient;3.0 in > public > [ivy:retrieve] found commons-codec#commons-codec;1.2 in public > [ivy:retrieve] found oro#oro;2.0.8 in public > [ivy:retrieve] found commons-vfs#commons-vfs;1.0 in public > [ivy:retrieve] found com.jcraft#jsch;0.1.31 in public > [ivy:retrieve] found ant#ant;1.6.2 in public > [ivy:retrieve] found ant#ant-nodeps;1.6.2 in public > [ivy:retrieve] found ant#ant-trax;1.6.2 in public > [ivy:retrieve] found junit#junit;3.8.2 in public > [ivy:retrieve] found commons-lang#commons-lang;2.4 in public > [ivy:retrieve] found org.apache.ant#ant-testutil;1.7.0 in public > [ivy:retrieve] found ant#ant-launcher;1.6.2 in public > [ivy:retrieve] found xerces#xercesImpl;2.6.2 in public > [ivy:retrieve] found xerces#xmlParserAPIs;2.6.2 in public > [ivy:retrieve] downloading > http://repo1.maven.org/maven2/junit/junit/3.8.2/junit-3.8.2.jar ... > [ivy:retrieve] ............ (117kB) > [ivy:retrieve] .. (0kB) > [ivy:retrieve] :: resolution report :: resolve 1250ms :: artifacts dl 250ms > [ivy:retrieve] :: evicted modules: > [ivy:retrieve] commons-logging#commons-logging;1.0.3 by > [commons-logging#commons-logging;1.0.4] in [default] > --------------------------------------------------------------------- > | | modules || artifacts | > | conf | number| search|dwnlded|evicted|| number|dwnlded| > --------------------------------------------------------------------- > | core | 0 | 0 | 0 | 0 || 0 | 0 | > | httpclient | 3 | 0 | 0 | 0 || 2 | 0 | > | oro | 1 | 0 | 0 | 0 || 1 | 0 | > | vfs | 2 | 0 | 0 | 0 || 1 | 0 | > | sftp | 1 | 0 | 0 | 0 || 1 | 0 | > | standalone | 0 | 0 | 0 | 0 || 0 | 0 | > | ant | 1 | 0 | 0 | 0 || 1 | 0 | > | default | 10 | 0 | 0 | 1 || 8 | 0 | > | test | 6 | 0 | 0 | 0 || 6 | 0 | > | source | 0 | 0 | 0 | 0 || 0 | 0 | > --------------------------------------------------------------------- > [ivy:retrieve] :: problems summary :: > [ivy:retrieve] :::: WARNINGS > [ivy:retrieve] problem while downloading module descriptor: > http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.3/commons-logging-1.0.3.pom: > invalid sha1: expected=< > [ivy:retrieve] г computed=88c58ea4a562116ab15fb76c9097ee1f25cc750b (62ms) > [ivy:retrieve] module not found: > commons-logging#commons-logging;1.0.3 > [ivy:retrieve] ==== local: tried > [ivy:retrieve] C:\Documents and > Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.3/ivys/ivy.xml > [ivy:retrieve] -- artifact > commons-logging#commons-logging;1.0.3!commons-logging.jar: > [ivy:retrieve] C:\Documents and > Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.3/jars/commons-logging.jar > [ivy:retrieve] ==== shared: tried > [ivy:retrieve] C:\Documents and > Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.3/ivys/ivy.xml > [ivy:retrieve] -- artifact > commons-logging#commons-logging;1.0.3!commons-logging.jar: > [ivy:retrieve] C:\Documents and > Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.3/jars/commons-logging.jar > [ivy:retrieve] ==== public: tried > [ivy:retrieve] > http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.3/commons-logging-1.0.3.pom > [ivy:retrieve] problem while downloading module descriptor: > http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.pom: > invalid sha1: expected=< > [ivy:retrieve] module not found: > commons-logging#commons-logging;1.0.4 > [ivy:retrieve] ==== local: tried > [ivy:retrieve] C:\Documents and > Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.4/ivys/ivy.xml > [ivy:retrieve] -- artifact > commons-logging#commons-logging;1.0.4!commons-logging.jar: > [ivy:retrieve] C:\Documents and > Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.4/jars/commons-logging.jar > [ivy:retrieve] ==== shared: tried > [ivy:retrieve] C:\Documents and > Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.4/ivys/ivy.xml > [ivy:retrieve] -- artifact > commons-logging#commons-logging;1.0.4!commons-logging.jar: > [ivy:retrieve] C:\Documents and > Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.4/jars/commons-logging.jar > [ivy:retrieve] ==== public: tried > [ivy:retrieve] > http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.pom > [ivy:retrieve] [FAILED ] junit#junit;3.8.2!junit.jar: > invalid sha1: expected=< > [ivy:retrieve] [FAILED ] junit#junit;3.8.2!junit.jar: > (0ms) > [ivy:retrieve] ==== shared: tried > [ivy:retrieve] C:\Documents and > Settings\Constantin.Plotnikov\.ivy2/shared/junit/junit/3.8.2/jars/junit.jar > [ivy:retrieve] ==== public: tried > [ivy:retrieve] > http://repo1.maven.org/maven2/junit/junit/3.8.2/junit-3.8.2.jar > [ivy:retrieve] :::::::::::::::::::::::::::::::::::::::::::::: > [ivy:retrieve] :: UNRESOLVED DEPENDENCIES :: > [ivy:retrieve] :::::::::::::::::::::::::::::::::::::::::::::: > [ivy:retrieve] :: commons-logging#commons-logging;1.0.3: not > found > [ivy:retrieve] :: commons-logging#commons-logging;1.0.4: not > found > [ivy:retrieve] :::::::::::::::::::::::::::::::::::::::::::::: > [ivy:retrieve] > [ivy:retrieve] :::::::::::::::::::::::::::::::::::::::::::::: > [ivy:retrieve] :: FAILED DOWNLOADS :: > [ivy:retrieve] :: ^ see resolution messages for details ^ :: > [ivy:retrieve] :::::::::::::::::::::::::::::::::::::::::::::: > [ivy:retrieve] :: junit#junit;3.8.2!junit.jar > [ivy:retrieve] :::::::::::::::::::::::::::::::::::::::::::::: > [ivy:retrieve] > [ivy:retrieve] > [ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS > {quote} > The sample in package with dependencies fails with the following log: > {quote} > Buildfile: build.xml > download-ivy: > [mkdir] Created dir: > C:\utils\apache\apache-ivy-2.1.0\src\example\go-ivy\ivy > [echo] installing ivy... > [get] Getting: > http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar > [get] To: > C:\utils\apache\apache-ivy-2.1.0\src\example\go-ivy\ivy\ivy.jar > install-ivy: > [taskdef] java.util.zip.ZipException: error in opening zip file > [taskdef] at java.util.zip.ZipFile.open(Native Method) > [taskdef] at java.util.zip.ZipFile.<init>(ZipFile.java:203) > [taskdef] at java.util.zip.ZipFile.<init>(ZipFile.java:234) > [taskdef] at > org.apache.tools.ant.AntClassLoader.getResourceURL(AntClassLoader.java:919) > [taskdef] at > org.apache.tools.ant.AntClassLoader$ResourceEnumeration.findNextResource(AntClassLoader.java:126) > [taskdef] at > org.apache.tools.ant.AntClassLoader$ResourceEnumeration.<init>(AntClassLoader.java:88) > [taskdef] at > org.apache.tools.ant.AntClassLoader.findResources(AntClassLoader.java:869) > [taskdef] at java.lang.ClassLoader.getResources(ClassLoader.java:1015) > [taskdef] at > org.apache.tools.ant.taskdefs.Definer.resourceToURLs(Definer.java:267) > [taskdef] at > org.apache.tools.ant.taskdefs.Definer.execute(Definer.java:211) > [taskdef] at > org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275) > [taskdef] at org.apache.tools.ant.Task.perform(Task.java:364) > [taskdef] at org.apache.tools.ant.Target.execute(Target.java:341) > [taskdef] at org.apache.tools.ant.Target.performTasks(Target.java:369) > [taskdef] at > org.apache.tools.ant.Project.executeSortedTargets(Project.java:1216) > [taskdef] at org.apache.tools.ant.Project.executeTarget(Project.java:1185) > [taskdef] at > org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:40) > [taskdef] at > org.apache.tools.ant.Project.executeTargets(Project.java:1068) > [taskdef] at org.apache.tools.ant.Main.runBuild(Main.java:668) > [taskdef] at org.apache.tools.ant.Main.startAnt(Main.java:187) > [taskdef] at org.apache.tools.ant.launch.Launcher.run(Launcher.java:246) > [taskdef] at org.apache.tools.ant.launch.Launcher.main(Launcher.java:67) > [taskdef] Could not load definitions from resource > org/apache/ivy/ant/antlib.xml. It could not be found. > generate-src: > go: > [echo] using ivy to resolve commons-lang 2.1... > BUILD FAILED > C:\utils\apache\apache-ivy-2.1.0\src\example\go-ivy\build.xml:85: Could not > create task or type of type: antlib:org.apache.ivy.ant:cachepath. > Ant could not find the task or a class this task relies upon. > This is common and has a number of causes; the usual > solutions are to read the manual pages then download and > install needed JAR files, or fix the build file: > - You have misspelt 'antlib:org.apache.ivy.ant:cachepath'. > Fix: check your spelling. > - The task needs an external JAR file to execute > and this is not found at the right place in the classpath. > Fix: check the documentation for dependencies. > Fix: declare the task. > - The task is an Ant optional task and the JAR file and/or libraries > implementing the functionality were not found at the time you > yourself built your installation of Ant from the Ant sources. > Fix: Look in the ANT_HOME/lib for the 'ant-' JAR corresponding to the > task and make sure it contains more than merely a META-INF/MANIFEST.MF. > If all it contains is the manifest, then rebuild Ant with the needed > libraries present in ${ant.home}/lib/optional/ , or alternatively, > download a pre-built release version from apache.org > - The build file was written for a later version of Ant > Fix: upgrade to at least the latest release version of Ant > - The task is not an Ant core or optional task > and needs to be declared using <taskdef>. > - You are attempting to use a task defined using > <presetdef> or <macrodef> but have spelt wrong or not > defined it at the point of use > Remember that for JAR files to be visible to Ant tasks implemented > in ANT_HOME/lib, the files must be in the same directory or on the > classpath > Please neither file bug reports on this problem, nor email the > Ant mailing lists, until all of these causes have been explored, > as this is not an Ant bug. > Total time: 1 second > {quote} > The reason for the bug is the squid. When request comes and it has a cached > content with gzip content encoding, it return a content with encoded content, > event if no encoding was specified in the request or "Accept-Encoding: > identity" was specified. The below is a sample log that demonstrates the > problem: > {quote} > DEBUG output created by Wget 1.11.4 on cygwin. > --2010-06-02 12:41:55-- > http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar > Resolving repo1.maven.org... 38.97.124.18 > Caching repo1.maven.org => 38.97.124.18 > Connecting to repo1.maven.org|38.97.124.18|:80... connected. > Created socket 4. > Releasing 0x006d9100 (new refcount 1). > ---request begin--- > GET /maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar HTTP/1.0 > User-Agent: Wget/1.11.4 > Accept: */* > Host: repo1.maven.org > Connection: Keep-Alive > ---request end--- > HTTP request sent, awaiting response... > ---response begin--- > HTTP/1.0 200 OK > Server: nginx/0.7.62 > Date: Tue, 01 Jun 2010 13:09:36 GMT > Content-Type: application/java-archive > Last-Modified: Mon, 12 Oct 2009 21:19:02 GMT > Content-Encoding: gzip > X-Cache: MISS from squid_havp_node1 > X-Cache-Lookup: MISS from squid_havp_node1:3128 > Age: 70339 > Content-Length: 834513 > X-Cache: HIT from squid_havp_node1 > X-Cache-Lookup: HIT from squid_havp_node1:3128 > Connection: keep-alive > ---response end--- > HTTP/1.0 200 OK > Server: nginx/0.7.62 > Date: Tue, 01 Jun 2010 13:09:36 GMT > Content-Type: application/java-archive > Last-Modified: Mon, 12 Oct 2009 21:19:02 GMT > Content-Encoding: gzip > X-Cache: MISS from squid_havp_node1 > X-Cache-Lookup: MISS from squid_havp_node1:3128 > Age: 70339 > Content-Length: 834513 > X-Cache: HIT from squid_havp_node1 > X-Cache-Lookup: HIT from squid_havp_node1:3128 > Connection: keep-alive > Registered socket 4 for persistent reuse. > Length: 834513 (815K) [application/java-archive] > Saving to: `ivy-2.1.0.jar' > 0K .......... .......... .......... .......... .......... 6% 8.49M 0s > 50K .......... .......... .......... .......... .......... 12% 12.0M 0s > 100K .......... .......... .......... .......... .......... 18% 7.26M 0s > 150K .......... .......... .......... .......... .......... 24% 9.28M 0s > 200K .......... .......... .......... .......... .......... 30% 5.33M 0s > 250K .......... .......... .......... .......... .......... 36% 9.31M 0s > 300K .......... .......... .......... .......... .......... 42% 5.26M 0s > 350K .......... .......... .......... .......... .......... 49% 9.15M 0s > 400K .......... .......... .......... .......... .......... 55% 7.27M 0s > 450K .......... .......... .......... .......... .......... 61% 7.42M 0s > 500K .......... .......... .......... .......... .......... 67% 9.21M 0s > 550K .......... .......... .......... .......... .......... 73% 6.11M 0s > 600K .......... .......... .......... .......... .......... 79% 9.50M 0s > 650K .......... .......... .......... .......... .......... 85% 8.84M 0s > 700K .......... .......... .......... .......... .......... 92% 9.54M 0s > 750K .......... .......... .......... .......... .......... 98% 6.04M 0s > 800K .......... .... 100% 16.2K=1.0s > 2010-06-02 12:41:56 (795 KB/s) - `ivy-2.1.0.jar' saved [834513/834513] > {quote} > In our environment, a bunch of files from repo1.maven.org are cached in the > squid. So there were a lot of conflicts with content encoding. > The possible workarounds: > 1. Check returned content encoding from http client, if it is not identity, > reissue the request with disabling the caches. > 2. Ungzip files fetched with gzip content encoding. > 3. Support gzip content encoding in common HTTP. > 3. Use local maven proxies -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.