Re: URL resolver ignores port numbers specified in Ivysettings credentials element

[Stefan Bodewig]

Hi Alex

On 2024-03-18, Alex Galvin wrote:

> However, Ant/Ivy errors out in this configuration, and I can see from
> logs that it is not using my credentials to authenticate to the server
> at port 8080. I fixed this issue by editing the host attribute of the
> credentials element to exclude the port number, simply writing
> host="localhost" rather than host="localhost:8080". While this
> behaviour does align with the technical meaning of "hostname" as a
> server located at an IP address (or several), to me it is
> counterintuitive as it implies that Ivy cannot access different URLs
> on the same host unless identical credentials are used for both.

> Is this a bug? If not, is there any way to specify credentials for
> each resolver with better-than-hostname granularity?

I've just had a quick look at the code. Here is how credentials are used
with Apache HttpClient:
https://github.com/apache/ant-ivy/blob/503aafc594b777ffb035a457ed341e31334c23a4/src/java/org/apache/ivy/util/url/HttpClientHandler.java#L381

It is the way you observed it, the port is not taken into account at
all. Neither is the URI scheme so you wouldn't be able to pin
credentials to https only if you wanted to avoid leaking them over
unencrypted connections by accident.

It does take the Basic auth realm into account, though, so this may or
may not help in your case. Not that it would be of any help against
leaking credentials if the attacker can chose the realm.

I don't see any way to make the current version of Ivy use the port.

Cheers

Stefan

Re: Differences in retrieve pattern between Ivy 2.5.0/2.5.1 & 2.5.2?

[Stefan Bodewig]

Hi

glad I could help. I completely agree with you the documentation is
lacking.

TBH you probably by now know more about how Ivy wants to be configured
than me. :-)

Stefan

On 2024-03-13, Lewis John McGibbney wrote:

> Thanks for this guidance Stefan :)
> I was able to get a patch together at https://github.com/apache/nutch/pull/803
> Hopefully this helps others who may be confused as I was.
> Thank you
> lewsmc

> On 2024/03/12 18:57:51 Stefan Bodewig wrote:
>> On 2024-03-11, lewis john mcgibbney wrote:

>>> I am working on upgrading Ivy to latest over in the Apache Nutch project.
>>> The build works just fine with 2.5.0 and 2.5.1 but with 2.5.2 the CI
>>> fails with the following complaint

>>> /home/runner/work/nutch/nutch/src/plugin/build-plugin.xml:234:
>>> impossible to ivy retrieve: java.lang.RuntimeException: problem during
>>> retrieve of org.apache.nutch#lib-htmlunit: java.lang.RuntimeException:
>>> Multiple artifacts of the module
>>> io.netty#netty-transport-native-kqueue;4.1.84.Final are retrieved to
>>> the same file! Update the retrieve pattern to fix this error.

>> Ivy 2.5.2 fixes a bug[1] when dealing with dependencies that have
>> multiple Maven artifacts with different Maven classifiers. Prior to
>> 2.5.2 Ivy would think they'd all be the same and just pick one.

>> io.netty#netty-transport-native-kqueue has several artifacts, at least
>> this is what the repo looks like. I completely fail to understand the
>> POM :-)

>> Your pattern probably needs a [classifier] to make sure two artifacts
>> that differ by Maven classifier also target different file names.

>> Something like

>> pattern="${local-maven2-dir}/[organisation]/[module]/[revision]/[module]-[revision](-[classifier]).[ext]"

>> Stefan

>> [1] https://issues.apache.org/jira/browse/IVY-1642

Re: Differences in retrieve pattern between Ivy 2.5.0/2.5.1 & 2.5.2?

[Stefan Bodewig]

On 2024-03-11, lewis john mcgibbney wrote:

> I am working on upgrading Ivy to latest over in the Apache Nutch project.
> The build works just fine with 2.5.0 and 2.5.1 but with 2.5.2 the CI
> fails with the following complaint

> /home/runner/work/nutch/nutch/src/plugin/build-plugin.xml:234:
> impossible to ivy retrieve: java.lang.RuntimeException: problem during
> retrieve of org.apache.nutch#lib-htmlunit: java.lang.RuntimeException:
> Multiple artifacts of the module
> io.netty#netty-transport-native-kqueue;4.1.84.Final are retrieved to
> the same file! Update the retrieve pattern to fix this error.

Ivy 2.5.2 fixes a bug[1] when dealing with dependencies that have
multiple Maven artifacts with different Maven classifiers. Prior to
2.5.2 Ivy would think they'd all be the same and just pick one.

io.netty#netty-transport-native-kqueue has several artifacts, at least
this is what the repo looks like. I completely fail to understand the
POM :-)

Your pattern probably needs a [classifier] to make sure two artifacts
that differ by Maven classifier also target different file names.

Something like

pattern="${local-maven2-dir}/[organisation]/[module]/[revision]/[module]-[revision](-[classifier]).[ext]"

Stefan

[1] https://issues.apache.org/jira/browse/IVY-1642

[ANN] Apache IvyDE Retired

[Stefan Bodewig]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

The Apache Ant Project Management Commitee has voted[1] to retire its
IvyDE subproject. This means all its resources are removed or made read
only and no further development will be done.

Apache IvyDE let you manage your dependencies declared in an Apache Ivy
ivy.xml in your Java Eclipse projects.

The existing website and documentation will stay but no longer get
updated. The same is true for the source code repositories and the JIRA
issue tracker. All releases of IvyDE remain avaiable from the Apache
release archive[2].

Should a new community grow around IvyDE, the subproject could be
reactivated[3].

We want to thank the people who created or contributed to IvyDE over the
years.

Stefan Bodewig on behalf of the Ant PMC.

[1] https://lists.apache.org/thread/wo32q8s8o8z9m126gz3m533q2fnqq21o

[2] 
https://ant.apache.org/processes.html#Reactivate%20a%20subproject%20or%20component

[3] 
https://ant.apache.org/processes.html#Reactivate%20a%20subproject%20or%20component
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAmVjbhsACgkQohFa4V9ri3LinQCeMiN4U0fZWnhboEdBqfOhjkVR
oYIAni6+c4Fuhj9Ql7DwUFGN7l0fsK5g
=rfPg
-END PGP SIGNATURE-

Re: Future of Ivy and IvyDE

[Stefan Bodewig]

Hi

sorry for my bad timing sending out an email and then being unbale to
answer for days. This is not what I intended.

Let me try to answer what I've seen so far. And I'll try to keep my
personal opinion out this time.

It is pretty obvious Ivy is used today and maybe even loved by
some. This is great for any project.

As is probably true with all volunteer based open source projects people
come and go as their interests and focus change.

In the case of Ivy this unfortunately means it is not maintained well
today and this is not going to change with the current set of Ant
developers. All of us work on Ant in our spare time and we've picked
developing Ant for several different reasons - none of these reasons
seem to apply to Ivy for anybody of us.

It is not my intention to kill Ivy. Not at all. But if maintenance of
Ivy is left with the current set of Ant developers this is what is
likely going to happen, eventually.

It just seemed to be fair to inform Ivy's users about the current
state.

Cheers

Stefan

Future of Ivy and IvyDE

[Stefan Bodewig]

Hi all

before I get to the actual content of this mail:

* I'm cross-posting to three lists but I ask you to keep responses to
  dev@ant only (and join the list if necessary) if you want to respond.

* what I write is my personal opinion and not shared by the PMC as a
  whole. The people on the PMC know I'd be writing a mail like this
  sooner or later, though.

* this is a discussion, not a vote.

phew

I'm not quite sure what I hope to achieve with this email, but I'd like
to share my thoughts - and raise the awareness of an elephant being in
the room.

Over the past year we've had three security vulnerabilities discovered
in Ivy and it took us much too long to get them fixed. The reason for
this is there are no people left around who are familiar with the Ivy
code base. Most of the remaining developers around Ant are not even
users of Ivy - I know I am not and have never been.

When it comes to IvyDE things are probably even worse as nobody of us
uses Eclipse, either. But then again I've not managed to create an
Eclipse update site for the last two Ivy releases so maybe nobody is
using IvyDE anymore anyway.

At least *I* don't see myself digging deeper into the Ivy code base in
order to fix non-critical bugs. And even for the critical ones I feel we
are not doing an adequate job. To me it looks as if Ivy and in
particilar IvyDE are no longer really supported by the Ant project.

TBH I'm not quite sure what to do about this. Even if people stepped up
to maintain Ivy, the rest of the Ant devs would probably be unable to
verify the changes they want to make. At least I certainly am not
willing to review bigger PRs/patches to a code base I don't understand
well.

Personally I believe we should send IvyDE to the Apache Attic
immediately, and this likely should be the destination for Ivy sooner or
later as well. In the case of Ivy we know there are people who depend on
it (hi, Groovy folks) so maybe we should give a date in the future until
which we are providing security bug fixes to give people time to move
off.

There may be the need for a dependency management system inside of Ant,
I'm not sure. If so, then this should be driven by people who feel the
actual need IMO. There may already be alternatives to Ivy I am not aware
of.

Stefan

CVE-2022-46751: Apache Ivy: XML External Entity vulnerability in Apache Ivy

[Stefan Bodewig]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Severity: moderate

Affected versions:

- - Apache Ivy 1.0.0 through 2.5.1

Description:

Improper Restriction of XML External Entity Reference, XML Injection (aka Blind 
XPath Injection) vulnerability in Apache Software Foundation Apache Ivy.This 
issue affects any version of Apache Ivy prior to 2.5.2.

When Apache Ivy prior to 2.5.2 parses XML files - either its own configuration, 
Ivy files or Apache Maven POMs - it will allow downloading external document 
type definitions and expand any entity references contained therein when used.

This can be used to exfiltrate data, access resources only the machine running 
Ivy has access to or disturb the execution of Ivy in different ways.

Starting with Ivy 2.5.2 DTD processing is disabled by default except when 
parsing Maven POMs where the default is to allow DTD processing but only to 
include a DTD snippet shipping with Ivy that is needed to deal with existing 
Maven POMs that are not valid XML files but are nevertheless accepted by Maven. 
Access can be be made more lenient via newly introduced system properties where 
needed.

Users of Ivy prior to version 2.5.2 can use Java system properties to restrict 
processing of external DTDs, see the section about "JAXP Properties for 
External Access restrictions" inside Oracle's "Java API for XML Processing 
(JAXP) Security Guide".

Credit:

CC Bomber, Kitri BoB (finder)
Jenkins Security Team (reporter)

References:

https://docs.oracle.com/en/java/javase/13/security/java-api-xml-processing-jaxp-security-guide.html#GUID-94ABC0EE-9DC8-44F0-84AD-47ADD5340477
https://gitbox.apache.org/repos/asf?p=ant-ivy.git;a=commit;h=2be17bc18b0e1d4123007d579e43ba1a4b6fab3d
https://lists.apache.org/thread/9gcz4xrsn8c7o9gb377xfzvkb8jltffr
https://ant.apache.org/
https://www.cve.org/CVERecord?id=CVE-2022-46751

Timeline:

2022-11-30: reported to the ASF security team
2023-08-20: made public
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAmTiYP0ACgkQohFa4V9ri3J5AACfQZlPRXWN0pU73Rm5o24EbBE+
cAoAn3XtnwhZYn395m1cw+9tadrX0oWv
=d+kA
-END PGP SIGNATURE-

[ANN] Apache Ivy 2.5.2 Released

[Stefan Bodewig]

The Apache Ant Team is pleased to announce the release of Apache Ivy
2.5.2.

Apache Ivy is a dependency manager focusing on flexibility and
simplicity with strong integration into the Apache Ant build tool.

Ivy 2.5.2 is bugfix release and addresses an XML external entity
injection vulnerability, see the upcoming CVE announcement or
https://ant.apache.org/ivy/security.html for details.

Source and binary distributions are available for download from the
Apache Ivy download site:

https://ant.apache.org/ivy/download.cgi

When downloading, please verify signatures using the KEYS file available
at the above location when downloading the release.

Changes in 2.5.2 include:
=

- FIX: ivy:retrieve could fail because of a `NullPointerException` 
(jira:IVY-1641[])
- FIX: reading POMs may loose dependencies when multiple Maven
  dependencies only differ in `classifier` (jira:IVY-1642[])
- IMPROVEMENT: Upgrade Apache HttpClient to 4.5.13 (jira:IVY-1644[])
- FIX: CVE-2022-46751: Apache Ivy Is Vulnerable to XML External Entity 
Injections

For complete information on Ivy, including instructions on how to submit
bug reports, patches, or suggestions for improvement, see the Apache Ivy
website:

https://ant.apache.org/ivy/

Stefan Bodewig, on behalf of the Apache Ant community

Re: Problem with resolving dependencies for saxon-HE on Maven repo

[Stefan Bodewig]

On 2023-04-26, LINUS FERNANDES wrote:

> Thanks, Stefan.

> I think that was caused by me trying the Ant way to install Ivy as

> https://ant.apache.org/ivy/history/2.4.0/install.html

> That's the one that shows up first when I Google it.

I don't think we can influence Google's decisions ...

Ivy 2.4.0 has been released 2014 so its documentation applies to a much
older version of Ant.

Please see https://ant.apache.org/ivy/history/2.5.1/ant.html and use the
"Ant 1.6.0 or superior" approach. The apprach that uses multiple
taskdefs is only required on a version of Ant that doesn't know th
concept of antlibs which has been introduced with Ant 1.6.0 in, errm,
2003. Ivy's docs seem to be quite conservative :-)

Please take anything Ivy-specific I say with a big grain of salt. I
happen to be an Ant developer who fixed a bug in Ivy but am far from
being familiar with Ivy itself.

Stefan

Re: Problem with resolving dependencies for saxon-HE on Maven repo

[Stefan Bodewig]

[you ran a different command than I asked for, this way we still don't
see which version of Ivy you use ;-)]

>  init-ivy:   parsing
> buildfile jar:file:/root/.ant/lib/ivy.jar!/org/apache/ivy/ant/antlib.xml
> with URI = jar:file:/root/.ant/lib/ivy.jar!/org/apache/ivy/ant/antlib.xml
> from a zip file

I believe this one is the culprit. There is an ivy.jar in your
~/.ant/lib directory and I'm willing to bet it contains an older version
of Ivy.

Adding -nouserlib to the Ant command line should help avoiding this.

The documentation for the pathid attribute in
https://ant.apache.org/ivy/history/2.5.1/use/retrieve.html (the current
docs) says it has been added with Ivy 2.3 so most likely the version you
actually use is older than 2.3.

Stefan

Re: Problem with resolving dependencies for saxon-HE on Maven repo

[Stefan Bodewig]

On 2023-04-26, LINUS FERNANDES wrote:

> I have the same error on Arch Linux on Termux:

> echo $ANT_HOME
> /usr/share/ant

> root07:40IvySample$ ls /usr/share/ant/lib
> ant-antlr.jar
> ant-apache-bcel.jar
> ant-apache-bsf.jar
> ant-apache-log4j.jar
> ant-apache-oro.jar
> ant-apache-regexp.jar
> ant-apache-resolver.jar
> ant-apache-xalan2.jar
> ant-commons-logging.jar
> ant-commons-net.jar
> ant-contrib-1.0b3.jar
> ant-contrib-20020829.jar
> ant-imageio.jar
> ant-jai.jar
> ant.jar
> ant-javamail.jar
> ant-jdepend.jar
> ant-jmf.jar
> ant-jsch.jar
> ant-junit4.jar
> ant-junit.jar
> ant-junitlauncher.jar
> ant-launcher.jar
> ant-netrexx.jar
> ant-swing.jar
> ant-testutil.jar
> ant-xz.jar
> apiguardian-api-1.1.2.jar
> commons-net-3.9.0.jar
> images
> ivy-2.5.1.jar
> junit-platform-commons-1.9.2.jar
> junit-platform-engine-1.9.2.jar
> junit-platform-launcher-1.9.2.jar
> opentest4j-1.2.0.jar
> root07:40IvySample$ vim build.xml
> root07:42IvySample$ ant
> Buildfile: /root/IvySample/build.xml

> init-ivy:

> retrieve:

> BUILD FAILED
> /root/IvySample/build.xml:12: ivy:retrieve doesn't support the "pathid"
> attribute

The only way I can explain this is there is another jar that contains an
older version of Ivy on your classpath and this one is loaded first.

As you can see in my output, Ivy announces its version number

>> retrieve:
>> [ivy:retrieve] :: Apache Ivy 2.5.1 - 20221101102211 ::
>> https://ant.apache.org/ivy/ ::

what does it say in your case when you remove the pathid attribute?

Stefan

Re: Problem with resolving dependencies for saxon-HE on Maven repo

[Stefan Bodewig]

On 2023-04-26, LINUS FERNANDES wrote:

> No, it doesn't . Ivy 2.5.1 is complaining that the task does not have the
> attribute pathid or pathId either.

I don't see any error here

$ cat ivy.xml

  
  

  


$ cat ivysettings.xml 

  
  
  

  


$ cat build.xml 

  

  


  
  


  


$ ant -Divy.jar.dir=$HOME/.m2/repository/org/apache/ivy/ivy/2.5.1/
Buildfile: /tmp/build.xml

init-ivy:

retrieve:
[ivy:retrieve] :: Apache Ivy 2.5.1 - 20221101102211 :: 
https://ant.apache.org/ivy/ ::
[ivy:retrieve] :: loading settings :: file = /tmp/ivysettings.xml
[ivy:retrieve] :: resolving dependencies :: org.example.foo#bar;1.0.0
[ivy:retrieve]  confs: [default]
[ivy:retrieve]  found org.apache.ivy#ivy;2.5.1 in public
[ivy:retrieve]  found junit#junit;4.12 in public
[ivy:retrieve]  found org.hamcrest#hamcrest-core;1.3 in public
[ivy:retrieve]  found org.hamcrest#hamcrest-library;1.3 in public
[ivy:retrieve]  found org.apache.ant#ant-testutil;1.9.14 in public
[ivy:retrieve]  found org.apache.ant#ant-launcher;1.9.14 in public
[ivy:retrieve]  found org.apache.ant#ant-junit;1.9.14 in public
[ivy:retrieve]  found org.apache.ant#ant-junit4;1.9.14 in public
[ivy:retrieve]  found ant-contrib#ant-contrib;1.0b3 in public
[ivy:retrieve]  found xmlunit#xmlunit;1.6 in public
[ivy:retrieve]  found org.apache.ant#ant;1.9.14 in public
[ivy:retrieve]  found org.apache.httpcomponents#httpclient;4.5.10 in public
[ivy:retrieve]  found org.apache.httpcomponents#httpcore;4.4.12 in public
[ivy:retrieve]  found commons-logging#commons-logging;1.2 in public
[ivy:retrieve]  found commons-codec#commons-codec;1.11 in public
[ivy:retrieve]  found oro#oro;2.0.8 in public
[ivy:retrieve]  found org.apache.commons#commons-vfs2;2.2 in public
[ivy:retrieve]  found com.jcraft#jsch;0.1.55 in public
[ivy:retrieve]  found com.jcraft#jsch.agentproxy;0.0.9 in public
[ivy:retrieve]  found com.jcraft#jsch.agentproxy.connector-factory;0.0.9 in 
public
[ivy:retrieve]  found com.jcraft#jsch.agentproxy.core;0.0.9 in public
[ivy:retrieve]  found com.jcraft#jsch.agentproxy.usocket-jna;0.0.9 in public
[ivy:retrieve]  found net.java.dev.jna#jna;4.1.0 in public
[ivy:retrieve]  found net.java.dev.jna#jna-platform;4.1.0 in public
[ivy:retrieve]  found com.jcraft#jsch.agentproxy.usocket-nc;0.0.9 in public
[ivy:retrieve]  found com.jcraft#jsch.agentproxy.sshagent;0.0.9 in public
[ivy:retrieve]  found com.jcraft#jsch.agentproxy.pageant;0.0.9 in public
[ivy:retrieve]  found com.jcraft#jsch.agentproxy.jsch;0.0.9 in public
[ivy:retrieve]  found org.bouncycastle#bcpg-jdk15on;1.64 in public
[ivy:retrieve]  found org.bouncycastle#bcprov-jdk15on;1.64 in public
[ivy:retrieve] :: resolution report :: resolve 290ms :: artifacts dl 12ms
[ivy:retrieve]  :: evicted modules:
[ivy:retrieve]  com.jcraft#jsch;0.1.49 by [com.jcraft#jsch;0.1.55] in [default]
-
|  |modules||   artifacts   |
|   conf   | number| search|dwnlded|evicted|| number|dwnlded|
-
|  default |   31  |   0   |   0   |   1   ||   31  |   0   |
-
[ivy:retrieve] :: retrieving :: org.example.foo#bar
[ivy:retrieve]  confs: [default]
[ivy:retrieve]  0 artifacts copied, 31 already retrieved (0kB/5ms)

BUILD SUCCESSFUL
Total time: 0 seconds

Re: Problem with resolving dependencies for saxon-HE on Maven repo

[Stefan Bodewig]

On 2023-04-26, LINUS FERNANDES wrote:

> Could you also let me know in which release of ivy was pathid removed as an
> attribute in the retrieve task and why? Documentation for that is hard to
> come by.

I must admit that I'm not really that familiar with Ivy myself, I just
jumped in to fix a bug.

AFAICT the IvyRetrieve Java class still contains a setPathId(String) [1]
method so the retrieve task should (still) have a pathId attribute in
2.5.1.

Stefan

[1] 
https://github.com/apache/ant-ivy/blob/master/src/java/org/apache/ivy/ant/IvyRetrieve.java#L77

Re: Problem with resolving dependencies for saxon-HE on Maven repo

[Stefan Bodewig]

Stefan Bodewig  writes:

> I believe there is a bug in PomModuleDescriptoBuilder that assumes it is
> just looking at the same dependency twice as it doesn't take the
> classifier into account.
>
> I've opened https://issues.apache.org/jira/browse/IVY-1642

should be fixed with
https://github.com/apache/ant-ivy/commit/5f11f05fe38b2fc3779d5bec6287505798bc701c
, will be fixed in Ivy 2.5.2 once it is released.

Stefan

Re: Problem with resolving dependencies for saxon-HE on Maven repo

[Stefan Bodewig]

Hi Linus

On 2023-04-13, LINUS FERNANDES wrote:

> There is a problem downloading the dependencies for Saxon-HE from Maven
> using Ivy.
> The xmlresolver-5.1.1.jar does not download but it's classified jar
> xmlresolver-data-5.1.1.jar does.  Both jars are listed as dependencies in
> the pom for Saxon-HE.

> You can read about it in more detail on the following thread:

> https://github.com/checkstyle/checkstyle/issues/12969

> You can execute an ant build for the same at :

> https://github.com/Fernal73/CheckStyleSample/

Yes, I can see what you describe when running "ant resolve" on your
example build file - this is what the ivy.xml created from Saxon-HE's
POM says:





so there is only one artifact in here. If I add a second artifact
manually (without classifier) the "normal" jar is downloaded and copied
from the cache as well.

I believe there is a bug in PomModuleDescriptoBuilder that assumes it is
just looking at the same dependency twice as it doesn't take the
classifier into account.

I've opened https://issues.apache.org/jira/browse/IVY-1642

Thank you

  Stefan

Re: NullPointerException while resolving dependency

[Stefan Bodewig]

Hi

Yes, I think it would be possible to enable preemptive authentication
but it comes at some security risk, so should never be enabled by
default. Likely we'd want to enable it at the credential level and
enforce the realm to be not set - as you can't know which realm the
server associates with the URI before it responds with a 401.

We even could create a Basic-Auth Authorization header ourselves for the
java.net.URLConnection based code.

As far as writing your own resolver goes. I must admit I'm not an expert
in Ivy's code base. I'm not even sure anybody is nowadays. The way
plugins interact is pretty
complex.

https://github.com/apache/ant-ivy/blob/master/src/java/org/apache/ivy/plugins/resolver/URLResolver.java
is the resolver you are using right now and almost everything happens
inside of the repository
https://github.com/apache/ant-ivy/blob/master/src/java/org/apache/ivy/plugins/repository/url/URLRepository.java

Rather than writing a repository of your own you are probably better off
writing an URLHandler implementation which performs preemptive
authentication (see
https://github.com/apache/ant-ivy/blob/master/src/java/org/apache/ivy/util/url/TimeoutConstrainedURLHandler.java
) and set that as default.

Ivy itself creates a URLHandlerDispatcher configured to use
URLHandlerRegistry.getHttp for http and https as default in
URLHandlerRegistry - so what might work is to read the
URLHandlerDispatcher that is the default once Ivy is configured and
replace the handler for https with your own. But I must admit I'm not
sure what side effects this may have. There still are other places that
invoke URLHandlerRegistry.getHttp() directly, for example.

Come to think of it, it is probably easier to add preemptive
authentication support in HttpClientHandler and BasicURLHandler.

Stefan

On 2023-03-21, Kamran Manzoor wrote:

> Hi Stefan,

> Thanks a lot for all your help here. I really appreciate it :).

> It was indeed the issue that the client doesn't do preemptive
> authentication which I wasn't aware of. I made the Beeceptor stub returns
> 401 with the www-authenticate header and then it does make a follow-up call
> with credentials.

> However, I have to download dependency using Bitbucket. Bitbucket returns
> *404* status code if the credentials are not being sent. This means that
> the client does *not* send the followup request with credentials and hence
> ivy fails to resolve the Bitbucket dependency with *NotFound* error. I
> believe the 404 response code is a layer of security from the Bitbucket
> side in order to avoid things like bruteforce/account locking. I
> believe this would be the case for other systems as well like Github.
> Therefore, I was thinking maybe you could consider putting a preemptive
> authentication option in apache-ivy?. Http-client does provide this option:
> https://hc.apache.org/httpcomponents-client-4.5.x/current/tutorial/html/authentication.html#d5e717

> For now, I may look into creating a custom resolver for resolving
> dependency from Bitbucket. If you have any pointers to any example of
> custom resolvers, then please do share otherwise i may ask ChatGPT to help
> here :D

> Thanks again for all your help and time.

> Kind Regards,
> Kamran Manzoor

> On 2023/03/20 21:32:12 Stefan Bodewig wrote:
>> On 2023-03-20, Stefan Bodewig wrote:

>>> On 2023-03-20, Stefan Bodewig wrote:

>>>> Hi

>>>> On 2023-03-20, Kamran Manzoor wrote:

>>>>> Thanks a lot for looking into this. I thought the issue was request
> related
>>>>> but it is in response handling. It means it does trigger the request
> which
>>>>> I have verified and it seems like the auth header is missing :(. I
> have
>>>>> tried with default URL handler and even with older ivy version 2.4.0
> with
>>>>> older commons-httpclient. Nothing seems to send the auth header with
>>>>> credentials :(. You may simply run this example i.e., *ant -verbose
>>>>> retrieve* and see the request captured here:
>>>>> *https://beeceptor.com/console/ivytry
>>>>> <https://beeceptor.com/console/ivytry> *

>>>> I believe this is because your service there doesn't require any
>>>> authentcation at all. Apache HttpClient doesn't do preemptive
>>>> authentication, it only sends credentials when really necessary. This
> is
>>>> understandable from a security perspective. Why send credentials to
>>>> somebody who never wanted them in the first place.

>>>> So it performs an HTTP GET without any credentials and if this request
>>>> succeeds, that's it. And this happens with your beeceptor example.

>>>> You probably need to make beeceptor return 401 on the first request
>>>&

Re: NullPointerException while resolving dependency

[Stefan Bodewig]

On 2023-03-20, Stefan Bodewig wrote:

> On 2023-03-20, Stefan Bodewig wrote:

>> Hi

>> On 2023-03-20, Kamran Manzoor wrote:

>>> Thanks a lot for looking into this. I thought the issue was request related
>>> but it is in response handling. It means it does trigger the request which
>>> I have verified and it seems like the auth header is missing :(. I have
>>> tried with default URL handler and even with older ivy version 2.4.0 with
>>> older commons-httpclient. Nothing seems to send the auth header with
>>> credentials :(. You may simply run this example i.e., *ant -verbose
>>> retrieve* and see the request captured here:
>>> *https://beeceptor.com/console/ivytry
>>> <https://beeceptor.com/console/ivytry> *

>> I believe this is because your service there doesn't require any
>> authentcation at all. Apache HttpClient doesn't do preemptive
>> authentication, it only sends credentials when really necessary. This is
>> understandable from a security perspective. Why send credentials to
>> somebody who never wanted them in the first place.

>> So it performs an HTTP GET without any credentials and if this request
>> succeeds, that's it. And this happens with your beeceptor example.

>> You probably need to make beeceptor return 401 on the first request
>> answering with

>> WWW-Authenticate: Basic realm=Basic

>> or the configured credentials are not going to be ever used.

> Taking a closer look at Ivy's code, your credentials' configuration must
> exactly match what is returned by the server. So you should probably
> perform an curl unauthenticated request (or whatever HTTP client you
> prefer) and examine the response headers of the 401 response to know
> which realm and host to set. Leaving off the optional realm in
>  is only going to match WWW-Authenticate headers without
> any "realm" at all.

One final thing, I just found
https://developer.atlassian.com/server/bitbucket/how-tos/example-basic-authentication/

Take a look at the last paragraph. Iyv is a "http client software [that]
expects to receive an authentication challenge before it will send an
authorization header" - and currently there is no way to enforce
authentication.

At least this is true when using Apache HttpClient. I haven't checked
whether java.net.URLConnection ever consults the configured
Authenticator - but wouldn't be surprised if it doesn't do so by default
either.

Stefan

Re: NullPointerException while resolving dependency

[Stefan Bodewig]

On 2023-03-20, Stefan Bodewig wrote:

> Hi

> On 2023-03-20, Kamran Manzoor wrote:

>> Thanks a lot for looking into this. I thought the issue was request related
>> but it is in response handling. It means it does trigger the request which
>> I have verified and it seems like the auth header is missing :(. I have
>> tried with default URL handler and even with older ivy version 2.4.0 with
>> older commons-httpclient. Nothing seems to send the auth header with
>> credentials :(. You may simply run this example i.e., *ant -verbose
>> retrieve* and see the request captured here:
>> *https://beeceptor.com/console/ivytry
>> <https://beeceptor.com/console/ivytry> *

> I believe this is because your service there doesn't require any
> authentcation at all. Apache HttpClient doesn't do preemptive
> authentication, it only sends credentials when really necessary. This is
> understandable from a security perspective. Why send credentials to
> somebody who never wanted them in the first place.

> So it performs an HTTP GET without any credentials and if this request
> succeeds, that's it. And this happens with your beeceptor example.

> You probably need to make beeceptor return 401 on the first request
> answering with

> WWW-Authenticate: Basic realm=Basic

> or the configured credentials are not going to be ever used.

Taking a closer look at Ivy's code, your credentials' configuration must
exactly match what is returned by the server. So you should probably
perform an curl unauthenticated request (or whatever HTTP client you
prefer) and examine the response headers of the 401 response to know
which realm and host to set. Leaving off the optional realm in
 is only going to match WWW-Authenticate headers without
any "realm" at all.

Stefan

Re: NullPointerException while resolving dependency

[Stefan Bodewig]

Hi

On 2023-03-20, Kamran Manzoor wrote:

> Thanks a lot for looking into this. I thought the issue was request related
> but it is in response handling. It means it does trigger the request which
> I have verified and it seems like the auth header is missing :(. I have
> tried with default URL handler and even with older ivy version 2.4.0 with
> older commons-httpclient. Nothing seems to send the auth header with
> credentials :(. You may simply run this example i.e., *ant -verbose
> retrieve* and see the request captured here:
> *https://beeceptor.com/console/ivytry
>  *

I believe this is because your service there doesn't require any
authentcation at all. Apache HttpClient doesn't do preemptive
authentication, it only sends credentials when really necessary. This is
understandable from a security perspective. Why send credentials to
somebody who never wanted them in the first place.

So it performs an HTTP GET without any credentials and if this request
succeeds, that's it. And this happens with your beeceptor example.

You probably need to make beeceptor return 401 on the first request
answering with

WWW-Authenticate: Basic realm=Basic

or the configured credentials are not going to be ever used.

Stefan

Re: NullPointerException while resolving dependency

[Stefan Bodewig]

On 2023-03-17, Stefan Bodewig wrote:

> On 2023-03-17, Stefan Bodewig wrote:

>> and will open a JIRA ticket attaching oyur files when using Apache
>> HttpClient.

> https://issues.apache.org/jira/browse/IVY-1641

should be fixed by
https://github.com/apache/ant-ivy/commit/c63ce79f52133857d1146cd2335d51178714effd

Stefan

Re: NullPointerException while resolving dependency

[Stefan Bodewig]

On 2023-03-17, Stefan Bodewig wrote:

> and will open a JIRA ticket attaching oyur files when using Apache
> HttpClient.

https://issues.apache.org/jira/browse/IVY-1641

Stefan

Re: NullPointerException while resolving dependency

[Stefan Bodewig]

On 2023-03-16, Kamran Manzoor wrote:

> I am new to Apache Ivy and I am using it to manage dependency in one of my
> ant projects. All I want is to retrieve a jar from a custom URL (Bitbucket)
> which requires Basic Authentication. However, I am getting
> NullPointerException when I am trying to retrieve the jar with version
> *2.5.1*.

> *java.lang.NullPointerException at
> org.apache.ivy.util.url.HttpClientHandler.getURLInfo(HttpClientHandler.java:243)*

> I don't get NPE with *2.4.0* version of ivy but it does not send
> credentials in the header. Any pointers/help will be greatly appreciated. I
> have attached the simple example files here along with log files of both
> the versions to reproduce the issue. Ant version is *1.10.7* for me.

I can reproduce the problem you see (and will open a JIRA ticket
attaching oyur files) when using Apache HttpClient. Everything works as
expected when not using Apache HttpClient but the "normal"
java.net.URLConnection based code path. At least I dodn't get any
exception, I haven't checked whether the credentials are included. Maybe
you can use this as a workaround until we get the bug fixed and a
patched release out.

Stefan

Re: Download by using a proxy

[Stefan Bodewig]

On Tue, 17 Jun 2008, [EMAIL PROTECTED] wrote:

 I'm trying to run the very first ivy tutorial described at [1]. When
 I execute ant with the provided build.xml, I'm getting http error
 407 (proxy authentication required).

Do you know what sort of Proxy you are using?

I've had serious trouble getting past a MS ISA server with NTLM
authentication using the built-in proxy support if the JDK I finally
gave in and used commons-httpclient instead of Java's URL class.

I realize this doesn't help you with the Ivy case, I'm just trying to
collect some data.

Can you use Ant's get with these ANT_OPTS or using Ant's setprox
task?

Stefan

Re: Download by using a proxy

[Stefan Bodewig]

On Tue, 17 Jun 2008, [EMAIL PROTECTED] wrote:

 So... If I'm not wrong, you should update you build.xml by adding
 setproxy and set the value of the task attributes with the value of
 properties given as parameters...

setproxy does a little bit more than just setting the system
properties you set via ANT_OPTS, it also registers a
java.net.Authenticator - the later seems to be what your proxy
required for Java to work.

Stefan

Re: Download by using a proxy

[Stefan Bodewig]

On Tue, 17 Jun 2008, Maarten Coene [EMAIL PROTECTED] wrote:

 Ivy should work without the setproxy task,

Maybe.

But Java's built-in proxy handling is broken and there will always be
cases where Ivy won't work and it will not be Ivy's fault.

Stefan