Re: NullPointerException while resolving dependency

[Stefan Bodewig]

Hi

Yes, I think it would be possible to enable preemptive authentication
but it comes at some security risk, so should never be enabled by
default. Likely we'd want to enable it at the credential level and
enforce the realm to be not set - as you can't know which realm the
server associates with the URI before it responds with a 401.

We even could create a Basic-Auth Authorization header ourselves for the
java.net.URLConnection based code.

As far as writing your own resolver goes. I must admit I'm not an expert
in Ivy's code base. I'm not even sure anybody is nowadays. The way
plugins interact is pretty
complex.

https://github.com/apache/ant-ivy/blob/master/src/java/org/apache/ivy/plugins/resolver/URLResolver.java
is the resolver you are using right now and almost everything happens
inside of the repository
https://github.com/apache/ant-ivy/blob/master/src/java/org/apache/ivy/plugins/repository/url/URLRepository.java

Rather than writing a repository of your own you are probably better off
writing an URLHandler implementation which performs preemptive
authentication (see
https://github.com/apache/ant-ivy/blob/master/src/java/org/apache/ivy/util/url/TimeoutConstrainedURLHandler.java
) and set that as default.

Ivy itself creates a URLHandlerDispatcher configured to use
URLHandlerRegistry.getHttp for http and https as default in
URLHandlerRegistry - so what might work is to read the
URLHandlerDispatcher that is the default once Ivy is configured and
replace the handler for https with your own. But I must admit I'm not
sure what side effects this may have. There still are other places that
invoke URLHandlerRegistry.getHttp() directly, for example.

Come to think of it, it is probably easier to add preemptive
authentication support in HttpClientHandler and BasicURLHandler.

Stefan

On 2023-03-21, Kamran Manzoor wrote:

> Hi Stefan,

> Thanks a lot for all your help here. I really appreciate it :).

> It was indeed the issue that the client doesn't do preemptive
> authentication which I wasn't aware of. I made the Beeceptor stub returns
> 401 with the www-authenticate header and then it does make a follow-up call
> with credentials.

> However, I have to download dependency using Bitbucket. Bitbucket returns
> *404* status code if the credentials are not being sent. This means that
> the client does *not* send the followup request with credentials and hence
> ivy fails to resolve the Bitbucket dependency with *NotFound* error. I
> believe the 404 response code is a layer of security from the Bitbucket
> side in order to avoid things like bruteforce/account locking. I
> believe this would be the case for other systems as well like Github.
> Therefore, I was thinking maybe you could consider putting a preemptive
> authentication option in apache-ivy?. Http-client does provide this option:
> https://hc.apache.org/httpcomponents-client-4.5.x/current/tutorial/html/authentication.html#d5e717

> For now, I may look into creating a custom resolver for resolving
> dependency from Bitbucket. If you have any pointers to any example of
> custom resolvers, then please do share otherwise i may ask ChatGPT to help
> here :D

> Thanks again for all your help and time.

> Kind Regards,
> Kamran Manzoor

> On 2023/03/20 21:32:12 Stefan Bodewig wrote:
>> On 2023-03-20, Stefan Bodewig wrote:

>>> On 2023-03-20, Stefan Bodewig wrote:

 Hi

 On 2023-03-20, Kamran Manzoor wrote:

> Thanks a lot for looking into this. I thought the issue was request
> related
> but it is in response handling. It means it does trigger the request
> which
> I have verified and it seems like the auth header is missing :(. I
> have
> tried with default URL handler and even with older ivy version 2.4.0
> with
> older commons-httpclient. Nothing seems to send the auth header with
> credentials :(. You may simply run this example i.e., *ant -verbose
> retrieve* and see the request captured here:
> *https://beeceptor.com/console/ivytry
>  *

 I believe this is because your service there doesn't require any
 authentcation at all. Apache HttpClient doesn't do preemptive
 authentication, it only sends credentials when really necessary. This
> is
 understandable from a security perspective. Why send credentials to
 somebody who never wanted them in the first place.

 So it performs an HTTP GET without any credentials and if this request
 succeeds, that's it. And this happens with your beeceptor example.

 You probably need to make beeceptor return 401 on the first request
 answering with

 WWW-Authenticate: Basic realm=Basic

 or the configured credentials are not going to be ever used.

>>> Taking a closer look at Ivy's code, your credentials' configuration must
>>> exactly match what is returned by the server. So you should probably
>>> perform an curl unauthenticated request (or whatever HTTP client you
>>> prefer) and examine the 

RE: Re: NullPointerException while resolving dependency

[Kamran Manzoor]

Hi Stefan,

Thanks a lot for all your help here. I really appreciate it :).

It was indeed the issue that the client doesn't do preemptive
authentication which I wasn't aware of. I made the Beeceptor stub returns
401 with the www-authenticate header and then it does make a follow-up call
with credentials.

However, I have to download dependency using Bitbucket. Bitbucket returns
*404* status code if the credentials are not being sent. This means that
the client does *not* send the followup request with credentials and hence
ivy fails to resolve the Bitbucket dependency with *NotFound* error. I
believe the 404 response code is a layer of security from the Bitbucket
side in order to avoid things like bruteforce/account locking. I
believe this would be the case for other systems as well like Github.
Therefore, I was thinking maybe you could consider putting a preemptive
authentication option in apache-ivy?. Http-client does provide this option:
https://hc.apache.org/httpcomponents-client-4.5.x/current/tutorial/html/authentication.html#d5e717

For now, I may look into creating a custom resolver for resolving
dependency from Bitbucket. If you have any pointers to any example of
custom resolvers, then please do share otherwise i may ask ChatGPT to help
here :D

Thanks again for all your help and time.

Kind Regards,
Kamran Manzoor

On 2023/03/20 21:32:12 Stefan Bodewig wrote:
> On 2023-03-20, Stefan Bodewig wrote:
>
> > On 2023-03-20, Stefan Bodewig wrote:
>
> >> Hi
>
> >> On 2023-03-20, Kamran Manzoor wrote:
>
> >>> Thanks a lot for looking into this. I thought the issue was request
related
> >>> but it is in response handling. It means it does trigger the request
which
> >>> I have verified and it seems like the auth header is missing :(. I
have
> >>> tried with default URL handler and even with older ivy version 2.4.0
with
> >>> older commons-httpclient. Nothing seems to send the auth header with
> >>> credentials :(. You may simply run this example i.e., *ant -verbose
> >>> retrieve* and see the request captured here:
> >>> *https://beeceptor.com/console/ivytry
> >>>  *
>
> >> I believe this is because your service there doesn't require any
> >> authentcation at all. Apache HttpClient doesn't do preemptive
> >> authentication, it only sends credentials when really necessary. This
is
> >> understandable from a security perspective. Why send credentials to
> >> somebody who never wanted them in the first place.
>
> >> So it performs an HTTP GET without any credentials and if this request
> >> succeeds, that's it. And this happens with your beeceptor example.
>
> >> You probably need to make beeceptor return 401 on the first request
> >> answering with
>
> >> WWW-Authenticate: Basic realm=Basic
>
> >> or the configured credentials are not going to be ever used.
>
> > Taking a closer look at Ivy's code, your credentials' configuration must
> > exactly match what is returned by the server. So you should probably
> > perform an curl unauthenticated request (or whatever HTTP client you
> > prefer) and examine the response headers of the 401 response to know
> > which realm and host to set. Leaving off the optional realm in
> >  is only going to match WWW-Authenticate headers without
> > any "realm" at all.
>
> One final thing, I just found
>
https://developer.atlassian.com/server/bitbucket/how-tos/example-basic-authentication/
>
> Take a look at the last paragraph. Iyv is a "http client software [that]
> expects to receive an authentication challenge before it will send an
> authorization header" - and currently there is no way to enforce
> authentication.
>
> At least this is true when using Apache HttpClient. I haven't checked
> whether java.net.URLConnection ever consults the configured
> Authenticator - but wouldn't be surprised if it doesn't do so by default
> either.
>
> Stefan
>

Re: NullPointerException while resolving dependency

[Stefan Bodewig]

On 2023-03-20, Stefan Bodewig wrote:

> On 2023-03-20, Stefan Bodewig wrote:

>> Hi

>> On 2023-03-20, Kamran Manzoor wrote:

>>> Thanks a lot for looking into this. I thought the issue was request related
>>> but it is in response handling. It means it does trigger the request which
>>> I have verified and it seems like the auth header is missing :(. I have
>>> tried with default URL handler and even with older ivy version 2.4.0 with
>>> older commons-httpclient. Nothing seems to send the auth header with
>>> credentials :(. You may simply run this example i.e., *ant -verbose
>>> retrieve* and see the request captured here:
>>> *https://beeceptor.com/console/ivytry
>>>  *

>> I believe this is because your service there doesn't require any
>> authentcation at all. Apache HttpClient doesn't do preemptive
>> authentication, it only sends credentials when really necessary. This is
>> understandable from a security perspective. Why send credentials to
>> somebody who never wanted them in the first place.

>> So it performs an HTTP GET without any credentials and if this request
>> succeeds, that's it. And this happens with your beeceptor example.

>> You probably need to make beeceptor return 401 on the first request
>> answering with

>> WWW-Authenticate: Basic realm=Basic

>> or the configured credentials are not going to be ever used.

> Taking a closer look at Ivy's code, your credentials' configuration must
> exactly match what is returned by the server. So you should probably
> perform an curl unauthenticated request (or whatever HTTP client you
> prefer) and examine the response headers of the 401 response to know
> which realm and host to set. Leaving off the optional realm in
>  is only going to match WWW-Authenticate headers without
> any "realm" at all.

One final thing, I just found
https://developer.atlassian.com/server/bitbucket/how-tos/example-basic-authentication/

Take a look at the last paragraph. Iyv is a "http client software [that]
expects to receive an authentication challenge before it will send an
authorization header" - and currently there is no way to enforce
authentication.

At least this is true when using Apache HttpClient. I haven't checked
whether java.net.URLConnection ever consults the configured
Authenticator - but wouldn't be surprised if it doesn't do so by default
either.

Stefan

Re: NullPointerException while resolving dependency

[Stefan Bodewig]

On 2023-03-20, Stefan Bodewig wrote:

> Hi

> On 2023-03-20, Kamran Manzoor wrote:

>> Thanks a lot for looking into this. I thought the issue was request related
>> but it is in response handling. It means it does trigger the request which
>> I have verified and it seems like the auth header is missing :(. I have
>> tried with default URL handler and even with older ivy version 2.4.0 with
>> older commons-httpclient. Nothing seems to send the auth header with
>> credentials :(. You may simply run this example i.e., *ant -verbose
>> retrieve* and see the request captured here:
>> *https://beeceptor.com/console/ivytry
>>  *

> I believe this is because your service there doesn't require any
> authentcation at all. Apache HttpClient doesn't do preemptive
> authentication, it only sends credentials when really necessary. This is
> understandable from a security perspective. Why send credentials to
> somebody who never wanted them in the first place.

> So it performs an HTTP GET without any credentials and if this request
> succeeds, that's it. And this happens with your beeceptor example.

> You probably need to make beeceptor return 401 on the first request
> answering with

> WWW-Authenticate: Basic realm=Basic

> or the configured credentials are not going to be ever used.

Taking a closer look at Ivy's code, your credentials' configuration must
exactly match what is returned by the server. So you should probably
perform an curl unauthenticated request (or whatever HTTP client you
prefer) and examine the response headers of the 401 response to know
which realm and host to set. Leaving off the optional realm in
 is only going to match WWW-Authenticate headers without
any "realm" at all.

Stefan

Re: NullPointerException while resolving dependency

[Stefan Bodewig]

Hi

On 2023-03-20, Kamran Manzoor wrote:

> Thanks a lot for looking into this. I thought the issue was request related
> but it is in response handling. It means it does trigger the request which
> I have verified and it seems like the auth header is missing :(. I have
> tried with default URL handler and even with older ivy version 2.4.0 with
> older commons-httpclient. Nothing seems to send the auth header with
> credentials :(. You may simply run this example i.e., *ant -verbose
> retrieve* and see the request captured here:
> *https://beeceptor.com/console/ivytry
>  *

I believe this is because your service there doesn't require any
authentcation at all. Apache HttpClient doesn't do preemptive
authentication, it only sends credentials when really necessary. This is
understandable from a security perspective. Why send credentials to
somebody who never wanted them in the first place.

So it performs an HTTP GET without any credentials and if this request
succeeds, that's it. And this happens with your beeceptor example.

You probably need to make beeceptor return 401 on the first request
answering with

WWW-Authenticate: Basic realm=Basic

or the configured credentials are not going to be ever used.

Stefan

RE: Re: NullPointerException while resolving dependency

[Kamran Manzoor]

Hi Stefan,

Thanks a lot for looking into this. I thought the issue was request related
but it is in response handling. It means it does trigger the request which
I have verified and it seems like the auth header is missing :(. I have
tried with default URL handler and even with older ivy version 2.4.0 with
older commons-httpclient. Nothing seems to send the auth header with
credentials :(. You may simply run this example i.e., *ant -verbose
retrieve* and see the request captured here:
*https://beeceptor.com/console/ivytry
 *

I have spent quite some time without luck but I might be doing something
wrong. I would greatly appreciate any pointers.

Kind Regards,
Kamran Manzoor

On 2023/03/17 12:00:23 Stefan Bodewig wrote:
> On 2023-03-17, Stefan Bodewig wrote:
>
> > On 2023-03-17, Stefan Bodewig wrote:
>
> >> and will open a JIRA ticket attaching oyur files when using Apache
> >> HttpClient.
>
> > https://issues.apache.org/jira/browse/IVY-1641
>
> should be fixed by
>
https://github.com/apache/ant-ivy/commit/c63ce79f52133857d1146cd2335d51178714effd
>
> Stefan
>

Re: NullPointerException while resolving dependency

[Stefan Bodewig]

On 2023-03-17, Stefan Bodewig wrote:

> On 2023-03-17, Stefan Bodewig wrote:

>> and will open a JIRA ticket attaching oyur files when using Apache
>> HttpClient.

> https://issues.apache.org/jira/browse/IVY-1641

should be fixed by
https://github.com/apache/ant-ivy/commit/c63ce79f52133857d1146cd2335d51178714effd

Stefan

Re: NullPointerException while resolving dependency

[Stefan Bodewig]

On 2023-03-17, Stefan Bodewig wrote:

> and will open a JIRA ticket attaching oyur files when using Apache
> HttpClient.

https://issues.apache.org/jira/browse/IVY-1641

Stefan

Re: NullPointerException while resolving dependency

[Stefan Bodewig]

On 2023-03-16, Kamran Manzoor wrote:

> I am new to Apache Ivy and I am using it to manage dependency in one of my
> ant projects. All I want is to retrieve a jar from a custom URL (Bitbucket)
> which requires Basic Authentication. However, I am getting
> NullPointerException when I am trying to retrieve the jar with version
> *2.5.1*.

> *java.lang.NullPointerException at
> org.apache.ivy.util.url.HttpClientHandler.getURLInfo(HttpClientHandler.java:243)*

> I don't get NPE with *2.4.0* version of ivy but it does not send
> credentials in the header. Any pointers/help will be greatly appreciated. I
> have attached the simple example files here along with log files of both
> the versions to reproduce the issue. Ant version is *1.10.7* for me.

I can reproduce the problem you see (and will open a JIRA ticket
attaching oyur files) when using Apache HttpClient. Everything works as
expected when not using Apache HttpClient but the "normal"
java.net.URLConnection based code path. At least I dodn't get any
exception, I haven't checked whether the credentials are included. Maybe
you can use this as a workaround until we get the bug fixed and a
patched release out.

Stefan