Re: [Bug-wget] Anyone want to add libcares support to wget?

[Tim Ruehsen]

Tim

On Thursday 17 March 2016 09:36:47 Ben Greear wrote:
> On 03/17/2016 08:58 AM, Giuseppe Scrivano wrote:
> > Hi Ben,
> > 
> > Ben Greear  writes:
> >>> Did you consider using a container (e.g. docker) for such a task ? Easy
> >>> to set up and you'll have your feature not only for wget. IMO, that is
> >>> much more flexible. (It was Giuseppe's idea during a private talk).
> >> 
> >> Containers will not work for me.  I need to scale to thousands of
> >> instances on modest hardware.  I'm certain the libcares and binding
> >> approach will work because we do similar things with curl and other
> >> programs already.
> > 
> > how modest is this hardware?
> > 
> > I tried to build a minimal Docker container for wget on top of Alpine:
> > 
> > Dockerfile:
> > 
> > FROM alpine
> > RUN apk add --update wget
> > WORKDIR /out
> > ENTRYPOINT ["/usr/bin/wget"]
> > 
> > and simply running (192.168.1.13 is an internal DNS server):
> > 
> > docker run --dns=192.168.1.13 --rm -v $(pwd):/out:Z wget wdserver
> > 
> > takes around 5M, plus the docker daemon.
> > 
> > If this is not enough, you can even just run wget in a chroot, and
> > provide a different /etc/resolv.conf.
> 
> I really am not at all interested in LD_PRELOAD, docker, chroot, or any
> other such a thing.  I have 15+ years invested in my current architecture,
> and it works very well for me.  It absolutely cannot be easier to manage
> docker instances on random platforms vs using a few cmd-line switches to
> wget.
> 
> I know that this feature will not be useful for most people, but then again,
> none of us can know what some user might be able to do with such a feature.
>  I know that some users of curl + libcares wanted the ability to specify
> certain DNS servers to get around oppressive firewalls (on systems they
> could not have root privs), for instance.
> 
> I don't mind making the changes to wget.  I thought if someone was
> interested in a project, then I'd be happy to work with them (and pay them,
> etc).

Why not put an offer here (with an exact description of what to deliver, OS, 
etc) ?


> I'll post my patches for consideration, and you will be welcome to pull it
> into your tree.  If they are not acceptable, then I can maintain the fork
> for as long as my customers care to use it..

Your patches are welcome, no doubt.

Just please understand our position that we try to find solutions by other 
means before introducing new options. 

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Implement --pinnedpubkey option to pin public keys

[moparisthebest]

And of course NOW I see the Test--https.py file and that https tests are
indeed supported.  I'll write up some tests and send them shortly.

On 03/18/2016 02:10 AM, moparisthebest wrote:
> The documentation in testenv/ says the test server doesn't support
> https, which would be needed for this test.  Has anyone started work on
> that?  Or would it be acceptable to just use socat or stunnel or similar
> in front of the current test server?
>



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] Anyone want to add libcares support to wget?

[Ángel González]

On 16/03/16 15:35, Ben Greear wrote:
wget can already bind to a local IP.  It might be nice to add support 
for SO_BINDTODEVICE, but not sure

it is required for what I need.

The rest of your suggestions are total hacks!


The place for setting this is the resolver. It's not reasonable to 
expect every application to be changed in order to link with a different 
resolver just for this tiny feature.

I would prefer that libc provided instead of relying on LD_PRELOAD, though.


Anyway, I'm quite sure of what I want, and it should not be horribly 
difficult

to accomplish.  I'll work on it if no one else is interested.

You look too much like xkcd's 1172 LongtimeUser4.
Good that you are willing to do the work, though.

Best regards

Re: [Bug-wget] [bug #47408] Wget sends malformed SNI host names

[Tim Ruehsen]

On Wednesday 16 March 2016 11:59:04 Daniel Stenberg wrote:
> On Wed, 16 Mar 2016, Tim Ruehsen wrote:
> > Here is a patch for both openssl and gnutls. Please comment, I'll push it
> > tomorrow.
> 
> The bug report says the SNI field should be different than the Host: header,
> but I question the sensibility in that. What would be the point? (pun not
> intended =B))
> 
> When requesting contents from an HTTPS site, the SNI field will tell the
> server which particular virtual server to get the data from and when the
> trailing dot gets stripped the two strings with and without dot will end up
> on the same virtual server. Sending a Host: header that doesn't match the
> virtual server name then is then likely to either get ignored or to cause
> the HTTP backend to complain.
> 
> It will also make it behave a bit different for HTTP than for HTTPS since
> then there's no SNI field and the Host: header is what will be used and
> then they clearly are different servers.
> 
> And incidentally, curl strips the trailing dot off from both SNI and Host:
> =)

That is what I would like to do as well. It seems consistent. And the patch 
introduces non-elegant code (not really my favor).

And for DNS lookups... is there a difference between dot and not-dot (e.g. 
example.com vs. example.com.) ?

The patch follows Yst Dawson's conclusion, though:
"That means that the SNI host name and HTTP Host header do not always match.
The SNI host name must never have a trailing dot, but the HTTP Host header
must reflect a host name that is identical to the host name of the URI, so if
the URI's host has a trailing dot, the HTTP Host header must include that
trailing dot."

Also what Jay Satiro says:
"I tried this in Firefox, Chrome and IE and all send the trailing dot for 
SNI."

Should we follow the browsers or curl ?

Tim

Re: [Bug-wget] [bug #47408] Wget sends malformed SNI host names

[Daniel Stenberg]

On Wed, 16 Mar 2016, Tim Ruehsen wrote:


Should we follow the browsers or curl ?


I brought this subject to the http-wg mailing list, possibly we can clear it 
up on a wider scale:


 https://lists.w3.org/Archives/Public/ietf-http-wg/2016JanMar/0430.html

--

 / daniel.haxx.se

Re: [Bug-wget] Anyone want to add libcares support to wget?

[Tim Ruehsen]

Tim

On Wednesday 16 March 2016 07:35:44 Ben Greear wrote:
> On 03/15/2016 06:40 PM, Ángel González wrote:
> > On 15/03/16 17:03, Ben Greear wrote:
> >> I want to use wget on multi-homed machines, and I use routing rules to
> >> route pkts out specific interfaces based on the source IP.
> >> 
> >> So, I need to be able to bind to a local IP address, including for the
> >> resolver logic, which is why I want to use libcares.
> >> 
> >> To test, I think it would be sufficient to put two IPs on a port
> >> (1.1.1.1, 1.1.1.2), and then use cmd-line args to choose one or the
> >> other.  Sniff and watch for proper source address.
> >> 
> >> For DNS servers, have 1.1.1.3 and 1.1.1.4 be specified.  Sniff and watch
> >> to see that the resolver destination IP is as specified.  This should be
> >> completely independent of whatever is in /etc/resolv.conf, etc.
> >> 
> >> I (and others) have patched libcares and curl in the past to handle all
> >> of this, so I know libcares can do it.  I think the main work is to
> >> handle new cmd-line arguments, pass appropriate info to libcares, and
> >> make wget use libcares for it's resolve logic.
> >> 
> >> Thanks,
> >> Ben
> > 
> > I think it's ok to add a --bind parameter to wget to choose the interface
> > for download, but I don't think it should use a different resolver just
> > for this.
> > 
> > You _could_ change the libc nameservers from the application by changing
> > _res.nsaddr_list after res_init(). It may be a bit fragile, but should
> > work fine. And even better than changing each application, I would
> > recommend you to create a simple LD_PRELOAD library that overrides
> > res_init() & friends, and allows you to specify the target dns server
> > with an environment variable. It's a pity there's no RESOLV_CONF var
> > similar to RESOLV_HOST_CONF, for using a separate /etc/resolv.conf.
> wget can already bind to a local IP.  It might be nice to add support for
> SO_BINDTODEVICE, but not sure it is required for what I need.

@Ángel The LD_PRELOAD idea is good. I really like it !

> 
> The rest of your suggestions are total hacks!
> 
> Anyway, I'm quite sure of what I want, and it should not be horribly
> difficult to accomplish.  I'll work on it if no one else is interested.

Thanks for your offer.
I expect a configure option to switch such a feature on (default=off) to avoid 
a new dependency (c-ares) for a feature that just one out of a million users 
need. How would you design an automated test ?

Did you consider using a container (e.g. docker) for such a task ? Easy to set 
up and you'll have your feature not only for wget. IMO, that is much more 
flexible. (It was Giuseppe's idea during a private talk).

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

[Bug-wget] [bug #47425] crash on download on specific hardware (Dell Dimension 2400)

[Tim Ruehsen]

Follow-up Comment #1, bug #47425 (project wget):

This sounds like your version of Wget contains assembler code that your old
P4s doesn't know. Try to compile a version on your P4 machines, or just stay
with a working executable !?


___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/

[Bug-wget] [bug #46611] log errors with --trust-server-names

[Ángel]

Follow-up Comment #9, bug #46611 (project wget):

The problem is not that with -E a .html is appended, but that it checks .exe
but reports about .exe.html

Running:
LANG=C strace -e stat,open wget -E -d --timestamping --trust-server-names
http://downloads.malwarebytes.org/file/mbam

I get:
304 Not Modified
Disabling further reuse of socket 3.
Closed fd 3
Registered socket 4 for persistent reuse.
stat("mbam-setup-2.2.0.1024.exe", {st_mode=S_IFREG|0644, st_size=4509107,
...}) = 0
File 'mbam-setup-2.2.0.1024.exe.html' not modified on server. Omitting
download.

which shows very clearly how wget worked on mbam-setup-2.2.0.1024.exe but is
reporting about the missing mbam-setup-2.2.0.1024.exe.html

I agree in not changing the default Content-Type, but that's not the bug :)


___

Reply to this item at:

  

___
  Mensaje enviado vía/por Savannah
  http://savannah.gnu.org/