Re: [twsocket] THttpCli and Socks Proxy
Thanks for the support Angus.
Yes, I was using ICS v7Gold previously, so have now updated to v8w, which
no doubt brings other benefits with it. I can confirm that your change is
working for me.
--
Colin
"TWSocket" <twsocket-boun...@lists.elists.org> wrote on 08/12/2017
07:51:00 a.m.:
> From: "Angus Robertson - Magenta Systems Ltd" <an...@magsys.co.uk>
> To: twsocket@lists.elists.org
> Date: 08/12/2017 07:52 a.m.
> Subject: Re: [twsocket] THttpCli and Socks Proxy
> Sent by: "TWSocket" <twsocket-boun...@lists.elists.org>
>
> > So today I have dived into OverByteIcsHttpProt.pas and made a few
> > small hacks as under to bypass the DNS lookup:
> > FCtrlSocket.DnsLookup('');
>
> I've fixed the SOCKS host name issue for the HTTP client and improved
> the error messages that come back when a proxy fails.
>
> It's in SVN now, it will be zipped overnight.
>
> My change is not quite the same as yours, since in the current ICS
> version attempting a blank DNS lookup raises an immediate exception, so
> perhaps you are using old code. I did improve DNS error handling last
> year.
>
> I've also improved the HTTP client sample for better proxy and socks
> testing, including authentication.
>
> Our SMTP and FTP clients also support socks so probably need fixing as
> well, but I guess few people are using it otherwise these issues would
> have come to light over the last 15 years.
>
> Most socks servers support UDP so DNS lookups would be possible, but
> ICS currently relies on Windows for all client DNS lookups and you can
> not specify the DNS server, Windows uses whatever is configured.
>
> ICS does have a DnsQuery component, but it would not be easy to plug
> into WSocket, it would need to handle timeouts which are common with
> DNS.
>
> Our DNS lookups do need improvement, for instance many large sites
> return a list of IP addresses for multiple servers, but we currently
> only contact the first and ignore the rest, ideally the next attempt
> should use another IP.
>
> Angus
>
>
>
> --
> To unsubscribe or change your settings for TWSocket mailing list
> please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
> Visit our website at http://www.overbyte.be
###
This email is confidential and may contain information subject to legal
privilege. If you are not the intended recipient please advise us of our
error by return e-mail then delete this email and any attached files.
You may not copy, disclose or use the contents in any way.
The views expressed in this email may not be those of Gallagher Group
Ltd or subsidiary companies thereof.
###
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be
Re: [twsocket] THttpCli and Socks Proxy
False alarm, I see that SSL handshake is negotiated after receiving response from proxy. That line was using HTTP proxy not Socks. I haven't seen servers that support SSL socks negotiation either. While it really has some sense (complete hiding all the communication including the host addresses), it's quite rare and probably hard to implement (will need two SSL negotiations). A little googling suggests the SOCKS protocols do not officially support SSL, although some implementations like Wingate will accept an SSL/TLS connection before the SOCKS protocol is negotiated. But that would not be end to end SSL, since the proxy would be making a non-HTTP connection. There are a couple of SSL SOCKS projects on Github, but not looked to see what they offer. My current feeling is we should fix SOCKS if it has got broken by new ICS releases, but any SOCKS improvements will need to be commercially organised. Angus -- A.S. -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] THttpCli and Socks Proxy
> Btw, isn't this line > > FCtrlSocket.SslEnable := ((FProxy = '') and (FProtocol = > > 'https')); > silently drops security transfer? A little googling suggests the SOCKS protocols do not officially support SSL, although some implementations like Wingate will accept an SSL/TLS connection before the SOCKS protocol is negotiated. But that would not be end to end SSL, since the proxy would be making a non-HTTP connection. There are a couple of SSL SOCKS projects on Github, but not looked to see what they offer. My current feeling is we should fix SOCKS if it has got broken by new ICS releases, but any SOCKS improvements will need to be commercially organised. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] THttpCli and Socks Proxy
Yes you're right. I see this quite weird code. From the 1st quick sight I guess: Login => DnsLookup --> SocketDNSLookupDone => Connect should be Login => Connect --> SocketDNSLookupDone (where => is direct call and --> is async handler) But I might be missing something important. FTP client has this issue too :) Except that HttpCli doesn't get this far when supplied with a domain name. As Angus suggests in his post, it tries to do a dns lookup first, and fails. -- Colin Btw, isn't this line FCtrlSocket.SslEnable := ((FProxy = '') and (FProtocol = 'https')); silently drops security transfer? I guess it should be an exception but not silent transition to plain text transfer with potential security risks. But I haven't used HttpCli with SSL so it might be a false alarm. -- A.S. -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] THttpCli and Socks Proxy
> So today I have dived into OverByteIcsHttpProt.pas and made a few > small hacks as under to bypass the DNS lookup: Thanks, I will look at this, but not this month. Stuff like this is easy, it only got broken because SOCKS got forgotten. > For me, a cleaner way would be to get the IP address from the > proxy server relating to the existing WSocket connection, but I > don't know how to go about it. ICS does have a DNS lookup component that will request full domain details from any DNS server, like the MX records to send email, look at the sample OverbyteIcsNsLookup.dpr. But this assumes the SOCKS proxy offers a DNS server, which is easy to test with the sample. Since passing a host name is part of the SOCKS protocol, they might not want you to know the final IP address. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] THttpCli and Socks Proxy
So today I have dived into OverByteIcsHttpProt.pas and made a few small
hacks as under to bypass the DNS lookup:
In procedure THttpCli.Login;
<>
FCtrlSocket.LocalAddr := FLocalAddr; {bb}
try
if SocksServer = '' then
FCtrlSocket.DnsLookup(FHostName)
else
FCtrlSocket.DnsLookup('');{CW Skip the local DNS lookup,
but trigger DnsLookupDone}
except
on E: Exception do begin
<...>
and in
procedure THttpCli.SocketDNSLookupDone(Sender: TObject; ErrCode: Word);
begin
if (ErrCode <> 0) AND ((SocksServer <> '') AND (ErrCode <> 10022))
then begin
if FState = httpAborting then
Exit;
<>
else begin
if SocksServer = '' then
FDnsResult:= FCtrlSocket.DnsResult
else
FDnsResult:= FHostName; {CW Use the domain name }
StateChange(httpDnsLookupDone); { 19/09/98 }
<>
It gets the job done, but I'm not really comfortable doing this to the
work of others who know so much more than me. But hey, I'll use it if a
better solution doesn't come to light!
For me, a cleaner way would be to get the IP address from the proxy server
relating to the existing WSocket connection, but I don't know how to go
about it.
Thanks to the respondents.
--
Colin
Colin Wall/GGL wrote on 24/11/2017 09:01:17 a.m.:
> From: Colin Wall/GGL
> To: ICS support mailing <twsocket@lists.elists.org>
> Date: 24/11/2017 09:01 a.m.
> Subject: Re: [twsocket] THttpCli and Socks Proxy
>
> Except that HttpCli doesn't get this far when supplied with a domain
> name. As Angus suggests in his post, it tries to do a dns lookup
> first, and fails.
>
> --
> Colin
>
> "TWSocket" <twsocket-boun...@lists.elists.org> wrote on 24/11/2017
> 01:24:18 a.m.:
>
> > From: "A.S." <an...@rambler.ru>
> > To: twsocket@lists.elists.org
> > Date: 24/11/2017 01:24 a.m.
> > Subject: Re: [twsocket] THttpCli and Socks Proxy
> > Sent by: "TWSocket" <twsocket-boun...@lists.elists.org>
> >
> > TCustomSocksWSocket.SocksDoConnect has this code for SocksLevel <>
'4':
> >
> > Buf[3] := #$03;{ Address type is domain name }
> > Buf[4] := AnsiChar((Length(FAddrStr)));
> > { Should check buffer overflow }
> > Move(AnsiString(FAddrStr)[1], Buf[5], Length(FAddrStr)); //
No
> > length change expected (ASCII)
> >
> > So everything is as it should be
> >
> > > Is there some way I can encourage it to use the proxy for DNS
resolution?
> > > Alternatively, how can I get the remote IP address from the existing
> > > TWSocket connection? (PeerAddr only gives me the IP of the proxy
server).
> > >
> > > Any assistance with this issue will be much appreciated.
> > >
> > > Thanks,
> > > Colin
> > >
> >
> > --
> > A.S.
> >
> > --
> > To unsubscribe or change your settings for TWSocket mailing list
> > please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
> > Visit our website at http://www.overbyte.be
###
This email is confidential and may contain information subject to legal
privilege. If you are not the intended recipient please advise us of our
error by return e-mail then delete this email and any attached files.
You may not copy, disclose or use the contents in any way.
The views expressed in this email may not be those of Gallagher Group
Ltd or subsidiary companies thereof.
###
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be
Re: [twsocket] THttpCli and Socks Proxy
Except that HttpCli doesn't get this far when supplied with a domain name.
As Angus suggests in his post, it tries to do a dns lookup first, and
fails.
--
Colin
"TWSocket" <twsocket-boun...@lists.elists.org> wrote on 24/11/2017
01:24:18 a.m.:
> From: "A.S." <an...@rambler.ru>
> To: twsocket@lists.elists.org
> Date: 24/11/2017 01:24 a.m.
> Subject: Re: [twsocket] THttpCli and Socks Proxy
> Sent by: "TWSocket" <twsocket-boun...@lists.elists.org>
>
> TCustomSocksWSocket.SocksDoConnect has this code for SocksLevel <> '4':
>
> Buf[3] := #$03;{ Address type is domain name }
> Buf[4] := AnsiChar((Length(FAddrStr)));
> { Should check buffer overflow }
> Move(AnsiString(FAddrStr)[1], Buf[5], Length(FAddrStr)); // No
> length change expected (ASCII)
>
> So everything is as it should be
>
> > Is there some way I can encourage it to use the proxy for DNS
resolution?
> > Alternatively, how can I get the remote IP address from the existing
> > TWSocket connection? (PeerAddr only gives me the IP of the proxy
server).
> >
> > Any assistance with this issue will be much appreciated.
> >
> > Thanks,
> > Colin
> >
>
> --
> A.S.
>
> --
> To unsubscribe or change your settings for TWSocket mailing list
> please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
> Visit our website at http://www.overbyte.be
###
This email is confidential and may contain information subject to legal
privilege. If you are not the intended recipient please advise us of our
error by return e-mail then delete this email and any attached files.
You may not copy, disclose or use the contents in any way.
The views expressed in this email may not be those of Gallagher Group
Ltd or subsidiary companies thereof.
###
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be
Re: [twsocket] THttpCli and Socks Proxy
TCustomSocksWSocket.SocksDoConnect has this code for SocksLevel <> '4':
Buf[3] := #$03; { Address type is domain name }
Buf[4] := AnsiChar((Length(FAddrStr)));
{ Should check buffer overflow }
Move(AnsiString(FAddrStr)[1], Buf[5], Length(FAddrStr)); // No
length change expected (ASCII)
So everything is as it should be
Is there some way I can encourage it to use the proxy for DNS resolution?
Alternatively, how can I get the remote IP address from the existing
TWSocket connection? (PeerAddr only gives me the IP of the proxy server).
Any assistance with this issue will be much appreciated.
Thanks,
Colin
--
A.S.
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be
Re: [twsocket] THttpCli and Socks Proxy
> I was quickly able to implement the telnet part using a TWSocket > component. This works successfully through the Socks proxy and I > am very happy with it. A particular point to note is that DNS > resolution is done by the proxy machine, which is a requirement > of the project. > > For the http part I am looking at using a THttpCli component. I > am experimenting using the OverbyteIcsHttpGet demo, setting up > the Socks properties on HttpCli1. It works if I provide the > actual IP address of the remote server, but not when I enter a > url. It appears to use the local machine for the DNS lookup, not > the proxy. > > Is there some way I can encourage it to use the proxy for DNS > resolution? Alternatively, how can I get the remote IP address > from the existing TWSocket connection? (PeerAddr only gives me > the IP of the proxy server). Don't know much about Socks proxies, not used one for many years. But a quick look at Wikipedia suggests SOCKS4a and SOCKS5 allow a domain name to be passed in the protocol setup rather than an IP address which is how sockets are normally opened. Normally ICS components do a DNS lookup first, then use that IP to open a socket. For SOCKS, the lookup would need to be skipped, and it's possible WSocket is doing that, but the HTTP client is not, which would explain your experience. This might have worked in the distant past, but got broken as other improvements were made, and no-one noticed because SOCKS use is rare. But I've not looked at any source code and will not be able to test any of this for a couple of weeks. Maybe someone else is actually using SOCKS with ICS? Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
