Re: [fossil-users] [PATCH] IPv6 support and improved reverse proxying support
Lluís Batlle i Rossell writes: On Mon, Oct 03, 2011 at 11:58:08AM -0700, Gé Weijers wrote: On Mon, 3 Oct 2011, Lluís Batlle i Rossell wrote: Additionally, I don't know how portable it is to use always getaddrinfo (POSIX-2001?) while requiring C89. C89 does not address networking, so this issue is unrelated to C89. If getaddrinfo is not supported on a platform that uses the socket API it may be time to move away from it :-( :) I agree. I was just wondering, how old (or simple) systems/compilers we expect fossil to build on. The only extra complexity it adds is that a server has to be able to listen to multiple sockets. Some OSes may give you a single socket that can handle both IPv4 and IPv6 connections, and others will not. I know I know. But posix allows a 'setsockopt' to force it do both ipv4 and ipv6 in the same socket, isn't it? Different OSes have different 'defaults' for that though. IPV6_V6ONLY, and this patch explicitly disables that socket option to make sure that AF_INET6 socket should be able to receive IPv4 connections as well. HTH -- Ashish SHUKLA “Vengeance is mine; I will repay.” (Leo Tolstoy, Anna Karenina, (1875–1877)) pgpaVuEjjmBpp.pgp Description: PGP signature ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [PATCH] IPv6 support and improved reverse proxying support
Hi Ashish,
On Wed, 5 Oct 2011, Ashish SHUKLA wrote:
I wasn't aware of both sockaddr_storage, and getnameinfo(). They seem good to
me, and I've updated diff[1] to use them.
References:
[1] http://people.freebsd.org/~ashish/fossil-ipv6-rev-proxy.diff
A few comments (not all about your code):
You are using the ss_len member of struct sockaddr_storage. 'ss_len' is
not required to be there by Posix AFAIK. You can just use the size
returned by 'getsockname' or 'getpeername' (third argument).
In my own code I use getaddrinfo(...localhost, port) to generate one
or more sockets to allow connection to localhost only. Works quite well,
and you don't need any special-purpose code for generating a local-only
socket. This would require restructuring the cgi server code to handle
multiple sockets, because on some systems (FreeBSD) you _will_ get
multiple sockets. In general it should be possible to write both the
client and server code without referring to a specific protocol (IPv4 or
IPv6), except if you want to pass a specific protocol to getaddrinfo.
It would simplify things if you pass the port number to 'getnameinfo' as a
string in stead of assigning it later in your code. This code could then
be removed:
+if(i-ai_family == AF_INET) {
+ ((struct sockaddr_in*)i-ai_addr)-sin_port = htons(g.urlPort);
+} else if(i-ai_family == AF_INET6) {
+ ((struct sockaddr_in6*)i-ai_addr)-sin6_port = htons(g.urlPort);
+}
The existing code (before your patch) has one problem: if a connection
disappears between the return of 'select' and the 'accept' call the accept
can 'hang' until the next connection gets made. The 60 second timeout will
therefor not always work. The way to get around that is set the listening
socket to non-blocking, and turn non-blocking back off for the socket you
get from 'accept'. If accept produces and error with errno equal to
EWOULDBLOCK or ECONNABORTED you just ignore it.
I've attached some sample code that more or less implements this. The code
does not fork a new process, but it does the other stuff.
Ge'#include stdio.h
#include stddef.h
#include stdlib.h
#include string.h
#include errno.h
#include sys/types.h
#include sys/socket.h
#include netinet/in.h
#include netdb.h
#include sys/select.h
#include fcntl.h
#include unistd.h
int open_server_sockets(int socklist[], unsigned maxsock, const char *host,
const char *svc)
{
int r;
struct addrinfo hints, *res, *res0;
const char *fail = NULL, *code = NULL;
unsigned numsock = 0;
memset(hints, 0, sizeof(hints));
hints.ai_flags = AI_ADDRCONFIG | AI_PASSIVE;
hints.ai_family = PF_UNSPEC;
hints.ai_socktype = SOCK_STREAM;
hints.ai_protocol = IPPROTO_TCP;
r = getaddrinfo(host, svc, hints, res0);
if(r != 0){
fprintf(stderr, can't resolve %s:%s: %s\n, (host ? host : ANY),
(svc ? svc : ANY), gai_strerror(r));
return -1;
}
for(res = res0; res != NULL numsock maxsock; res = res-ai_next){
int s = socket(res-ai_family, res-ai_socktype, res-ai_protocol);
int yes = 1, t;
if(s 0){
fail = socket; code = strerror(errno);
continue;
}
if(setsockopt(s, SOL_SOCKET, SO_REUSEADDR, yes, sizeof(yes)) 0){
fail = setsockopt; code = strerror(errno);
continue;
}
if(bind(s, res-ai_addr, res-ai_addrlen) 0){
fail = bind; code = strerror(errno);
continue;
}
t = fcntl(s, F_GETFL, 0);
if(t 0 || fcntl(s, F_SETFL, t | O_NONBLOCK) 0){
fail = fcntl; code = strerror(errno);
continue;
}
if(listen(s, 256) 0){
fail = listen; code = strerror(errno);
continue;
}
socklist[numsock++] = s;
}
freeaddrinfo(res0);
if(numsock == 0 fail){
fprintf(stderr, %s failed: %s\n, fail, code);
return -1;
}else if(fail){
fprintf(stderr, warning: %s failed: %s\n, fail, code);
}
return (int)numsock;
}
int simple_server(int sock)
{
char host[NI_MAXHOST], svc[NI_MAXSERV];
int err, t;
struct sockaddr_storage sa;
socklen_t salen = sizeof(sa);
if(getpeername(sock, (struct sockaddr *)sa, salen) 0){
fprintf(stderr, can't get peer address: %s\n, strerror(errno));
return -1;
}
if((err = getnameinfo((struct sockaddr *)sa, salen, host, sizeof(host),
svc, sizeof(svc), NI_NUMERICHOST | NI_NUMERICSERV)) 0){
fprintf(stderr, can decode socket address: %s\n, gai_strerror(err));
return -1;
}else{
printf(connection originating from %s:%s\n, host, svc);
}
t = fcntl(sock, F_GETFL, 0);
if(t 0 || fcntl(sock, F_SETFL, t ~O_NONBLOCK) 0){
fprintf(stderr, can't turn off O_NONBLOCK: %s\n, strerror(errno));
return -1;
}
[fossil-users] [PATCH] IPv6 support and improved reverse proxying support
Hi everyone,
I've written attached diff sometime ago to add:
1. Improved reverse proxy support. I'm not sure if there is any explicit
support for reverse proxy, but the way in which I was trying to use it,
i.e. exposing fossil repository at a _location_ in nginx, it wasn't
working. So I decided to fix issues I encountered and added explicit
support for it in the code which now works as I expected:
--8---cut here---start-8---
chateau.d.if!abbe fossils % fossil server -R /code -P 8080 .
--8---cut here---end---8---
And following is the corresponding nginx configuration to proxy traffic at
/code/ to fossil server listening at port 8080:
--8---cut here---start-8---
location /code/ {
proxy_pass http://localhost:8080/ ;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect http://localhost:8080/code/ http://$http_host/code/;
}
--8---cut here---end---8---
If this is achievable using current version (hosted at: fossil-scm.org),
please let me know.
2. IPv6 support. Fossil uses IPv4 sockets by default. I was not sure if there
was any technical reason to not add support for IPv6, so I modified it all
relevant places (I was able to find) to use IPv6 sockets, perform hostname
lookups using getaddrinfo(3), accept push/pulls to/from IPv6 address, and
persist this information in the repository database.
I'm using it with the Fossil snapshot of September, 2011 with FreeBSD 8.2-R
(amd64) without any issues. Please let me know if you find any problems with
code.
I'll be glad if these changes makes to the trunk.
Thanks
--
Ashish SHUKLA
“The wonderful thing about science is that it doesn't ask for your
faith, it just ask for your eyes.” (xkcd #154)
Index: auto.def
===
--- auto.def
+++ auto.def
@@ -8,10 +8,11 @@
with-zlib:path = {Look for zlib in the given path}
internal-sqlite=1= {Don't use the internal sqlite, use the system one}
static=0 = {Link a static executable}
lineedit=1 = {Disable line editing}
fossil-debug=0 = {Build with fossil debugging enabled}
+ipv6=1 = {Disable IPv6 support}
}
# sqlite wants these types if possible
cc-with {-includes {stdint.h inttypes.h}} {
cc-check-types uint32_t uint16_t int16_t uint8_t
@@ -65,10 +66,19 @@
}
if {[opt-bool static]} {
# XXX: This will not work on all systems.
define-append EXTRA_LDFLAGS -static
+}
+
+if {[opt-bool ipv6]} {
+ define-append EXTRA_CFLAGS -DWITH_IPV6
+ msg-result IPv6 support enabled
+ if {[cc-check-functions getaddrinfo]} {
+ define-append EXTRA_CFLAGS -DHAVE_GETADDRINFO
+ msg-result getaddrinfo() enabled
+ }
}
# Check for zlib, using the given location if specified
set zlibpath [opt-val with-zlib]
Index: src/cgi.c
===
--- src/cgi.c
+++ src/cgi.c
@@ -992,12 +992,12 @@
** and subsequent code handles the actual generation of the webpage.
*/
void cgi_handle_http_request(const char *zIpAddr){
char *z, *zToken;
int i;
- struct sockaddr_in remoteName;
- socklen_t size = sizeof(struct sockaddr_in);
+ char remoteName[256];
+ socklen_t size = sizeof(remoteName);
char zLine[2000]; /* A single line of input. */
g.fullHttpReply = 1;
if( fgets(zLine, sizeof(zLine),g.httpIn)==0 ){
malformed_request();
@@ -1019,20 +1019,10 @@
cgi_setenv(REQUEST_URI, zToken);
for(i=0; zToken[i] zToken[i]!='?'; i++){}
if( zToken[i] ) zToken[i++] = 0;
cgi_setenv(PATH_INFO, zToken);
cgi_setenv(QUERY_STRING, zToken[i]);
- if( zIpAddr==0
-getpeername(fileno(g.httpIn), (struct sockaddr*)remoteName,
-size)=0
- ){
-zIpAddr = inet_ntoa(remoteName.sin_addr);
- }
- if( zIpAddr ){
-cgi_setenv(REMOTE_ADDR, zIpAddr);
-g.zIpAddr = mprintf(%s, zIpAddr);
- }
/* Get all the optional fields that follow the first line.
*/
while( fgets(zLine,sizeof(zLine),g.httpIn) ){
char *zFieldName;
@@ -1059,18 +1049,57 @@
cgi_setenv(HTTP_HOST, zVal);
}else if( fossil_strcmp(zFieldName,if-none-match:)==0 ){
cgi_setenv(HTTP_IF_NONE_MATCH, zVal);
}else if( fossil_strcmp(zFieldName,if-modified-since:)==0 ){
cgi_setenv(HTTP_IF_MODIFIED_SINCE, zVal);
+}else if( fossil_strcmp(zFieldName,x-forwarded-for:)==0 ){
+ char* p = zVal;
+ /*
+ ** x-forwarded-for header is a list of comma-separated addresses,
+ ** with leftmost address corresponding to the client
+ */
+ while(*p *p != ',') p++;
+ *p = '\0';
+ zIpAddr = mprintf( %s, zVal );
}
#if 0
else if( fossil_strcmp(zFieldName,referer:)==0 ){
cgi_setenv(HTTP_REFERER, zVal);
}else if(
Re: [fossil-users] [PATCH] IPv6 support and improved reverse proxying support
Please send in a Contributors License Agreement (
http://www.fossil-scm.org/fossil/doc/trunk/www/copyright-release.html) so
that we can use your code.
On Mon, Oct 3, 2011 at 6:30 AM, Ashish SHUKLA ashish...@lostca.se wrote:
Hi everyone,
I've written attached diff sometime ago to add:
1. Improved reverse proxy support. I'm not sure if there is any explicit
support for reverse proxy, but the way in which I was trying to use it,
i.e. exposing fossil repository at a _location_ in nginx, it wasn't
working. So I decided to fix issues I encountered and added explicit
support for it in the code which now works as I expected:
--8---cut here---start-8---
chateau.d.if!abbe fossils % fossil server -R /code -P 8080 .
--8---cut here---end---8---
And following is the corresponding nginx configuration to proxy traffic
at
/code/ to fossil server listening at port 8080:
--8---cut here---start-8---
location /code/ {
proxy_pass http://localhost:8080/ ;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect http://localhost:8080/code/ http://$http_host/code/;
}
--8---cut here---end---8---
If this is achievable using current version (hosted at: fossil-scm.org),
please let me know.
2. IPv6 support. Fossil uses IPv4 sockets by default. I was not sure if
there
was any technical reason to not add support for IPv6, so I modified it
all
relevant places (I was able to find) to use IPv6 sockets, perform
hostname
lookups using getaddrinfo(3), accept push/pulls to/from IPv6 address, and
persist this information in the repository database.
I'm using it with the Fossil snapshot of September, 2011 with FreeBSD 8.2-R
(amd64) without any issues. Please let me know if you find any problems
with
code.
I'll be glad if these changes makes to the trunk.
Thanks
--
Ashish SHUKLA
“The wonderful thing about science is that it doesn't ask for your
faith, it just ask for your eyes.” (xkcd #154)
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
--
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [PATCH] IPv6 support and improved reverse proxying support
On Mon, 03 Oct 2011 16:00:14 +0530 ashish...@lostca.se (Ashish SHUKLA) wrote: [...] 2. IPv6 support. Fossil uses IPv4 sockets by default. I was not sure if there was any technical reason to not add support for IPv6, so I modified it all relevant places (I was able to find) to use IPv6 sockets, perform hostname lookups using getaddrinfo(3), accept push/pulls to/from IPv6 address, and persist this information in the repository database. [...] That is cool, but please be sure to make such IPv6 mode not enabled by default unless it's somehow possible to make it work transparently on a conventional box with IPv4 only networking. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [PATCH] IPv6 support and improved reverse proxying support
On Mon, Oct 03, 2011 at 04:19:34PM +0400, Konstantin Khomoutov wrote: On Mon, 03 Oct 2011 16:00:14 +0530 ashish...@lostca.se (Ashish SHUKLA) wrote: [...] 2. IPv6 support. Fossil uses IPv4 sockets by default. I was not sure if there was any technical reason to not add support for IPv6, so I modified it all relevant places (I was able to find) to use IPv6 sockets, perform hostname lookups using getaddrinfo(3), accept push/pulls to/from IPv6 address, and persist this information in the repository database. [...] That is cool, but please be sure to make such IPv6 mode not enabled by default unless it's somehow possible to make it work transparently on a conventional box with IPv4 only networking. That's what I saw from the patch at first glance; that building with -DWITH_IPV6 fossil required a connected ipv6 stack. Additionally, I don't know how portable it is to use always getaddrinfo (POSIX-2001?) while requiring C89. The code could go right now into a branch, and then we change it according to needs and tests. Regards, Lluís. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [PATCH] IPv6 support and improved reverse proxying support
On Mon, 3 Oct 2011 14:39:18 +0200 Lluís Batlle i Rossell virik...@gmail.com wrote: 2. IPv6 support. Fossil uses IPv4 sockets by default. I was not sure if there was any technical reason to not add support for IPv6, so I modified it all relevant places (I was able to find) to use IPv6 sockets, perform hostname lookups using getaddrinfo (3), accept push/pulls to/from IPv6 address, and persist this information in the repository database. That is cool, but please be sure to make such IPv6 mode not enabled by default unless it's somehow possible to make it work transparently on a conventional box with IPv4 only networking. That's what I saw from the patch at first glance; that building with -DWITH_IPV6 fossil required a connected ipv6 stack. Additionally, I don't know how portable it is to use always getaddrinfo (POSIX-2001?) while requiring C89. Tcl developers recently completed implementing IPv6 in the Tcl runtime, and Tcl runs even on obscure systems like AIX. So I think this should not be a problem these days. Personally I'm only concerned with this change not breaking things for the majority of users (who use IPv4 and will continue to do so in the foreseeable future). The code could go right now into a branch, and then we change it according to needs and tests. Good idea, thanks. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [PATCH] IPv6 support and improved reverse proxying support
Konstantin Khomoutov writes: On Mon, 03 Oct 2011 16:00:14 +0530 ashish...@lostca.se (Ashish SHUKLA) wrote: [...] 2. IPv6 support. Fossil uses IPv4 sockets by default. I was not sure if there was any technical reason to not add support for IPv6, so I modified it all relevant places (I was able to find) to use IPv6 sockets, perform hostname lookups using getaddrinfo(3), accept push/pulls to/from IPv6 address, and persist this information in the repository database. [...] That is cool, but please be sure to make such IPv6 mode not enabled by default unless it's somehow possible to make it work transparently on a conventional box with IPv4 only networking. Well, you can enable/disable at compile time. At runtime, it won't try to use IPv6 address if you don't have desired connectivity available, RFC3484[1] which getaddrinfo(3) implements. But if you enable IPv6 support at compile-time, it expects IPv6 support (in kernel) to be available at runtime, as it's using AF_INET6 sockets. References: [1] http://tools.ietf.org/html/rfc3484 HTH -- Ashish SHUKLA “I couldn't help but overhear, probably because I was eavesdropping.” (anonymous) pgpUKUcOQUQp1.pgp Description: PGP signature ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [PATCH] IPv6 support and improved reverse proxying support
On Mon, Oct 03, 2011 at 04:19:34PM +0400, Konstantin Khomoutov wrote: On Mon, 03 Oct 2011 16:00:14 +0530 ashish...@lostca.se (Ashish SHUKLA) wrote: [...] 2. IPv6 support. Fossil uses IPv4 sockets by default. I was not sure if there was any technical reason to not add support for IPv6, so I modified it all relevant places (I was able to find) to use IPv6 sockets, perform hostname lookups using getaddrinfo(3), accept push/pulls to/from IPv6 address, and persist this information in the repository database. [...] That is cool, but please be sure to make such IPv6 mode not enabled by default unless it's somehow possible to make it work transparently on a conventional box with IPv4 only networking. It should just work as long as your IPv6 setup is not broken. E.g. as long as nothing in your network advertises a IPv6 default route AND doesn't provide ICMP6 failures, it will behave. It might also be a saner option to just always use getaddrinfo() and one of the fallback implementations that emulate getaddrinfo based on gethostbyname etc. That has the big advantage of keeping the main code #if free. Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [PATCH] IPv6 support and improved reverse proxying support
On Mon, 3 Oct 2011, Lluís Batlle i Rossell wrote: Additionally, I don't know how portable it is to use always getaddrinfo (POSIX-2001?) while requiring C89. C89 does not address networking, so this issue is unrelated to C89. If getaddrinfo is not supported on a platform that uses the socket API it may be time to move away from it :-( The only extra complexity it adds is that a server has to be able to listen to multiple sockets. Some OSes may give you a single socket that can handle both IPv4 and IPv6 connections, and others will not. Ge' The code could go right now into a branch, and then we change it according to needs and tests. Regards, Lluís. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [PATCH] IPv6 support and improved reverse proxying support
On Mon, 03 Oct 2011 22:24:02 +0530 ashish...@lostca.se (Ashish SHUKLA) wrote: That is cool, but please be sure to make such IPv6 mode not enabled by default unless it's somehow possible to make it work transparently on a conventional box with IPv4 only networking. Well, you can enable/disable at compile time. At runtime, it won't try to use IPv6 address if you don't have desired connectivity available, RFC3484[1] which getaddrinfo(3) implements. But if you enable IPv6 support at compile-time, it expects IPv6 support (in kernel) to be available at runtime, as it's using AF_INET6 sockets. Is there no way to make it switchable at runtime (like via `fossil set ipv6 on`) or is it doable but just complicated to implement? Having a compile-time switch effectively doubles the number of binaries one have to built to offer for downloading. The same issue arises for downstream distro packagers. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [PATCH] IPv6 support and improved reverse proxying support
On Mon, Oct 03, 2011 at 11:58:08AM -0700, Gé Weijers wrote: On Mon, 3 Oct 2011, Lluís Batlle i Rossell wrote: Additionally, I don't know how portable it is to use always getaddrinfo (POSIX-2001?) while requiring C89. C89 does not address networking, so this issue is unrelated to C89. If getaddrinfo is not supported on a platform that uses the socket API it may be time to move away from it :-( :) I agree. I was just wondering, how old (or simple) systems/compilers we expect fossil to build on. The only extra complexity it adds is that a server has to be able to listen to multiple sockets. Some OSes may give you a single socket that can handle both IPv4 and IPv6 connections, and others will not. I know I know. But posix allows a 'setsockopt' to force it do both ipv4 and ipv6 in the same socket, isn't it? Different OSes have different 'defaults' for that though. Regards, Lluís. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [PATCH] IPv6 support and improved reverse proxying support
2011/10/3 Lluís Batlle i Rossell virik...@gmail.com On Mon, Oct 03, 2011 at 11:58:08AM -0700, Gé Weijers wrote: On Mon, 3 Oct 2011, Lluís Batlle i Rossell wrote: Additionally, I don't know how portable it is to use always getaddrinfo (POSIX-2001?) while requiring C89. C89 does not address networking, so this issue is unrelated to C89. If getaddrinfo is not supported on a platform that uses the socket API it may be time to move away from it :-( :) I agree. I was just wondering, how old (or simple) systems/compilers we expect fossil to build on. I need to build Fossil on a circa 2002 Mac iBook PPC running MacOS 10.2 (Juguar). And that currently works, with a few extra -D options manually added to the Makefile. I'd like to keep it working. The only extra complexity it adds is that a server has to be able to listen to multiple sockets. Some OSes may give you a single socket that can handle both IPv4 and IPv6 connections, and others will not. I know I know. But posix allows a 'setsockopt' to force it do both ipv4 and ipv6 in the same socket, isn't it? Different OSes have different 'defaults' for that though. Regards, Lluís. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
