httpd: New log format to log X-Forwarded-{For|Port} headers
Hi,
I've completely reworked my patch for httpd(8). The last patch broke the
log format combined. And the config option was ugly. This time I've
added another log format called forwarded. It appends two fields to the
log format combined: The first field contains the value of the header
X-Forwarded-For and the second one the value of X-Forwarded-Port. If
either of the headers is empty or missing a dash (-) is written.
The new log format is compatible with log analyzing tools like Webalizer
or GoAccess. If you run httpd(8) behind a proxy like relayd(8) the new
log format finally gives you a way to track the origin of the requests.
Cheers,
Bruno
Index: usr.sbin/httpd/httpd.conf.5
===
RCS file: /cvs/src/usr.sbin/httpd/httpd.conf.5,v
retrieving revision 1.103
diff -u -p -r1.103 httpd.conf.5
--- usr.sbin/httpd/httpd.conf.5 19 Feb 2019 11:37:26 - 1.103
+++ usr.sbin/httpd/httpd.conf.5 27 Feb 2019 15:26:48 -
@@ -450,7 +450,8 @@ The
.Ar style
can be
.Cm common ,
-.Cm combined
+.Cm combined ,
+.Cm forwarded
or
.Cm connection .
The styles
@@ -459,6 +460,14 @@ and
.Cm combined
write a log entry after each request similar to the standard Apache
and nginx access log formats.
+The style
+.Cm forwarded
+extends the style
+.Cm combined
+by appending two fields containing the values of the headers
+.Ar X-Forwarded-For
+and
+.Ar X-Forwarded-Port .
The style
.Cm connection
writes a summarized log entry after each connection,
Index: usr.sbin/httpd/httpd.h
===
RCS file: /cvs/src/usr.sbin/httpd/httpd.h,v
retrieving revision 1.143
diff -u -p -r1.143 httpd.h
--- usr.sbin/httpd/httpd.h 19 Feb 2019 11:37:26 - 1.143
+++ usr.sbin/httpd/httpd.h 27 Feb 2019 15:26:48 -
@@ -437,7 +437,8 @@ SPLAY_HEAD(client_tree, client);
enum log_format {
LOG_FORMAT_COMMON,
LOG_FORMAT_COMBINED,
- LOG_FORMAT_CONNECTION
+ LOG_FORMAT_CONNECTION,
+ LOG_FORMAT_FORWARDED
};
struct log_file {
Index: usr.sbin/httpd/parse.y
===
RCS file: /cvs/src/usr.sbin/httpd/parse.y,v
retrieving revision 1.110
diff -u -p -r1.110 parse.y
--- usr.sbin/httpd/parse.y 19 Feb 2019 11:37:26 - 1.110
+++ usr.sbin/httpd/parse.y 27 Feb 2019 15:26:48 -
@@ -140,7 +140,7 @@ typedef struct {
%token PROTOCOLS REQUESTS ROOT SACK SERVER SOCKET STRIP STYLE SYSLOG TCP TICKET
%token TIMEOUT TLS TYPE TYPES HSTS MAXAGE SUBDOMAINS DEFAULT PRELOAD REQUEST
%token ERROR INCLUDE AUTHENTICATE WITH BLOCK DROP RETURN PASS REWRITE
-%token CA CLIENT CRL OPTIONAL PARAM
+%token CA CLIENT CRL OPTIONAL PARAM FORWARDED
%token STRING
%token NUMBER
%type port
@@ -1024,6 +1024,11 @@ logstyle : COMMON{
srv_conf->flags |= SRVFLAG_LOG;
srv_conf->logformat = LOG_FORMAT_CONNECTION;
}
+ | FORWARDED {
+ srv_conf->flags &= ~SRVFLAG_NO_LOG;
+ srv_conf->flags |= SRVFLAG_LOG;
+ srv_conf->logformat = LOG_FORMAT_FORWARDED;
+ }
;
filter : block RETURN NUMBER optstring {
@@ -1295,6 +1300,7 @@ lookup(char *s)
{ "ecdhe", ECDHE },
{ "error", ERR },
{ "fastcgi",FCGI },
+ { "forwarded", FORWARDED },
{ "hsts", HSTS },
{ "include",INCLUDE },
{ "index", INDEX },
Index: usr.sbin/httpd/server_http.c
===
RCS file: /cvs/src/usr.sbin/httpd/server_http.c,v
retrieving revision 1.129
diff -u -p -r1.129 server_http.c
--- usr.sbin/httpd/server_http.c10 Feb 2019 13:41:27 - 1.129
+++ usr.sbin/httpd/server_http.c27 Feb 2019 15:26:49 -
@@ -1632,7 +1632,7 @@ server_log_http(struct client *clt, unsi
static char tstamp[64];
static char ip[INET6_ADDRSTRLEN];
time_t t;
- struct kvkey, *agent, *referrer;
+ struct kvkey, *agent, *referrer, *xff, *xfp;
struct tm *tm;
struct server_config*srv_conf;
struct http_descriptor *desc;
@@ -1642,6 +1642,8 @@ server_log_http(struct client *clt, unsi
char*version = NULL;
char*referrer_v = NULL;
char*agent_v = NULL;
+ char*xff_v = NULL;
+ char*xfp_v = NULL;
if ((srv_conf = clt->clt_srv_conf) == NULL)
return (-1);
@@ -1698,6 +1700,7 @@ server_log_http(struct client *clt, unsi
brea
Re: httpd: New log format to log X-Forwarded-{For|Port} headers
Hi,
On Mon, Mar 04, 2019 at 02:06:02PM +0100, Bruno Flueckiger wrote:
> I've completely reworked my patch for httpd(8). The last patch broke the
> log format combined. And the config option was ugly. This time I've
> added another log format called forwarded. It appends two fields to the
> log format combined: The first field contains the value of the header
> X-Forwarded-For and the second one the value of X-Forwarded-Port. If
> either of the headers is empty or missing a dash (-) is written.
>
> The new log format is compatible with log analyzing tools like Webalizer
> or GoAccess. If you run httpd(8) behind a proxy like relayd(8) the new
> log format finally gives you a way to track the origin of the requests.
>
Your diff looks clean and makes a lot of sense.
Especially since X-Forwarded-For is a feature in relayd that I first
used and documented around 2006/2007. Adding the forwarded style to
httpd is a complementary feature in OpenBSD and not something for a
random external web stack.
OK reyk@
Anyone else, any objections?
Reyk
> Cheers,
> Bruno
>
> Index: usr.sbin/httpd/httpd.conf.5
> ===
> RCS file: /cvs/src/usr.sbin/httpd/httpd.conf.5,v
> retrieving revision 1.103
> diff -u -p -r1.103 httpd.conf.5
> --- usr.sbin/httpd/httpd.conf.5 19 Feb 2019 11:37:26 - 1.103
> +++ usr.sbin/httpd/httpd.conf.5 27 Feb 2019 15:26:48 -
> @@ -450,7 +450,8 @@ The
> .Ar style
> can be
> .Cm common ,
> -.Cm combined
> +.Cm combined ,
> +.Cm forwarded
> or
> .Cm connection .
> The styles
> @@ -459,6 +460,14 @@ and
> .Cm combined
> write a log entry after each request similar to the standard Apache
> and nginx access log formats.
> +The style
> +.Cm forwarded
> +extends the style
> +.Cm combined
> +by appending two fields containing the values of the headers
> +.Ar X-Forwarded-For
> +and
> +.Ar X-Forwarded-Port .
> The style
> .Cm connection
> writes a summarized log entry after each connection,
> Index: usr.sbin/httpd/httpd.h
> ===
> RCS file: /cvs/src/usr.sbin/httpd/httpd.h,v
> retrieving revision 1.143
> diff -u -p -r1.143 httpd.h
> --- usr.sbin/httpd/httpd.h19 Feb 2019 11:37:26 - 1.143
> +++ usr.sbin/httpd/httpd.h27 Feb 2019 15:26:48 -
> @@ -437,7 +437,8 @@ SPLAY_HEAD(client_tree, client);
> enum log_format {
> LOG_FORMAT_COMMON,
> LOG_FORMAT_COMBINED,
> - LOG_FORMAT_CONNECTION
> + LOG_FORMAT_CONNECTION,
> + LOG_FORMAT_FORWARDED
> };
>
> struct log_file {
> Index: usr.sbin/httpd/parse.y
> ===
> RCS file: /cvs/src/usr.sbin/httpd/parse.y,v
> retrieving revision 1.110
> diff -u -p -r1.110 parse.y
> --- usr.sbin/httpd/parse.y19 Feb 2019 11:37:26 - 1.110
> +++ usr.sbin/httpd/parse.y27 Feb 2019 15:26:48 -
> @@ -140,7 +140,7 @@ typedef struct {
> %token PROTOCOLS REQUESTS ROOT SACK SERVER SOCKET STRIP STYLE SYSLOG
> TCP TICKET
> %token TIMEOUT TLS TYPE TYPES HSTS MAXAGE SUBDOMAINS DEFAULT PRELOAD
> REQUEST
> %token ERROR INCLUDE AUTHENTICATE WITH BLOCK DROP RETURN PASS REWRITE
> -%token CA CLIENT CRL OPTIONAL PARAM
> +%token CA CLIENT CRL OPTIONAL PARAM FORWARDED
> %token STRING
> %token NUMBER
> %typeport
> @@ -1024,6 +1024,11 @@ logstyle : COMMON{
> srv_conf->flags |= SRVFLAG_LOG;
> srv_conf->logformat = LOG_FORMAT_CONNECTION;
> }
> + | FORWARDED {
> + srv_conf->flags &= ~SRVFLAG_NO_LOG;
> + srv_conf->flags |= SRVFLAG_LOG;
> + srv_conf->logformat = LOG_FORMAT_FORWARDED;
> + }
> ;
>
> filter : block RETURN NUMBER optstring {
> @@ -1295,6 +1300,7 @@ lookup(char *s)
> { "ecdhe", ECDHE },
> { "error", ERR },
> { "fastcgi",FCGI },
> + { "forwarded", FORWARDED },
> { "hsts", HSTS },
> { "include",INCLUDE },
> { "index", INDEX },
> Index: usr.sbin/httpd/server_http.c
> ===
> RCS file: /cvs/src/usr.sbin/httpd/server_http.c,v
> retrieving revision 1.129
> diff -u -p -r1.129 server_http.c
> --- usr.sbin/httpd/server_http.c 10 Feb 2019 13:41:27 - 1.129
> +++ usr.sbin/httpd/server_http.c 27 Feb 2019 15:26:49 -
> @@ -1632,7 +1632,7 @@ server_log_http(struct client *clt, unsi
> static char tstamp[64];
> static char ip[INET6_ADDRSTRLEN];
> time_t t;
> - struct kvkey, *agent, *referrer;
> + struct kvke
Re: httpd: New log format to log X-Forwarded-{For|Port} headers
On Fri, Mar 08, 2019 at 10:52:28AM +0100, Reyk Floeter wrote:
> Hi,
>
> On Mon, Mar 04, 2019 at 02:06:02PM +0100, Bruno Flueckiger wrote:
> > I've completely reworked my patch for httpd(8). The last patch broke the
> > log format combined. And the config option was ugly. This time I've
> > added another log format called forwarded. It appends two fields to the
> > log format combined: The first field contains the value of the header
> > X-Forwarded-For and the second one the value of X-Forwarded-Port. If
> > either of the headers is empty or missing a dash (-) is written.
> >
> > The new log format is compatible with log analyzing tools like Webalizer
> > or GoAccess. If you run httpd(8) behind a proxy like relayd(8) the new
> > log format finally gives you a way to track the origin of the requests.
> >
>
> Your diff looks clean and makes a lot of sense.
>
> Especially since X-Forwarded-For is a feature in relayd that I first
> used and documented around 2006/2007. Adding the forwarded style to
> httpd is a complementary feature in OpenBSD and not something for a
> random external web stack.
>
> OK reyk@
>
> Anyone else, any objections?
That would be really nice to have. Did this slip through the cracks or
are there concerns with this diff?
>
> Reyk
>
> > Cheers,
> > Bruno
> >
> > Index: usr.sbin/httpd/httpd.conf.5
> > ===
> > RCS file: /cvs/src/usr.sbin/httpd/httpd.conf.5,v
> > retrieving revision 1.103
> > diff -u -p -r1.103 httpd.conf.5
> > --- usr.sbin/httpd/httpd.conf.5 19 Feb 2019 11:37:26 - 1.103
> > +++ usr.sbin/httpd/httpd.conf.5 27 Feb 2019 15:26:48 -
> > @@ -450,7 +450,8 @@ The
> > .Ar style
> > can be
> > .Cm common ,
> > -.Cm combined
> > +.Cm combined ,
> > +.Cm forwarded
> > or
> > .Cm connection .
> > The styles
> > @@ -459,6 +460,14 @@ and
> > .Cm combined
> > write a log entry after each request similar to the standard Apache
> > and nginx access log formats.
> > +The style
> > +.Cm forwarded
> > +extends the style
> > +.Cm combined
> > +by appending two fields containing the values of the headers
> > +.Ar X-Forwarded-For
> > +and
> > +.Ar X-Forwarded-Port .
> > The style
> > .Cm connection
> > writes a summarized log entry after each connection,
> > Index: usr.sbin/httpd/httpd.h
> > ===
> > RCS file: /cvs/src/usr.sbin/httpd/httpd.h,v
> > retrieving revision 1.143
> > diff -u -p -r1.143 httpd.h
> > --- usr.sbin/httpd/httpd.h 19 Feb 2019 11:37:26 - 1.143
> > +++ usr.sbin/httpd/httpd.h 27 Feb 2019 15:26:48 -
> > @@ -437,7 +437,8 @@ SPLAY_HEAD(client_tree, client);
> > enum log_format {
> > LOG_FORMAT_COMMON,
> > LOG_FORMAT_COMBINED,
> > - LOG_FORMAT_CONNECTION
> > + LOG_FORMAT_CONNECTION,
> > + LOG_FORMAT_FORWARDED
> > };
> >
> > struct log_file {
> > Index: usr.sbin/httpd/parse.y
> > ===
> > RCS file: /cvs/src/usr.sbin/httpd/parse.y,v
> > retrieving revision 1.110
> > diff -u -p -r1.110 parse.y
> > --- usr.sbin/httpd/parse.y 19 Feb 2019 11:37:26 - 1.110
> > +++ usr.sbin/httpd/parse.y 27 Feb 2019 15:26:48 -
> > @@ -140,7 +140,7 @@ typedef struct {
> > %token PROTOCOLS REQUESTS ROOT SACK SERVER SOCKET STRIP STYLE SYSLOG
> > TCP TICKET
> > %token TIMEOUT TLS TYPE TYPES HSTS MAXAGE SUBDOMAINS DEFAULT PRELOAD
> > REQUEST
> > %token ERROR INCLUDE AUTHENTICATE WITH BLOCK DROP RETURN PASS REWRITE
> > -%token CA CLIENT CRL OPTIONAL PARAM
> > +%token CA CLIENT CRL OPTIONAL PARAM FORWARDED
> > %token STRING
> > %token NUMBER
> > %type port
> > @@ -1024,6 +1024,11 @@ logstyle : COMMON{
> > srv_conf->flags |= SRVFLAG_LOG;
> > srv_conf->logformat = LOG_FORMAT_CONNECTION;
> > }
> > + | FORWARDED {
> > + srv_conf->flags &= ~SRVFLAG_NO_LOG;
> > + srv_conf->flags |= SRVFLAG_LOG;
> > + srv_conf->logformat = LOG_FORMAT_FORWARDED;
> > + }
> > ;
> >
> > filter : block RETURN NUMBER optstring {
> > @@ -1295,6 +1300,7 @@ lookup(char *s)
> > { "ecdhe", ECDHE },
> > { "error", ERR },
> > { "fastcgi",FCGI },
> > + { "forwarded", FORWARDED },
> > { "hsts", HSTS },
> > { "include",INCLUDE },
> > { "index", INDEX },
> > Index: usr.sbin/httpd/server_http.c
> > ===
> > RCS file: /cvs/src/usr.sbin/httpd/server_http.c,v
> > retrieving revision 1.129
> > diff -u -p -r1.129 server_http.c
> > --- usr.sbin/httpd/server_http.c10 Feb 2019 13:41:27 - 1.129
> > +++ usr.sbin/h
Re: httpd: New log format to log X-Forwarded-{For|Port} headers
> On 3 May 2019, at 04:59, Theo Buehler wrote:
>
>> On Fri, Mar 08, 2019 at 10:52:28AM +0100, Reyk Floeter wrote:
>> Hi,
>>
>>> On Mon, Mar 04, 2019 at 02:06:02PM +0100, Bruno Flueckiger wrote:
>>> I've completely reworked my patch for httpd(8). The last patch broke the
>>> log format combined. And the config option was ugly. This time I've
>>> added another log format called forwarded. It appends two fields to the
>>> log format combined: The first field contains the value of the header
>>> X-Forwarded-For and the second one the value of X-Forwarded-Port. If
>>> either of the headers is empty or missing a dash (-) is written.
>>>
>>> The new log format is compatible with log analyzing tools like Webalizer
>>> or GoAccess. If you run httpd(8) behind a proxy like relayd(8) the new
>>> log format finally gives you a way to track the origin of the requests.
>>>
>>
>> Your diff looks clean and makes a lot of sense.
>>
>> Especially since X-Forwarded-For is a feature in relayd that I first
>> used and documented around 2006/2007. Adding the forwarded style to
>> httpd is a complementary feature in OpenBSD and not something for a
>> random external web stack.
>>
>> OK reyk@
>>
>> Anyone else, any objections?
>
> That would be really nice to have. Did this slip through the cracks or
> are there concerns with this diff?
>
I believe it fell through the cracks. Would be super useful.
Mischa
>>
>> Reyk
>>
>>> Cheers,
>>> Bruno
>>>
>>> Index: usr.sbin/httpd/httpd.conf.5
>>> ===
>>> RCS file: /cvs/src/usr.sbin/httpd/httpd.conf.5,v
>>> retrieving revision 1.103
>>> diff -u -p -r1.103 httpd.conf.5
>>> --- usr.sbin/httpd/httpd.conf.519 Feb 2019 11:37:26 -1.103
>>> +++ usr.sbin/httpd/httpd.conf.527 Feb 2019 15:26:48 -
>>> @@ -450,7 +450,8 @@ The
>>> .Ar style
>>> can be
>>> .Cm common ,
>>> -.Cm combined
>>> +.Cm combined ,
>>> +.Cm forwarded
>>> or
>>> .Cm connection .
>>> The styles
>>> @@ -459,6 +460,14 @@ and
>>> .Cm combined
>>> write a log entry after each request similar to the standard Apache
>>> and nginx access log formats.
>>> +The style
>>> +.Cm forwarded
>>> +extends the style
>>> +.Cm combined
>>> +by appending two fields containing the values of the headers
>>> +.Ar X-Forwarded-For
>>> +and
>>> +.Ar X-Forwarded-Port .
>>> The style
>>> .Cm connection
>>> writes a summarized log entry after each connection,
>>> Index: usr.sbin/httpd/httpd.h
>>> ===
>>> RCS file: /cvs/src/usr.sbin/httpd/httpd.h,v
>>> retrieving revision 1.143
>>> diff -u -p -r1.143 httpd.h
>>> --- usr.sbin/httpd/httpd.h19 Feb 2019 11:37:26 -1.143
>>> +++ usr.sbin/httpd/httpd.h27 Feb 2019 15:26:48 -
>>> @@ -437,7 +437,8 @@ SPLAY_HEAD(client_tree, client);
>>> enum log_format {
>>>LOG_FORMAT_COMMON,
>>>LOG_FORMAT_COMBINED,
>>> -LOG_FORMAT_CONNECTION
>>> +LOG_FORMAT_CONNECTION,
>>> +LOG_FORMAT_FORWARDED
>>> };
>>>
>>> struct log_file {
>>> Index: usr.sbin/httpd/parse.y
>>> ===
>>> RCS file: /cvs/src/usr.sbin/httpd/parse.y,v
>>> retrieving revision 1.110
>>> diff -u -p -r1.110 parse.y
>>> --- usr.sbin/httpd/parse.y19 Feb 2019 11:37:26 -1.110
>>> +++ usr.sbin/httpd/parse.y27 Feb 2019 15:26:48 -
>>> @@ -140,7 +140,7 @@ typedef struct {
>>> %tokenPROTOCOLS REQUESTS ROOT SACK SERVER SOCKET STRIP STYLE SYSLOG TCP
>>> TICKET
>>> %tokenTIMEOUT TLS TYPE TYPES HSTS MAXAGE SUBDOMAINS DEFAULT PRELOAD
>>> REQUEST
>>> %tokenERROR INCLUDE AUTHENTICATE WITH BLOCK DROP RETURN PASS REWRITE
>>> -%tokenCA CLIENT CRL OPTIONAL PARAM
>>> +%tokenCA CLIENT CRL OPTIONAL PARAM FORWARDED
>>> %tokenSTRING
>>> %token NUMBER
>>> %typeport
>>> @@ -1024,6 +1024,11 @@ logstyle: COMMON{
>>>srv_conf->flags |= SRVFLAG_LOG;
>>>srv_conf->logformat = LOG_FORMAT_CONNECTION;
>>>}
>>> +| FORWARDED{
>>> +srv_conf->flags &= ~SRVFLAG_NO_LOG;
>>> +srv_conf->flags |= SRVFLAG_LOG;
>>> +srv_conf->logformat = LOG_FORMAT_FORWARDED;
>>> +}
>>>;
>>>
>>> filter: block RETURN NUMBER optstring{
>>> @@ -1295,6 +1300,7 @@ lookup(char *s)
>>>{ "ecdhe",ECDHE },
>>>{ "error",ERR },
>>>{ "fastcgi",FCGI },
>>> +{ "forwarded",FORWARDED },
>>>{ "hsts",HSTS },
>>>{ "include",INCLUDE },
>>>{ "index",INDEX },
>>> Index: usr.sbin/httpd/server_http.c
>>> ===
>>> RCS file: /cvs/src/usr.sbin/httpd/server_http.c,v
>>> retrieving revision 1.129
>>> diff -u -p -r1.129 server_http.c
>>> --- usr.sbin/httpd/server_http.c10 Feb 2019 13:41:27 -1.129
>>> +++ usr.sbin/httpd/server_http.c
Re: httpd: New log format to log X-Forwarded-{For|Port} headers
On 03.05., Theo Buehler wrote:
> On Fri, Mar 08, 2019 at 10:52:28AM +0100, Reyk Floeter wrote:
> > Hi,
> >
> > On Mon, Mar 04, 2019 at 02:06:02PM +0100, Bruno Flueckiger wrote:
> > > I've completely reworked my patch for httpd(8). The last patch broke the
> > > log format combined. And the config option was ugly. This time I've
> > > added another log format called forwarded. It appends two fields to the
> > > log format combined: The first field contains the value of the header
> > > X-Forwarded-For and the second one the value of X-Forwarded-Port. If
> > > either of the headers is empty or missing a dash (-) is written.
> > >
> > > The new log format is compatible with log analyzing tools like Webalizer
> > > or GoAccess. If you run httpd(8) behind a proxy like relayd(8) the new
> > > log format finally gives you a way to track the origin of the requests.
> > >
> >
> > Your diff looks clean and makes a lot of sense.
> >
> > Especially since X-Forwarded-For is a feature in relayd that I first
> > used and documented around 2006/2007. Adding the forwarded style to
> > httpd is a complementary feature in OpenBSD and not something for a
> > random external web stack.
> >
> > OK reyk@
> >
> > Anyone else, any objections?
>
> That would be really nice to have. Did this slip through the cracks or
> are there concerns with this diff?
>
In an earlier discussion there were some concerns about log analyzers
like Webalizer and GoAccess. For GoAccess the following awk script
converts the access.log to the combined format:
#!/usr/bin/awk
{
if ($0 ~ /newsyslog/) next
ip = $(NF - 1)
sub(/127.0.0.1/, ip)
for (i = 2; i <= NF - 2; i++)
printf("%s ", $i)
printf("\n")
}
Cheers,
Bruno
Re: httpd: New log format to log X-Forwarded-{For|Port} headers
Theo Buehler(t...@theobuehler.org) on 2019.05.03 04:59:16 +0200:
> On Fri, Mar 08, 2019 at 10:52:28AM +0100, Reyk Floeter wrote:
> > Hi,
> >
> > On Mon, Mar 04, 2019 at 02:06:02PM +0100, Bruno Flueckiger wrote:
> > > I've completely reworked my patch for httpd(8). The last patch broke the
> > > log format combined. And the config option was ugly. This time I've
> > > added another log format called forwarded. It appends two fields to the
> > > log format combined: The first field contains the value of the header
> > > X-Forwarded-For and the second one the value of X-Forwarded-Port. If
> > > either of the headers is empty or missing a dash (-) is written.
> > >
> > > The new log format is compatible with log analyzing tools like Webalizer
> > > or GoAccess. If you run httpd(8) behind a proxy like relayd(8) the new
> > > log format finally gives you a way to track the origin of the requests.
> > >
> >
> > Your diff looks clean and makes a lot of sense.
> >
> > Especially since X-Forwarded-For is a feature in relayd that I first
> > used and documented around 2006/2007. Adding the forwarded style to
> > httpd is a complementary feature in OpenBSD and not something for a
> > random external web stack.
> >
> > OK reyk@
> >
> > Anyone else, any objections?
>
> That would be really nice to have. Did this slip through the cracks or
> are there concerns with this diff?
Hi,
please commit it, my ok too.
If it turns out that there are still problems with any tool, we can come
back to it, but as far as i know these tools should work with this output.
/Benno
>
> >
> > Reyk
> >
> > > Cheers,
> > > Bruno
> > >
> > > Index: usr.sbin/httpd/httpd.conf.5
> > > ===
> > > RCS file: /cvs/src/usr.sbin/httpd/httpd.conf.5,v
> > > retrieving revision 1.103
> > > diff -u -p -r1.103 httpd.conf.5
> > > --- usr.sbin/httpd/httpd.conf.5 19 Feb 2019 11:37:26 - 1.103
> > > +++ usr.sbin/httpd/httpd.conf.5 27 Feb 2019 15:26:48 -
> > > @@ -450,7 +450,8 @@ The
> > > .Ar style
> > > can be
> > > .Cm common ,
> > > -.Cm combined
> > > +.Cm combined ,
> > > +.Cm forwarded
> > > or
> > > .Cm connection .
> > > The styles
> > > @@ -459,6 +460,14 @@ and
> > > .Cm combined
> > > write a log entry after each request similar to the standard Apache
> > > and nginx access log formats.
> > > +The style
> > > +.Cm forwarded
> > > +extends the style
> > > +.Cm combined
> > > +by appending two fields containing the values of the headers
> > > +.Ar X-Forwarded-For
> > > +and
> > > +.Ar X-Forwarded-Port .
> > > The style
> > > .Cm connection
> > > writes a summarized log entry after each connection,
> > > Index: usr.sbin/httpd/httpd.h
> > > ===
> > > RCS file: /cvs/src/usr.sbin/httpd/httpd.h,v
> > > retrieving revision 1.143
> > > diff -u -p -r1.143 httpd.h
> > > --- usr.sbin/httpd/httpd.h19 Feb 2019 11:37:26 - 1.143
> > > +++ usr.sbin/httpd/httpd.h27 Feb 2019 15:26:48 -
> > > @@ -437,7 +437,8 @@ SPLAY_HEAD(client_tree, client);
> > > enum log_format {
> > > LOG_FORMAT_COMMON,
> > > LOG_FORMAT_COMBINED,
> > > - LOG_FORMAT_CONNECTION
> > > + LOG_FORMAT_CONNECTION,
> > > + LOG_FORMAT_FORWARDED
> > > };
> > >
> > > struct log_file {
> > > Index: usr.sbin/httpd/parse.y
> > > ===
> > > RCS file: /cvs/src/usr.sbin/httpd/parse.y,v
> > > retrieving revision 1.110
> > > diff -u -p -r1.110 parse.y
> > > --- usr.sbin/httpd/parse.y19 Feb 2019 11:37:26 - 1.110
> > > +++ usr.sbin/httpd/parse.y27 Feb 2019 15:26:48 -
> > > @@ -140,7 +140,7 @@ typedef struct {
> > > %token PROTOCOLS REQUESTS ROOT SACK SERVER SOCKET STRIP STYLE SYSLOG
> > > TCP TICKET
> > > %token TIMEOUT TLS TYPE TYPES HSTS MAXAGE SUBDOMAINS DEFAULT PRELOAD
> > > REQUEST
> > > %token ERROR INCLUDE AUTHENTICATE WITH BLOCK DROP RETURN PASS REWRITE
> > > -%token CA CLIENT CRL OPTIONAL PARAM
> > > +%token CA CLIENT CRL OPTIONAL PARAM FORWARDED
> > > %token STRING
> > > %token NUMBER
> > > %typeport
> > > @@ -1024,6 +1024,11 @@ logstyle : COMMON{
> > > srv_conf->flags |= SRVFLAG_LOG;
> > > srv_conf->logformat = LOG_FORMAT_CONNECTION;
> > > }
> > > + | FORWARDED {
> > > + srv_conf->flags &= ~SRVFLAG_NO_LOG;
> > > + srv_conf->flags |= SRVFLAG_LOG;
> > > + srv_conf->logformat = LOG_FORMAT_FORWARDED;
> > > + }
> > > ;
> > >
> > > filter : block RETURN NUMBER optstring {
> > > @@ -1295,6 +1300,7 @@ lookup(char *s)
> > > { "ecdhe", ECDHE },
> > > { "error", ERR },
> > > { "fastcgi",FCGI },
> > > + { "forwarded", FORWARDED },
> > >
Re: httpd: New log format to log X-Forwarded-{For|Port} headers
On Mon, Mar 04, 2019 at 02:06:02PM +0100, Bruno Flueckiger wrote: > Hi, > > I've completely reworked my patch for httpd(8). The last patch broke the > log format combined. And the config option was ugly. This time I've > added another log format called forwarded. It appends two fields to the > log format combined: The first field contains the value of the header > X-Forwarded-For and the second one the value of X-Forwarded-Port. If > either of the headers is empty or missing a dash (-) is written. > > The new log format is compatible with log analyzing tools like Webalizer > or GoAccess. If you run httpd(8) behind a proxy like relayd(8) the new > log format finally gives you a way to track the origin of the requests. Committed, thanks!
Re: httpd: New log format to log X-Forwarded-{For|Port} headers
-- > On 3 May 2019, at 19:19, Theo Buehler wrote: > >> On Mon, Mar 04, 2019 at 02:06:02PM +0100, Bruno Flueckiger wrote: >> Hi, >> >> I've completely reworked my patch for httpd(8). The last patch broke the >> log format combined. And the config option was ugly. This time I've >> added another log format called forwarded. It appends two fields to the >> log format combined: The first field contains the value of the header >> X-Forwarded-For and the second one the value of X-Forwarded-Port. If >> either of the headers is empty or missing a dash (-) is written. >> >> The new log format is compatible with log analyzing tools like Webalizer >> or GoAccess. If you run httpd(8) behind a proxy like relayd(8) the new >> log format finally gives you a way to track the origin of the requests. > > Committed, thanks! Great! You are making a lot of people very happy! Mischa
