Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Liping Zhang]

2016-10-11 11:57 GMT+08:00 Chris Caputo :
> I have tested the above patch with 4.8.1, with and without nflog-size
> defined in an iptables configuration, and it works well.
>
> The ulogd-2.0.5 segfaults no longer happen when nflog-size is not present
> in a target.
>
> I recommend this fix.

Thanks, I will send an official patch later.

Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Liping Zhang]

2016-10-11 11:57 GMT+08:00 Chris Caputo :
> I have tested the above patch with 4.8.1, with and without nflog-size
> defined in an iptables configuration, and it works well.
>
> The ulogd-2.0.5 segfaults no longer happen when nflog-size is not present
> in a target.
>
> I recommend this fix.

Thanks, I will send an official patch later.

Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Chris Caputo]

On Tue, 11 Oct 2016, Liping Zhang wrote:
> Yes, thanks for clarifying this. There's a bug in kernel, can you try
> this patch:
> 
> diff --git a/net/netfilter/xt_NFLOG.c b/net/netfilter/xt_NFLOG.c
> index 018eed7..8c069b4 100644
> --- a/net/netfilter/xt_NFLOG.c
> +++ b/net/netfilter/xt_NFLOG.c
> @@ -32,6 +32,7 @@ nflog_tg(struct sk_buff *skb, const struct
> xt_action_param *par)
> li.u.ulog.copy_len   = info->len;
> li.u.ulog.group  = info->group;
> li.u.ulog.qthreshold = info->threshold;
> +   li.u.ulog.flags  = 0;
> 
> if (info->flags & XT_NFLOG_F_COPY_LEN)
> li.u.ulog.flags |= NF_LOG_F_COPY_LEN;

I have tested the above patch with 4.8.1, with and without nflog-size 
defined in an iptables configuration, and it works well.

The ulogd-2.0.5 segfaults no longer happen when nflog-size is not present 
in a target.

I recommend this fix.

Thanks,
Chris

Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Chris Caputo]

On Tue, 11 Oct 2016, Liping Zhang wrote:
> Yes, thanks for clarifying this. There's a bug in kernel, can you try
> this patch:
> 
> diff --git a/net/netfilter/xt_NFLOG.c b/net/netfilter/xt_NFLOG.c
> index 018eed7..8c069b4 100644
> --- a/net/netfilter/xt_NFLOG.c
> +++ b/net/netfilter/xt_NFLOG.c
> @@ -32,6 +32,7 @@ nflog_tg(struct sk_buff *skb, const struct
> xt_action_param *par)
> li.u.ulog.copy_len   = info->len;
> li.u.ulog.group  = info->group;
> li.u.ulog.qthreshold = info->threshold;
> +   li.u.ulog.flags  = 0;
> 
> if (info->flags & XT_NFLOG_F_COPY_LEN)
> li.u.ulog.flags |= NF_LOG_F_COPY_LEN;

I have tested the above patch with 4.8.1, with and without nflog-size 
defined in an iptables configuration, and it works well.

The ulogd-2.0.5 segfaults no longer happen when nflog-size is not present 
in a target.

I recommend this fix.

Thanks,
Chris

Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Liping Zhang]

2016-10-11 2:33 GMT+08:00 Chris Caputo :
>>
>> What numbers did you specify after --nflog-size option?
>> --nflog-size 0 or ...? If you want log the whole packet to
>> the ulogd, please do not specify this nflog-size option.
>
> Not specifying nflog-size does not appear to log the whole packet...
>
> If "--nflog-size" is unspecified, and the iptables config is left
> unchanged when the kernel is upgraded to 4.8, ulogd-2.0.5 crashes.
>
> If "--nflog-size 0" is used, ulogd-2.0.5 crashes.
>
> If "--nflog-size" is used with size 1 or greater, ulogd-2.0.5 is fine.
>
>> > I'm surprised to see a kernel change cause unexpected userspace segfaults,
>> > so further investigation into a kernel fix would seem a good idea.
>>
>> According to the original user's manual, nflog-range option was
>> designed to be the number of bytes copied to userspace, but
>> unfortunately there's a bug from the beginning and it never works,
>> i.e. in kernel, it just ignored this option.
>>
>> Try to change the current nflog-range option's semantics may
>> cause unexpected results(maybe like this ulogd crash) ...
>>
>> In order to keep compatibility, Vishwanath introduce a new
>> nflog-size option and keep nflog-range unchanged. If you just
>> upgrade the kernel, and do not change iptables rules, this
>> problem will not happen.
>
> I am reporting that the problem does happen simply with an upgrade to
> kernel 4.8 and no other changes.  When "--nflog-size" is unspecified or
> set to 0, the bug in ulogd-2.0.5 gets triggered.
>
> I agree there is a bug in ulogd-2.0.5 that this kernel change exposed, but
> I am trying to explain that all ulogd users risk this segfault if they
> upgrade to kernel 4.8 and don't either update to a fixed ulogd (possibly
> using your patch below) or an unreleased iptables with iptables config
> changes to implement nflog-size on each NFLOG target.

Yes, thanks for clarifying this. There's a bug in kernel, can you try
this patch:

diff --git a/net/netfilter/xt_NFLOG.c b/net/netfilter/xt_NFLOG.c
index 018eed7..8c069b4 100644
--- a/net/netfilter/xt_NFLOG.c
+++ b/net/netfilter/xt_NFLOG.c
@@ -32,6 +32,7 @@ nflog_tg(struct sk_buff *skb, const struct
xt_action_param *par)
li.u.ulog.copy_len   = info->len;
li.u.ulog.group  = info->group;
li.u.ulog.qthreshold = info->threshold;
+   li.u.ulog.flags  = 0;

if (info->flags & XT_NFLOG_F_COPY_LEN)
li.u.ulog.flags |= NF_LOG_F_COPY_LEN;

Thanks

Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Liping Zhang]

2016-10-11 2:33 GMT+08:00 Chris Caputo :
>>
>> What numbers did you specify after --nflog-size option?
>> --nflog-size 0 or ...? If you want log the whole packet to
>> the ulogd, please do not specify this nflog-size option.
>
> Not specifying nflog-size does not appear to log the whole packet...
>
> If "--nflog-size" is unspecified, and the iptables config is left
> unchanged when the kernel is upgraded to 4.8, ulogd-2.0.5 crashes.
>
> If "--nflog-size 0" is used, ulogd-2.0.5 crashes.
>
> If "--nflog-size" is used with size 1 or greater, ulogd-2.0.5 is fine.
>
>> > I'm surprised to see a kernel change cause unexpected userspace segfaults,
>> > so further investigation into a kernel fix would seem a good idea.
>>
>> According to the original user's manual, nflog-range option was
>> designed to be the number of bytes copied to userspace, but
>> unfortunately there's a bug from the beginning and it never works,
>> i.e. in kernel, it just ignored this option.
>>
>> Try to change the current nflog-range option's semantics may
>> cause unexpected results(maybe like this ulogd crash) ...
>>
>> In order to keep compatibility, Vishwanath introduce a new
>> nflog-size option and keep nflog-range unchanged. If you just
>> upgrade the kernel, and do not change iptables rules, this
>> problem will not happen.
>
> I am reporting that the problem does happen simply with an upgrade to
> kernel 4.8 and no other changes.  When "--nflog-size" is unspecified or
> set to 0, the bug in ulogd-2.0.5 gets triggered.
>
> I agree there is a bug in ulogd-2.0.5 that this kernel change exposed, but
> I am trying to explain that all ulogd users risk this segfault if they
> upgrade to kernel 4.8 and don't either update to a fixed ulogd (possibly
> using your patch below) or an unreleased iptables with iptables config
> changes to implement nflog-size on each NFLOG target.

Yes, thanks for clarifying this. There's a bug in kernel, can you try
this patch:

diff --git a/net/netfilter/xt_NFLOG.c b/net/netfilter/xt_NFLOG.c
index 018eed7..8c069b4 100644
--- a/net/netfilter/xt_NFLOG.c
+++ b/net/netfilter/xt_NFLOG.c
@@ -32,6 +32,7 @@ nflog_tg(struct sk_buff *skb, const struct
xt_action_param *par)
li.u.ulog.copy_len   = info->len;
li.u.ulog.group  = info->group;
li.u.ulog.qthreshold = info->threshold;
+   li.u.ulog.flags  = 0;

if (info->flags & XT_NFLOG_F_COPY_LEN)
li.u.ulog.flags |= NF_LOG_F_COPY_LEN;

Thanks

Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Chris Caputo]

On Mon, 10 Oct 2016, Liping Zhang wrote:
> 2016-10-10 15:02 GMT+08:00 Chris Caputo :
> >   Program received signal SIGSEGV, Segmentation fault.
> >   0x765fd18a in _interp_iphdr (pi=0x617f50, len=0) at 
> > ulogd_raw2packet_BASE.c:720
> >
> >   715 static int _interp_iphdr(struct ulogd_pluginstance *pi, uint32_t 
> > len)
> >   716 {
> >   717 struct ulogd_key *ret = pi->output.keys;
> >   718 struct iphdr *iph =
> >   719 ikey_get_ptr(>input.keys[INKEY_RAW_PCKT]);
> >   720 void *nexthdr = (uint32_t *)iph + iph->ihl;
> >
> > I believe 7643507fe8b5bd8ab7522f6a81058cc1209d2585 changed previous
> > behavior by not always copying IP header data to user space.
> >
> > On my machine IPv4 log packets result in a ulogd segfault while IPv6
> > packets do not.  I'm not sure of the cause of the difference.
> >
> > The corresponding userspace commit for the 209d2585 kernel change is:
> >
> >   
> > https://git.netfilter.org/iptables/commit/?id=7070b1f3c88a0c3d4e315c00cca61f05b0fbc882
> >
> > This adds --nflog-size to iptables.  When --nflog-size is used with my
> > iptables NFLOG lines, the ulogd-2.0.5 segfaults cease.
> 
> What numbers did you specify after --nflog-size option?
> --nflog-size 0 or ...? If you want log the whole packet to
> the ulogd, please do not specify this nflog-size option.

Not specifying nflog-size does not appear to log the whole packet...

If "--nflog-size" is unspecified, and the iptables config is left 
unchanged when the kernel is upgraded to 4.8, ulogd-2.0.5 crashes.

If "--nflog-size 0" is used, ulogd-2.0.5 crashes.

If "--nflog-size" is used with size 1 or greater, ulogd-2.0.5 is fine.

> > I'm surprised to see a kernel change cause unexpected userspace segfaults,
> > so further investigation into a kernel fix would seem a good idea.
> 
> According to the original user's manual, nflog-range option was
> designed to be the number of bytes copied to userspace, but
> unfortunately there's a bug from the beginning and it never works,
> i.e. in kernel, it just ignored this option.
> 
> Try to change the current nflog-range option's semantics may
> cause unexpected results(maybe like this ulogd crash) ...
> 
> In order to keep compatibility, Vishwanath introduce a new
> nflog-size option and keep nflog-range unchanged. If you just
> upgrade the kernel, and do not change iptables rules, this
> problem will not happen.

I am reporting that the problem does happen simply with an upgrade to 
kernel 4.8 and no other changes.  When "--nflog-size" is unspecified or 
set to 0, the bug in ulogd-2.0.5 gets triggered.

I agree there is a bug in ulogd-2.0.5 that this kernel change exposed, but 
I am trying to explain that all ulogd users risk this segfault if they 
upgrade to kernel 4.8 and don't either update to a fixed ulogd (possibly 
using your patch below) or an unreleased iptables with iptables config 
changes to implement nflog-size on each NFLOG target.

> So I think this is ulogd's bug, in _interp_iphdr, it try to
> dereference the iphdr pointer before validation check, meanwhile
> this problem does not exist in ipv6 path.  Can you try this patch:
> 
> diff --git a/filter/raw2packet/ulogd_raw2packet_BASE.c
> b/filter/raw2packet/ulogd_raw2packet_BASE.c
> index 8a6180c..fd2665a 100644
> --- a/filter/raw2packet/ulogd_raw2packet_BASE.c
> +++ b/filter/raw2packet/ulogd_raw2packet_BASE.c
> @@ -717,7 +717,7 @@ static int _interp_iphdr(struct ulogd_pluginstance
> *pi, uint32_t len)
> struct ulogd_key *ret = pi->output.keys;
> struct iphdr *iph =
> ikey_get_ptr(>input.keys[INKEY_RAW_PCKT]);
> -   void *nexthdr = (uint32_t *)iph + iph->ihl;
> +   void *nexthdr;
> 
> if (len < sizeof(struct iphdr) || len <= (uint32_t)(iph->ihl * 4))
> return ULOGD_IRET_OK;
> @@ -734,6 +734,7 @@ static int _interp_iphdr(struct ulogd_pluginstance
> *pi, uint32_t len)
> okey_set_u16([KEY_IP_ID], ntohs(iph->id));
> okey_set_u16([KEY_IP_FRAGOFF], ntohs(iph->frag_off));
> 
> +   nexthdr = (uint32_t *)iph + iph->ihl;
> switch (iph->protocol) {
> case IPPROTO_TCP:
> _interp_tcp(pi, nexthdr, len);

I agree this will likely fix ulogd, but this misses the point about the 
new kernel defaulting to a zero size return when it used to return the 
packet.

Thanks,
Chris

Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Chris Caputo]

On Mon, 10 Oct 2016, Liping Zhang wrote:
> 2016-10-10 15:02 GMT+08:00 Chris Caputo :
> >   Program received signal SIGSEGV, Segmentation fault.
> >   0x765fd18a in _interp_iphdr (pi=0x617f50, len=0) at 
> > ulogd_raw2packet_BASE.c:720
> >
> >   715 static int _interp_iphdr(struct ulogd_pluginstance *pi, uint32_t 
> > len)
> >   716 {
> >   717 struct ulogd_key *ret = pi->output.keys;
> >   718 struct iphdr *iph =
> >   719 ikey_get_ptr(>input.keys[INKEY_RAW_PCKT]);
> >   720 void *nexthdr = (uint32_t *)iph + iph->ihl;
> >
> > I believe 7643507fe8b5bd8ab7522f6a81058cc1209d2585 changed previous
> > behavior by not always copying IP header data to user space.
> >
> > On my machine IPv4 log packets result in a ulogd segfault while IPv6
> > packets do not.  I'm not sure of the cause of the difference.
> >
> > The corresponding userspace commit for the 209d2585 kernel change is:
> >
> >   
> > https://git.netfilter.org/iptables/commit/?id=7070b1f3c88a0c3d4e315c00cca61f05b0fbc882
> >
> > This adds --nflog-size to iptables.  When --nflog-size is used with my
> > iptables NFLOG lines, the ulogd-2.0.5 segfaults cease.
> 
> What numbers did you specify after --nflog-size option?
> --nflog-size 0 or ...? If you want log the whole packet to
> the ulogd, please do not specify this nflog-size option.

Not specifying nflog-size does not appear to log the whole packet...

If "--nflog-size" is unspecified, and the iptables config is left 
unchanged when the kernel is upgraded to 4.8, ulogd-2.0.5 crashes.

If "--nflog-size 0" is used, ulogd-2.0.5 crashes.

If "--nflog-size" is used with size 1 or greater, ulogd-2.0.5 is fine.

> > I'm surprised to see a kernel change cause unexpected userspace segfaults,
> > so further investigation into a kernel fix would seem a good idea.
> 
> According to the original user's manual, nflog-range option was
> designed to be the number of bytes copied to userspace, but
> unfortunately there's a bug from the beginning and it never works,
> i.e. in kernel, it just ignored this option.
> 
> Try to change the current nflog-range option's semantics may
> cause unexpected results(maybe like this ulogd crash) ...
> 
> In order to keep compatibility, Vishwanath introduce a new
> nflog-size option and keep nflog-range unchanged. If you just
> upgrade the kernel, and do not change iptables rules, this
> problem will not happen.

I am reporting that the problem does happen simply with an upgrade to 
kernel 4.8 and no other changes.  When "--nflog-size" is unspecified or 
set to 0, the bug in ulogd-2.0.5 gets triggered.

I agree there is a bug in ulogd-2.0.5 that this kernel change exposed, but 
I am trying to explain that all ulogd users risk this segfault if they 
upgrade to kernel 4.8 and don't either update to a fixed ulogd (possibly 
using your patch below) or an unreleased iptables with iptables config 
changes to implement nflog-size on each NFLOG target.

> So I think this is ulogd's bug, in _interp_iphdr, it try to
> dereference the iphdr pointer before validation check, meanwhile
> this problem does not exist in ipv6 path.  Can you try this patch:
> 
> diff --git a/filter/raw2packet/ulogd_raw2packet_BASE.c
> b/filter/raw2packet/ulogd_raw2packet_BASE.c
> index 8a6180c..fd2665a 100644
> --- a/filter/raw2packet/ulogd_raw2packet_BASE.c
> +++ b/filter/raw2packet/ulogd_raw2packet_BASE.c
> @@ -717,7 +717,7 @@ static int _interp_iphdr(struct ulogd_pluginstance
> *pi, uint32_t len)
> struct ulogd_key *ret = pi->output.keys;
> struct iphdr *iph =
> ikey_get_ptr(>input.keys[INKEY_RAW_PCKT]);
> -   void *nexthdr = (uint32_t *)iph + iph->ihl;
> +   void *nexthdr;
> 
> if (len < sizeof(struct iphdr) || len <= (uint32_t)(iph->ihl * 4))
> return ULOGD_IRET_OK;
> @@ -734,6 +734,7 @@ static int _interp_iphdr(struct ulogd_pluginstance
> *pi, uint32_t len)
> okey_set_u16([KEY_IP_ID], ntohs(iph->id));
> okey_set_u16([KEY_IP_FRAGOFF], ntohs(iph->frag_off));
> 
> +   nexthdr = (uint32_t *)iph + iph->ihl;
> switch (iph->protocol) {
> case IPPROTO_TCP:
> _interp_tcp(pi, nexthdr, len);

I agree this will likely fix ulogd, but this misses the point about the 
new kernel defaulting to a zero size return when it used to return the 
packet.

Thanks,
Chris

Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Liping Zhang]

Hi Chris,

2016-10-10 15:02 GMT+08:00 Chris Caputo :
> On Tue, 4 Oct 2016, Justin Piszcz wrote:
>> kernel 4.8 with ulogd-2.0.5- IPs are no longer logged:
>>
>> Oct  4 17:51:30 atom INPUT_BLOCK IN=eth1 OUT=
>> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
>> TTL=0 ID=0 PROTO=0 MARK=0
>> Oct  4 17:51:31 atom INPUT_BLOCK IN=eth1 OUT=
>> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
>> TTL=0 ID=0 PROTO=0 MARK=0
>> Oct  4 17:51:32 atom INPUT_BLOCK IN=eth1 OUT=
>> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
>> TTL=0 ID=0 PROTO=0 MARK=0
>>
>> (reboot back to kernel 4.7, works fine)
>>
>> kernel 4.7 with ulogd-2.0.5:
>> Oct  4 17:56:44 atom INPUT_BLOCK IN=eth1 OUT=
>> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 SRC=74.125.22.125
>> DST=1.2.3.4 LEN=397 TOS=00 PREC=0x00 TTL=48 ID=58093 PROTO=TCP
>> SPT=5222 DPT=19804 SEQ=2032644254 ACK=2273184383 WINDOW=55272 ACK PSH
>> URGP=0 MARK=0
>> Oct  4 17:56:45 atom INPUT_BLOCK IN=eth1 OUT=
>> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 SRC=74.125.22.125
>> DST=1.2.3.4 LEN=397 TOS=00 PREC=0x00 TTL=48 ID=58725 PROTO=TCP
>> SPT=5222 DPT=19804 SEQ=2032644254 ACK=2273184383 WINDOW=55272 ACK PSH
>> URGP=0 MARK=0
>>
>> Looks like there were some changes in the 4.8 kernel regarding ulogd,
>> has anyone else run into this problem?
>
> For me, kernel 4.8.1 results in segfaults in ulogd-2.0.5 at:
>
>   Program received signal SIGSEGV, Segmentation fault.
>   0x765fd18a in _interp_iphdr (pi=0x617f50, len=0) at 
> ulogd_raw2packet_BASE.c:720
>
>   715 static int _interp_iphdr(struct ulogd_pluginstance *pi, uint32_t 
> len)
>   716 {
>   717 struct ulogd_key *ret = pi->output.keys;
>   718 struct iphdr *iph =
>   719 ikey_get_ptr(>input.keys[INKEY_RAW_PCKT]);
>   720 void *nexthdr = (uint32_t *)iph + iph->ihl;
>
> I believe 7643507fe8b5bd8ab7522f6a81058cc1209d2585 changed previous
> behavior by not always copying IP header data to user space.
>
> On my machine IPv4 log packets result in a ulogd segfault while IPv6
> packets do not.  I'm not sure of the cause of the difference.
>
> The corresponding userspace commit for the 209d2585 kernel change is:
>
>   
> https://git.netfilter.org/iptables/commit/?id=7070b1f3c88a0c3d4e315c00cca61f05b0fbc882
>
> This adds --nflog-size to iptables.  When --nflog-size is used with my
> iptables NFLOG lines, the ulogd-2.0.5 segfaults cease.

What numbers did you specify after --nflog-size option?
--nflog-size 0 or ...? If you want log the whole packet to
the ulogd, please do not specify this nflog-size option.

>
> I'm surprised to see a kernel change cause unexpected userspace segfaults,
> so further investigation into a kernel fix would seem a good idea.

According to the original user's manual, nflog-range option was
designed to be the number of bytes copied to userspace, but
unfortunately there's a bug from the beginning and it never works,
i.e. in kernel, it just ignored this option.

Try to change the current nflog-range option's semantics may
cause unexpected results(maybe like this ulogd crash) ...

In order to keep compatibility, Vishwanath introduce a new
nflog-size option and keep nflog-range unchanged. If you just
upgrade the kernel, and do not change iptables rules, this
problem will not happen.

So I think this is ulogd's bug, in _interp_iphdr, it try to
dereference the iphdr pointer before validation check, meanwhile
this problem does not exist in ipv6 path.  Can you try this patch:

diff --git a/filter/raw2packet/ulogd_raw2packet_BASE.c
b/filter/raw2packet/ulogd_raw2packet_BASE.c
index 8a6180c..fd2665a 100644
--- a/filter/raw2packet/ulogd_raw2packet_BASE.c
+++ b/filter/raw2packet/ulogd_raw2packet_BASE.c
@@ -717,7 +717,7 @@ static int _interp_iphdr(struct ulogd_pluginstance
*pi, uint32_t len)
struct ulogd_key *ret = pi->output.keys;
struct iphdr *iph =
ikey_get_ptr(>input.keys[INKEY_RAW_PCKT]);
-   void *nexthdr = (uint32_t *)iph + iph->ihl;
+   void *nexthdr;

if (len < sizeof(struct iphdr) || len <= (uint32_t)(iph->ihl * 4))
return ULOGD_IRET_OK;
@@ -734,6 +734,7 @@ static int _interp_iphdr(struct ulogd_pluginstance
*pi, uint32_t len)
okey_set_u16([KEY_IP_ID], ntohs(iph->id));
okey_set_u16([KEY_IP_FRAGOFF], ntohs(iph->frag_off));

+   nexthdr = (uint32_t *)iph + iph->ihl;
switch (iph->protocol) {
case IPPROTO_TCP:
_interp_tcp(pi, nexthdr, len);

Thanks

> Having to add the likes of "--nflog-size 200" (200 simply being what I am
> using) to every NFLOG line in firewall configs is a significant burden for
> many.
>
> Putting out a new release of iptables may help ease this transition if the
> kernel is not patched to fix this.  I had to use the git code since 1.6.0
> doesn't have it.
>
> Chris

Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Liping Zhang]

Hi Chris,

2016-10-10 15:02 GMT+08:00 Chris Caputo :
> On Tue, 4 Oct 2016, Justin Piszcz wrote:
>> kernel 4.8 with ulogd-2.0.5- IPs are no longer logged:
>>
>> Oct  4 17:51:30 atom INPUT_BLOCK IN=eth1 OUT=
>> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
>> TTL=0 ID=0 PROTO=0 MARK=0
>> Oct  4 17:51:31 atom INPUT_BLOCK IN=eth1 OUT=
>> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
>> TTL=0 ID=0 PROTO=0 MARK=0
>> Oct  4 17:51:32 atom INPUT_BLOCK IN=eth1 OUT=
>> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
>> TTL=0 ID=0 PROTO=0 MARK=0
>>
>> (reboot back to kernel 4.7, works fine)
>>
>> kernel 4.7 with ulogd-2.0.5:
>> Oct  4 17:56:44 atom INPUT_BLOCK IN=eth1 OUT=
>> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 SRC=74.125.22.125
>> DST=1.2.3.4 LEN=397 TOS=00 PREC=0x00 TTL=48 ID=58093 PROTO=TCP
>> SPT=5222 DPT=19804 SEQ=2032644254 ACK=2273184383 WINDOW=55272 ACK PSH
>> URGP=0 MARK=0
>> Oct  4 17:56:45 atom INPUT_BLOCK IN=eth1 OUT=
>> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 SRC=74.125.22.125
>> DST=1.2.3.4 LEN=397 TOS=00 PREC=0x00 TTL=48 ID=58725 PROTO=TCP
>> SPT=5222 DPT=19804 SEQ=2032644254 ACK=2273184383 WINDOW=55272 ACK PSH
>> URGP=0 MARK=0
>>
>> Looks like there were some changes in the 4.8 kernel regarding ulogd,
>> has anyone else run into this problem?
>
> For me, kernel 4.8.1 results in segfaults in ulogd-2.0.5 at:
>
>   Program received signal SIGSEGV, Segmentation fault.
>   0x765fd18a in _interp_iphdr (pi=0x617f50, len=0) at 
> ulogd_raw2packet_BASE.c:720
>
>   715 static int _interp_iphdr(struct ulogd_pluginstance *pi, uint32_t 
> len)
>   716 {
>   717 struct ulogd_key *ret = pi->output.keys;
>   718 struct iphdr *iph =
>   719 ikey_get_ptr(>input.keys[INKEY_RAW_PCKT]);
>   720 void *nexthdr = (uint32_t *)iph + iph->ihl;
>
> I believe 7643507fe8b5bd8ab7522f6a81058cc1209d2585 changed previous
> behavior by not always copying IP header data to user space.
>
> On my machine IPv4 log packets result in a ulogd segfault while IPv6
> packets do not.  I'm not sure of the cause of the difference.
>
> The corresponding userspace commit for the 209d2585 kernel change is:
>
>   
> https://git.netfilter.org/iptables/commit/?id=7070b1f3c88a0c3d4e315c00cca61f05b0fbc882
>
> This adds --nflog-size to iptables.  When --nflog-size is used with my
> iptables NFLOG lines, the ulogd-2.0.5 segfaults cease.

What numbers did you specify after --nflog-size option?
--nflog-size 0 or ...? If you want log the whole packet to
the ulogd, please do not specify this nflog-size option.

>
> I'm surprised to see a kernel change cause unexpected userspace segfaults,
> so further investigation into a kernel fix would seem a good idea.

According to the original user's manual, nflog-range option was
designed to be the number of bytes copied to userspace, but
unfortunately there's a bug from the beginning and it never works,
i.e. in kernel, it just ignored this option.

Try to change the current nflog-range option's semantics may
cause unexpected results(maybe like this ulogd crash) ...

In order to keep compatibility, Vishwanath introduce a new
nflog-size option and keep nflog-range unchanged. If you just
upgrade the kernel, and do not change iptables rules, this
problem will not happen.

So I think this is ulogd's bug, in _interp_iphdr, it try to
dereference the iphdr pointer before validation check, meanwhile
this problem does not exist in ipv6 path.  Can you try this patch:

diff --git a/filter/raw2packet/ulogd_raw2packet_BASE.c
b/filter/raw2packet/ulogd_raw2packet_BASE.c
index 8a6180c..fd2665a 100644
--- a/filter/raw2packet/ulogd_raw2packet_BASE.c
+++ b/filter/raw2packet/ulogd_raw2packet_BASE.c
@@ -717,7 +717,7 @@ static int _interp_iphdr(struct ulogd_pluginstance
*pi, uint32_t len)
struct ulogd_key *ret = pi->output.keys;
struct iphdr *iph =
ikey_get_ptr(>input.keys[INKEY_RAW_PCKT]);
-   void *nexthdr = (uint32_t *)iph + iph->ihl;
+   void *nexthdr;

if (len < sizeof(struct iphdr) || len <= (uint32_t)(iph->ihl * 4))
return ULOGD_IRET_OK;
@@ -734,6 +734,7 @@ static int _interp_iphdr(struct ulogd_pluginstance
*pi, uint32_t len)
okey_set_u16([KEY_IP_ID], ntohs(iph->id));
okey_set_u16([KEY_IP_FRAGOFF], ntohs(iph->frag_off));

+   nexthdr = (uint32_t *)iph + iph->ihl;
switch (iph->protocol) {
case IPPROTO_TCP:
_interp_tcp(pi, nexthdr, len);

Thanks

> Having to add the likes of "--nflog-size 200" (200 simply being what I am
> using) to every NFLOG line in firewall configs is a significant burden for
> many.
>
> Putting out a new release of iptables may help ease this transition if the
> kernel is not patched to fix this.  I had to use the git code since 1.6.0
> doesn't have it.
>
> Chris

Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Chris Caputo]

On Tue, 4 Oct 2016, Justin Piszcz wrote:
> kernel 4.8 with ulogd-2.0.5- IPs are no longer logged:
> 
> Oct  4 17:51:30 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
> TTL=0 ID=0 PROTO=0 MARK=0
> Oct  4 17:51:31 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
> TTL=0 ID=0 PROTO=0 MARK=0
> Oct  4 17:51:32 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
> TTL=0 ID=0 PROTO=0 MARK=0
> 
> (reboot back to kernel 4.7, works fine)
> 
> kernel 4.7 with ulogd-2.0.5:
> Oct  4 17:56:44 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 SRC=74.125.22.125
> DST=1.2.3.4 LEN=397 TOS=00 PREC=0x00 TTL=48 ID=58093 PROTO=TCP
> SPT=5222 DPT=19804 SEQ=2032644254 ACK=2273184383 WINDOW=55272 ACK PSH
> URGP=0 MARK=0
> Oct  4 17:56:45 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 SRC=74.125.22.125
> DST=1.2.3.4 LEN=397 TOS=00 PREC=0x00 TTL=48 ID=58725 PROTO=TCP
> SPT=5222 DPT=19804 SEQ=2032644254 ACK=2273184383 WINDOW=55272 ACK PSH
> URGP=0 MARK=0
> 
> Looks like there were some changes in the 4.8 kernel regarding ulogd,
> has anyone else run into this problem?

For me, kernel 4.8.1 results in segfaults in ulogd-2.0.5 at:

  Program received signal SIGSEGV, Segmentation fault.
  0x765fd18a in _interp_iphdr (pi=0x617f50, len=0) at 
ulogd_raw2packet_BASE.c:720

  715 static int _interp_iphdr(struct ulogd_pluginstance *pi, uint32_t len)
  716 {
  717 struct ulogd_key *ret = pi->output.keys;
  718 struct iphdr *iph =
  719 ikey_get_ptr(>input.keys[INKEY_RAW_PCKT]);
  720 void *nexthdr = (uint32_t *)iph + iph->ihl;

I believe 7643507fe8b5bd8ab7522f6a81058cc1209d2585 changed previous 
behavior by not always copying IP header data to user space.

On my machine IPv4 log packets result in a ulogd segfault while IPv6 
packets do not.  I'm not sure of the cause of the difference.

The corresponding userspace commit for the 209d2585 kernel change is:

  
https://git.netfilter.org/iptables/commit/?id=7070b1f3c88a0c3d4e315c00cca61f05b0fbc882

This adds --nflog-size to iptables.  When --nflog-size is used with my 
iptables NFLOG lines, the ulogd-2.0.5 segfaults cease.

I'm surprised to see a kernel change cause unexpected userspace segfaults, 
so further investigation into a kernel fix would seem a good idea.  
Having to add the likes of "--nflog-size 200" (200 simply being what I am 
using) to every NFLOG line in firewall configs is a significant burden for 
many.

Putting out a new release of iptables may help ease this transition if the 
kernel is not patched to fix this.  I had to use the git code since 1.6.0 
doesn't have it.

Chris

Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Chris Caputo]

On Tue, 4 Oct 2016, Justin Piszcz wrote:
> kernel 4.8 with ulogd-2.0.5- IPs are no longer logged:
> 
> Oct  4 17:51:30 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
> TTL=0 ID=0 PROTO=0 MARK=0
> Oct  4 17:51:31 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
> TTL=0 ID=0 PROTO=0 MARK=0
> Oct  4 17:51:32 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
> TTL=0 ID=0 PROTO=0 MARK=0
> 
> (reboot back to kernel 4.7, works fine)
> 
> kernel 4.7 with ulogd-2.0.5:
> Oct  4 17:56:44 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 SRC=74.125.22.125
> DST=1.2.3.4 LEN=397 TOS=00 PREC=0x00 TTL=48 ID=58093 PROTO=TCP
> SPT=5222 DPT=19804 SEQ=2032644254 ACK=2273184383 WINDOW=55272 ACK PSH
> URGP=0 MARK=0
> Oct  4 17:56:45 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 SRC=74.125.22.125
> DST=1.2.3.4 LEN=397 TOS=00 PREC=0x00 TTL=48 ID=58725 PROTO=TCP
> SPT=5222 DPT=19804 SEQ=2032644254 ACK=2273184383 WINDOW=55272 ACK PSH
> URGP=0 MARK=0
> 
> Looks like there were some changes in the 4.8 kernel regarding ulogd,
> has anyone else run into this problem?

For me, kernel 4.8.1 results in segfaults in ulogd-2.0.5 at:

  Program received signal SIGSEGV, Segmentation fault.
  0x765fd18a in _interp_iphdr (pi=0x617f50, len=0) at 
ulogd_raw2packet_BASE.c:720

  715 static int _interp_iphdr(struct ulogd_pluginstance *pi, uint32_t len)
  716 {
  717 struct ulogd_key *ret = pi->output.keys;
  718 struct iphdr *iph =
  719 ikey_get_ptr(>input.keys[INKEY_RAW_PCKT]);
  720 void *nexthdr = (uint32_t *)iph + iph->ihl;

I believe 7643507fe8b5bd8ab7522f6a81058cc1209d2585 changed previous 
behavior by not always copying IP header data to user space.

On my machine IPv4 log packets result in a ulogd segfault while IPv6 
packets do not.  I'm not sure of the cause of the difference.

The corresponding userspace commit for the 209d2585 kernel change is:

  
https://git.netfilter.org/iptables/commit/?id=7070b1f3c88a0c3d4e315c00cca61f05b0fbc882

This adds --nflog-size to iptables.  When --nflog-size is used with my 
iptables NFLOG lines, the ulogd-2.0.5 segfaults cease.

I'm surprised to see a kernel change cause unexpected userspace segfaults, 
so further investigation into a kernel fix would seem a good idea.  
Having to add the likes of "--nflog-size 200" (200 simply being what I am 
using) to every NFLOG line in firewall configs is a significant burden for 
many.

Putting out a new release of iptables may help ease this transition if the 
kernel is not patched to fix this.  I had to use the git code since 1.6.0 
doesn't have it.

Chris

Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Justin Piszcz]

On Tue, Oct 4, 2016 at 8:58 PM, Liping Zhang  wrote:
> Hi Justin,
>
> 2016-10-05 6:02 GMT+08:00 Justin Piszcz :
>> Hello,
>>

[ .. ]

>
> Which one are you using? iptables or nftables?
# iptables -V
iptables v1.6.0

>
> Could you please paste the related iptables/nftables rules here?

Rules:

  $IPTABLES -N INPUT_BLOCK
  $IPTABLES -A INPUT_BLOCK -i eth1 -j NFLOG --nflog-group 2
--nflog-prefix "INPUT_BLOCK"
  $IPTABLES -A INPUT_BLOCK -j DROP
  $IPTABLES -A INPUT -j INPUT_BLOCK

Justin.

Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Justin Piszcz]

On Tue, Oct 4, 2016 at 8:58 PM, Liping Zhang  wrote:
> Hi Justin,
>
> 2016-10-05 6:02 GMT+08:00 Justin Piszcz :
>> Hello,
>>

[ .. ]

>
> Which one are you using? iptables or nftables?
# iptables -V
iptables v1.6.0

>
> Could you please paste the related iptables/nftables rules here?

Rules:

  $IPTABLES -N INPUT_BLOCK
  $IPTABLES -A INPUT_BLOCK -i eth1 -j NFLOG --nflog-group 2
--nflog-prefix "INPUT_BLOCK"
  $IPTABLES -A INPUT_BLOCK -j DROP
  $IPTABLES -A INPUT -j INPUT_BLOCK

Justin.

Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Liping Zhang]

Hi Justin,

2016-10-05 6:02 GMT+08:00 Justin Piszcz :
> Hello,
>
> kernel 4.8 with ulogd-2.0.5- IPs are no longer logged:
>
> Oct  4 17:51:30 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
> TTL=0 ID=0 PROTO=0 MARK=0
> Oct  4 17:51:31 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
> TTL=0 ID=0 PROTO=0 MARK=0
> Oct  4 17:51:32 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
> TTL=0 ID=0 PROTO=0 MARK=0
>
> (reboot back to kernel 4.7, works fine)
>
> kernel 4.7 with ulogd-2.0.5:
> Oct  4 17:56:44 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 SRC=74.125.22.125
> DST=1.2.3.4 LEN=397 TOS=00 PREC=0x00 TTL=48 ID=58093 PROTO=TCP
> SPT=5222 DPT=19804 SEQ=2032644254 ACK=2273184383 WINDOW=55272 ACK PSH
> URGP=0 MARK=0
> Oct  4 17:56:45 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 SRC=74.125.22.125
> DST=1.2.3.4 LEN=397 TOS=00 PREC=0x00 TTL=48 ID=58725 PROTO=TCP
> SPT=5222 DPT=19804 SEQ=2032644254 ACK=2273184383 WINDOW=55272 ACK PSH
> URGP=0 MARK=0
>
> Looks like there were some changes in the 4.8 kernel regarding ulogd,
> has anyone else run into this problem?
>
> } ulog;
> +   if ((li->u.ulog.flags & NF_LOG_F_COPY_LEN) &&
> +   (li->u.ulog.copy_len < data_len))
> +   data_len = li->u.ulog.copy_len;
> li->u.ulog.group = ntohs(nla_get_be16(tb[NFTA_LOG_GROUP]));
> +   li->u.ulog.flags |= NF_LOG_F_COPY_LEN;
> li->u.ulog.copy_len =
> if (nla_put_be16(skb, NFTA_LOG_GROUP, 
> htons(li->u.ulog.group)))
> -   if (li->u.ulog.copy_len) {
> +   if (li->u.ulog.flags & NF_LOG_F_COPY_LEN) {
>  htonl(li->u.ulog.copy_len)))
> li.u.ulog.group  = info->group;
> li.u.ulog.qthreshold = info->threshold;
> +   li.u.ulog.flags |= NF_LOG_F_COPY_LEN;
>
> Thanks,
>
> Justin.

Which one are you using? iptables or nftables?

Could you please paste the related iptables/nftables rules here?

Re: kernel v4.8: iptables logs are truncated with the 4.8 kernel?

[Liping Zhang]

Hi Justin,

2016-10-05 6:02 GMT+08:00 Justin Piszcz :
> Hello,
>
> kernel 4.8 with ulogd-2.0.5- IPs are no longer logged:
>
> Oct  4 17:51:30 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
> TTL=0 ID=0 PROTO=0 MARK=0
> Oct  4 17:51:31 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
> TTL=0 ID=0 PROTO=0 MARK=0
> Oct  4 17:51:32 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 LEN=0 TOS=00 PREC=0x00
> TTL=0 ID=0 PROTO=0 MARK=0
>
> (reboot back to kernel 4.7, works fine)
>
> kernel 4.7 with ulogd-2.0.5:
> Oct  4 17:56:44 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 SRC=74.125.22.125
> DST=1.2.3.4 LEN=397 TOS=00 PREC=0x00 TTL=48 ID=58093 PROTO=TCP
> SPT=5222 DPT=19804 SEQ=2032644254 ACK=2273184383 WINDOW=55272 ACK PSH
> URGP=0 MARK=0
> Oct  4 17:56:45 atom INPUT_BLOCK IN=eth1 OUT=
> MAC=00:1b:21:9c:3b:fa:3e:94:d5:d2:49:1e:08:00 SRC=74.125.22.125
> DST=1.2.3.4 LEN=397 TOS=00 PREC=0x00 TTL=48 ID=58725 PROTO=TCP
> SPT=5222 DPT=19804 SEQ=2032644254 ACK=2273184383 WINDOW=55272 ACK PSH
> URGP=0 MARK=0
>
> Looks like there were some changes in the 4.8 kernel regarding ulogd,
> has anyone else run into this problem?
>
> } ulog;
> +   if ((li->u.ulog.flags & NF_LOG_F_COPY_LEN) &&
> +   (li->u.ulog.copy_len < data_len))
> +   data_len = li->u.ulog.copy_len;
> li->u.ulog.group = ntohs(nla_get_be16(tb[NFTA_LOG_GROUP]));
> +   li->u.ulog.flags |= NF_LOG_F_COPY_LEN;
> li->u.ulog.copy_len =
> if (nla_put_be16(skb, NFTA_LOG_GROUP, 
> htons(li->u.ulog.group)))
> -   if (li->u.ulog.copy_len) {
> +   if (li->u.ulog.flags & NF_LOG_F_COPY_LEN) {
>  htonl(li->u.ulog.copy_len)))
> li.u.ulog.group  = info->group;
> li.u.ulog.qthreshold = info->threshold;
> +   li.u.ulog.flags |= NF_LOG_F_COPY_LEN;
>
> Thanks,
>
> Justin.

Which one are you using? iptables or nftables?

Could you please paste the related iptables/nftables rules here?