Re: npf bug(?)

[Robert Elz]

Date:Thu, 13 Apr 2017 08:13:12 +0200 (CEST)
From:6b...@6bone.informatik.uni-leipzig.de
Message-ID:  


Further source analysis confirms that for netstat, the "bad" fragments
are not counted as fragments received, but that (as expected) a fragment
which completes a packet is counted as both a fragment received, and a
packet reassembled (unlike with npf).

Further it is clear that ...

  | There is again a difference between the npf counters and the counters of 
  | the IP stack.

(side from them counting different things - slightly) it is clear that this
cannot happen, as the way it works (it appears to me) is ...

packet arrives at interface
npf examines packet
npf determines packet is fragmented
calls IPv4 reassembly routine for IPv4 packets
(drops or passes through IPv6 fragments ... didn't check)
if packet is now complete, pass to IP
otherwise, done and wait for another frag

IP gets the packet
tests if it is fragmemted.
answer must be "no" or npf would not have let the packet through.
processing continues...

That is, the IP reassembly stats come from when npf reassembles the
packet (reassembly is not done twice) and so whatever is seen by the
IP frag counters is coming from npf's reaassembly process.

This also explains npf's lack of a "timed out" counter - npf never does
that, it is invoked from an incoming packet, and that's it ... fragments
hang about in the regular IP reassembly queues, and the normal IP
reassembly timer runs on them, the 33 packets that failed to be completed
in the IP stats, also failed to be completed in npf's reassembly (as they
are the same thing) but npf never knows that happened, and so cannot
implement a counter of its own.

This is still all *currently anyway) just v4 of course...

kre

ps: there is one (apparent) place in npf where an incoming fragment is
counted nowhere - that's where the packet reassembly completes, but
turns out to still be a fragment.   That is, there's some disagreement
somewhere in the code whether reassembly succeeded or not.   That case
should probably be a panic() rather than a "return EINVAL".

Re: npf bug(?)

[Robert Elz]

Date:Thu, 13 Apr 2017 08:13:12 +0200 (CEST)
From:6b...@6bone.informatik.uni-leipzig.de
Message-ID:  


  | npf:
  | 
  | Fragmentation:
  |  870 fragments

I thought I should look at the sources...  that counter is slightly
different than what I had guessed it might be, what's being counted
there is the number of times a fragment is received which does not
complete a packet - that is, none of first/last/middle is it, just the
number of times npf ends up waiting on another packet arriving, and ...

  |  102 failed reassembly

those are not included in the 870, rather this counts the number of
incoming fragmented packets that couldn't be reassembled for some
reason (and counts both v4 & v6 if both are being passed to npf ... all
incoming v6 fragments will be counted here with npf currently, as it
does not attempt to reassemble v6 packets).

  |  774 reassembled

counts the number of incoming frags that did successfully complete a
packet - so the total number of incoming fragments is the sum of
those three, which is 1746.

If we were then to also assume that in ...

  | netstat:
  |  1644 fragments received
  |  102 malformed fragments dropped

"fragments received" means "valid fragments received", then we also have
1746 received (valid + invalid) fragments.

That makes the npf and netstat numbers agree precisely.

It does however cause a question about

  |  33 fragments dropped after timeout

from netstat, which don't seem to be counted anywhere else.

kre

Re: npf bug(?)

[Robert Elz]

Date:Thu, 13 Apr 2017 08:13:12 +0200 (CEST)
From:6b...@6bone.informatik.uni-leipzig.de
Message-ID:  



  | npf:
  | 
  | Fragmentation:
  |  870 fragments
  |  774 reassembled
  |  102 failed reassembly
  | 
  | netstat:
  |  1644 fragments received
  |  0 fragments dropped (dup or out of space)
  |  0 fragments dropped (out of ipqent)
  |  102 malformed fragments dropped
  |  33 fragments dropped after timeout
  |  774 packets reassembled ok
  | 
  | There is again a difference between the npf counters and the counters of 
  | the IP stack.

Not a lot - perhaps not any - it just looks to me as if they're counting
slightly different things.

Both agree that 774 packets were reassembled.   To be reassembled at least
2 incoming fragments need to be joined, so we can expect that at least 1548
fragments need to have been received.

But npf says only 870 fragments received - considering all of that info
together tells me that npf is either only counting initial fragments (offset 0)
or only considering non-initial fragments (offset != 0) or one of the
two similar cases with the last fragment (M != 0 or M == 0).   That is.
in each completed packet, one (at least) of the incoming fragments is not
counted in the npf "fragments" counter.

On the other hand. with the netstat stats (the IP level data) it looks
obvious that all fragments (initial, last, and any in the middle) are
being counted as fragments received.

Now from netstat we see 1644 frags were received, and 135 were dropped
(102 considered "malformed" and 33 after a timeout - which would mean, I
assume, that the companion fragment never appeared - it can also mean that
one of the frags was duplicated in transit (though that is rare) and while
the packet was reassembled from one of the duplicated pair, and the other
fragment from the original packet, the other of the duplicated pair never
receives its companion (both fragments from one initial packet being duplicated
in the network would be too improbable to believe)).   Take the (minimum)
1548 fragments that were joined into packets and the 135 that were not,
and we should have received (at least) 1548+135 == 1683 frags, but apparently
we only got 1644.   The only real explanation for that is that something is
being counted twice (or less likely, that the incoming fragments count is
missing some fragments.)   Note that even if the 33 timed out frags were
also counted as malformed, 1548+102 == 1650 which is more than 1644.

The npf stats show a similar anomaly - aside from agreeing that 102 of the
received frags failed reassembly (presumably the same 102 that netstat
reported were malformed -- and apparently not revealing anything that
failed reassembly as a companion never showed up), there were 774 packets
reassembled, and 102 that weren't, which should mean at least 876 fragments
received, but npf counts only 870.   Here there are other possible explanations
for the discrepancy though, some of the malformed fragments counted might
have been whichever fragment npf failed to count as a "fragment" in its
count of fragments, or npf might be counting only initial, or final, fragments
in its "fragments" count, and there could have been a few packets fragmented
into more than 2 fragments.

It is perphaps interesting that the 1650 (netstat) frags that we should have
received (to produce 774 packets while dropping 102 frags) is 6 more than
the count of frags actually received, so is the 876 that npf should have
counted (774 + 102) 6 more than its actual count.

That the difference is 6 in both cases indicates (to me, as a probability
anyway) that the same counting "issue" is happening in both netstat and
npf, but again this makes it look more as if (aside from the counters
counting different things, which is OK if it is understood) that both
are seeing (and counting) the same fragments/packets.

While these oddities in the counting probably ought be explained, there's
nothing really different in what the two of them are reporting happened,
both agree that 774 packets were eventually produced from recombining
fragments, and that 102 fragments apparently had some defect which prevented
reasembly.

Perhaps if npf happens to be counting only initial, or final, packets in
its "fragments" counter, it could be reported as "fragmented packets"
rather than "fragments" in the npf ouput.All its numbers would be
consistent (or at least might be) if that is the case, 870 fragmented
packets received, 774 successfully, 102 incoming frags dropped (of which
it is possible that some were from the same initially fragmented packet,
probably 6, as 870-774 == 96, so 96 packets actually failed to be
reassembled - under this hypothesis - with 102 frags dropped, the extra
6 dropped frags would have been duplicates, or extra frags from packets
that failed reassembly).

Re: npf bug(?)

[6bone]

On Mon, 10 Apr 2017, Christos Zoulas wrote:


Npf just calls "error = ip_reass_packet(mp, ip)" and if that fails
it increments NPF_STAT_REASSFAIL. Since it uses the same exact
call the regular ip stack uses, I would expect that:

   NPF_STAT_REASSFAIL = IP_STAT_BADFRAGS + IP_STAT_RCVMEMDROP;


From your stats I see (without npf):


1795 fragments received

 335 malformed fragments dropped
 440 fragments dropped after timeout
 636 packets reassembled ok
  ---
 1411 what happened to the rest?

I guess we should look at the code to figure out if we are not printing
some, or we and not updating the status for some. Nevertheless, the stack
seems to be able to reassemble a bit more 1/3 of the fragments, while 2/3
of them are dropped. What's the ratio with npf?



Sorry for the long response time. I could not restart the router. Now runs 
the kernel without the patch. I have made sure that all counter at 0 are 
started.


After a few hours the statistics look as follows:

npf:

Fragmentation:
870 fragments
774 reassembled
102 failed reassembly

netstat:
1644 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped (out of ipqent)
102 malformed fragments dropped
33 fragments dropped after timeout
774 packets reassembled ok

There is again a difference between the npf counters and the counters of 
the IP stack.



Regards
Uwe

Re: npf bug(?)

[Christos Zoulas]

On Apr 9, 10:53pm, 6b...@6bone.informatik.uni-leipzig.de 
(6b...@6bone.informatik.uni-leipzig.de) wrote:
-- Subject: Re: npf bug(?)

| On Sun, 9 Apr 2017, Christos Zoulas wrote:
| 
| > Perhaps you get a lot of dup fragments? netstat -s should show you the
| > stack's reassembly and fragment stats. Perhaps those agree with what
| > npf shows?
| 
| Currently the patch is active. That's why I have no npf statistics. The 
| netstat statistics seem to me credible.
| 
| If npf checks the fragmentation, then the counters of npf and the ip stack 
| run parallel? Or are the ip stack only counted the packets the npf leaves?

Npf just calls "error = ip_reass_packet(mp, ip)" and if that fails
it increments NPF_STAT_REASSFAIL. Since it uses the same exact
call the regular ip stack uses, I would expect that:

NPF_STAT_REASSFAIL = IP_STAT_BADFRAGS + IP_STAT_RCVMEMDROP;

>From your stats I see (without npf):

 1795 fragments received

  335 malformed fragments dropped
  440 fragments dropped after timeout
  636 packets reassembled ok
  ---
 1411 what happened to the rest?

I guess we should look at the code to figure out if we are not printing
some, or we and not updating the status for some. Nevertheless, the stack
seems to be able to reassemble a bit more 1/3 of the fragments, while 2/3
of them are dropped. What's the ratio with npf?

christos

Re: npf bug(?)

[6bone]

On Sun, 9 Apr 2017, Christos Zoulas wrote:


Perhaps you get a lot of dup fragments? netstat -s should show you the
stack's reassembly and fragment stats. Perhaps those agree with what
npf shows?


Currently the patch is active. That's why I have no npf statistics. The 
netstat statistics seem to me credible.


If npf checks the fragmentation, then the counters of npf and the ip stack 
run parallel? Or are the ip stack only counted the packets the npf leaves?


netstat -s shows:

ip:
413339977 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with length > max ip packet size
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
1795 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped (out of ipqent)
335 malformed fragments dropped
440 fragments dropped after timeout
636 packets reassembled ok
410154493 packets for this host
35 packets for unknown/unsupported protocol
10 packets forwarded (6 packets fast forwarded)
3183945 packets not forwardable
0 redirects sent
0 packets no matching gif found
218900862 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
31819922 output datagrams fragmented
33505129 fragments created
0 datagrams that can't be fragmented
0 datagrams with bad address in header


Regards
Uwe

Re: npf bug(?)

[6bone]

On Thu, 6 Apr 2017, Christos Zoulas wrote:



| Thanks for the patch. For me it works very well.

Well, the question is do you need it? I.e. why don't you let the v4
traffic flow through npf? Is it a performance issue or doesn't npf
do reassembly correctly?


I'm not sure if the reassembling works properly. After a few hours the 
statistics showed over 7000 failed reassembly but only 1296 fragments 
with 1104 reassembled. I find these numbers unusual. I also know no way to 
check if the reassembly works correctly. So I would like to disable packet 
reassembly.




| Is this a special solution just for me or will the patch be part of the
| current kernel.

Special, but we can consider adding the functionality in a more general
way if it is needed.



Regards
Uwe

Re: npf bug(?)

[Christos Zoulas]

On Apr 6, 10:04am, 6b...@6bone.informatik.uni-leipzig.de 
(6b...@6bone.informatik.uni-leipzig.de) wrote:
-- Subject: Re: npf bug(?)

| Thanks for the patch. For me it works very well.

Well, the question is do you need it? I.e. why don't you let the v4
traffic flow through npf? Is it a performance issue or doesn't npf
do reassembly correctly?

| Is this a special solution just for me or will the patch be part of the 
| current kernel.

Special, but we can consider adding the functionality in a more general
way if it is needed.

christos

Re: npf bug(?)

[6bone]

On Mon, 3 Apr 2017, Christos Zoulas wrote:



Here's a rough patch that kills v4 processing.

christos



Thanks for the patch. For me it works very well.

Is this a special solution just for me or will the patch be part of the 
current kernel.



Regards
Uwe

Re: npf bug(?)

[6bone]

On Sun, 2 Apr 2017, Christos Zoulas wrote:



I am trying to understand the use case here:
1. you want to have V4 DNS and 6to4 service that can generate V4 fragments
2. you want V4 fragments dropped.
3. you can't put V4 rules in your firewall to restrict traffic to only
  those services.

Is that correct?


That is not completely right. I want to filter IPv6 with npf. IPv4 should 
not be filtered. After the activation of npf the statistics shows:


Fragmentation:
1296 fragments
1104 reassembled
7160 failed reassembly

Since IPv6 is no longer reassambling, it must be IPv4 packets. I want to 
make sure that the reassembly errors do not lead to packet losses, 
especially at 6to4.



Regards
Uwe

Re: npf bug(?)

[Tom Ivar Helbekkmo]

6b...@6bone.informatik.uni-leipzig.de writes:

> Thanks for the patch. Unfortunately, I could not apply it to -7. I've
> tested -current. The problem seems to be solved.

Here it is, adjusted for the "netbsd-7" cvs tag:

--- sys/net/npf/npf_handler.c~  2017-03-30 17:26:49.458901595 +0200
+++ sys/net/npf/npf_handler.c   2017-03-30 17:29:52.833241529 +0200
@@ -146,7 +146,7 @@
npf_conn_t *con;
npf_rule_t *rl;
npf_rproc_t *rp;
-   int error, retfl;
+   int error, retfl, flags;
int decision;
 
/*
@@ -164,9 +164,17 @@
rp = NULL;
 
/* Cache everything.  Determine whether it is an IP fragment. */
-   if (__predict_false(npf_cache_all() & NPC_IPFRAG)) {
+   flags = npf_cache_all();
+   if (__predict_false(flags & NPC_IPFRAG)) {
/*
-* Pass to IPv4 or IPv6 reassembly mechanism.
+* We pass IPv6 fragments unconditionally
+* The first IPv6 fragment is not marked as such
+* and passes through the filter
+*/
+   if (flags & NPC_IP6)
+   return 0;
+   /*
+* Pass to IPv4 reassembly mechanism.
 */
error = npf_reassembly(, mp);
if (error) {
--- sys/net/npf/npf_inet.c~ 2017-03-30 17:27:07.661343255 +0200
+++ sys/net/npf/npf_inet.c  2017-03-30 17:30:45.721564537 +0200
@@ -352,6 +352,7 @@
case (IPV6_VERSION >> 4): {
struct ip6_hdr *ip6;
struct ip6_ext *ip6e;
+   struct ip6_frag *ip6f;
size_t off, hlen;
 
ip6 = nbuf_ensure_contig(nbuf, sizeof(struct ip6_hdr));
@@ -384,8 +385,21 @@
hlen = (ip6e->ip6e_len + 1) << 3;
break;
case IPPROTO_FRAGMENT:
+   ip6f = nbuf_ensure_contig(nbuf, sizeof(*ip6f));
+   if (ip6f == NULL)
+   return 0;
+   /*
+* We treat the first fragment as a regular
+* packet and then we pass the rest of the
+* fragments unconditionally. This way if
+* the first packet passes the rest will
+* be able to reassembled, if not they will
+* be ignored. We can do better later.
+*/
+   if (ntohs(ip6f->ip6f_offlg & IP6F_OFF_MASK) != 
0)
+   flags |= NPC_IPFRAG;
+
hlen = sizeof(struct ip6_frag);
-   flags |= NPC_IPFRAG;
break;
case IPPROTO_AH:
hlen = (ip6e->ip6e_len + 2) << 2;


-tih
-- 
Most people who graduate with CS degrees don't understand the significance
of Lisp.  Lisp is the most important idea in computer science.  --Alan Kay