Re: Significant missing item in 11.0 release notes

[Kevin Oberman]

Thanks for the quick fix, Andrey! Now that this is taken care of, time to
start playing with the cool new features... especially naming tables.

Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkober...@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683

On Mon, Aug 1, 2016 at 10:02 AM, Ian Smith  wrote:

> On Mon, 1 Aug 2016 18:47:37 +0300, Andrey V. Elsukov wrote:
>  > On 01.08.16 18:43, Ian Smith wrote:
>  > > Fast work Andrey, and sorry for rushing in.  I ASSumed, after reading
>  > > the new tables section in 11.0-R ipfw(8), that Kevin had run into:
>  > >
>  > >Tables require explicit creation via create before use.
>  > >
>  > > but diving - not too deeply - into the log of /head/sbin/ipfw/tables.c
>  > > from your commit, I think that statement must be out of date, at least
>  > > regarding existing ruleset table configuration?  Is that right?
>  >
>  > If you want to use some new specific feature you need to create table
>  > explicitly. But for old rules generic tables will be created
>  > automatically (with warning).
>
> Exactly how I was hoped it would work, thankyou ..
>
> cheers, Ian
>
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: Significant missing item in 11.0 release notes

[Ian Smith]

On Mon, 1 Aug 2016 18:47:37 +0300, Andrey V. Elsukov wrote:
 > On 01.08.16 18:43, Ian Smith wrote:
 > > Fast work Andrey, and sorry for rushing in.  I ASSumed, after reading 
 > > the new tables section in 11.0-R ipfw(8), that Kevin had run into:
 > > 
 > >Tables require explicit creation via create before use.
 > > 
 > > but diving - not too deeply - into the log of /head/sbin/ipfw/tables.c 
 > > from your commit, I think that statement must be out of date, at least 
 > > regarding existing ruleset table configuration?  Is that right?
 > 
 > If you want to use some new specific feature you need to create table
 > explicitly. But for old rules generic tables will be created
 > automatically (with warning).

Exactly how I was hoped it would work, thankyou ..

cheers, Ian
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: Significant missing item in 11.0 release notes

[Andrey V. Elsukov]

On 01.08.16 18:43, Ian Smith wrote:
> Fast work Andrey, and sorry for rushing in.  I ASSumed, after reading 
> the new tables section in 11.0-R ipfw(8), that Kevin had run into:
> 
>   Tables require explicit creation via create before use.
> 
> but diving - not too deeply - into the log of /head/sbin/ipfw/tables.c 
> from your commit, I think that statement must be out of date, at least 
> regarding existing ruleset table configuration?  Is that right?

If you want to use some new specific feature you need to create table
explicitly. But for old rules generic tables will be created
automatically (with warning).

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature

Re: Significant missing item in 11.0 release notes

[Ian Smith]

On Mon, 1 Aug 2016 16:39:45 +0300, Andrey V. Elsukov wrote:
 > On 31.07.16 22:28, Kevin Oberman wrote:
 > > I assumed that I had missed this in the release notes, but I can find no
 > > reference to this significant change that simultaneously greatly enhanced
 > > ipfw table functionality, but also broke my configuration. While the fix
 > > was trivial, if the Release Notes had addressed this, I would not have had
 > > the problem in the first place.
 > 
 > I fixed this in r303615. Thanks for the report!

Fast work Andrey, and sorry for rushing in.  I ASSumed, after reading 
the new tables section in 11.0-R ipfw(8), that Kevin had run into:

Tables require explicit creation via create before use.

but diving - not too deeply - into the log of /head/sbin/ipfw/tables.c 
from your commit, I think that statement must be out of date, at least 
regarding existing ruleset table configuration?  Is that right?

cheers, Ian
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Julian Elischer]

On 1/08/2016 7:16 PM, Dr. Rolf Jansen wrote:

Am 01.08.2016 um 03:17 schrieb Julian Elischer :
On 30/07/2016 10:17 PM, Dr. Rolf Jansen wrote:

I finished the work on CIDR conformity of the IP ranges tables generated by the 
tool geoip. The main constraint is that the start and end address of an IP 
block given by the delegation files MUST BE PRESERVED during the transformation 
to a set of CIDR records. This target is achieved by:

  1. Finding the largest common netmask boundary of the start address utilizing
 int(log2(addr_count)); then iteration like Euclid's algorithm in computing
 a GCD.

  2. Output the CIDR with the given start address and the masklen belonging
 to the found netmask.

  3. If the CIDR does not match the whole original IP range then set the start
 address of the next CIDR block to the next boundary of the common netmask,
 and loop over starting at 1. until the original range has been satisfied.


check out the appletalk code I pointed out  to you.. I wrote that in 93 or so 
but I remember sweating blood
over it to get it right.

I read the description of the code and the following sentence made me suspicious that 
aa_dorangeroute() would guarantee the above mentioned main constraint  "start and 
end address of an IP block given by the delegation files MUST BE PRESERVED" can be 
matched. Start/end address are said to be anything (even undefined) but fixed in the 
description

So you get your data as ranges (i.e. a start and an end address)


...
Split the range into two subranges such that the middle
of the two ranges is the point where the highest bit of difference
between the two addresses makes its transition.
...

I do not want this.

I think I may explain it better by example..

Any range of addresses can conceptually be broken into an optimal 
range of binary chunks.
for example, for the range 1. 62   we have an optimal set of 
binary subranges:

 1-1  2-3  4-7 8-15 16-31 (*) 32-47 48-55 56-59 60-61 and 62-62
there is always a point (*)  where we separate the left hand side from 
the right hand side.  You can also only merge the two items either 
side of the (*) point, and even then, only if they have the same 
width. Ranges not immediately adjacent to the (*) point can never be 
merged into any other range.
In some ranges, there are no items to the right, and for some ranges 
there are no items to the left. There may also be skipped ranges. The 
above example is "worst case".  All 10 subranges are present, and only 
2 can be merged, leading to needing 9 binary subranges to correctly 
express the whole range.
So the term "middle" is misleading.  It however allows you to take an 
arbitrary range of addresses and generate the optimal set of cidr 
addresses.
In this case if you had 10.0.1.0 through 10.0.62.255 you would need 9 
cidr addresses to express the range correctly (the middle two can be 
joined).

That is what the code referred to does.

if you have exceptions as you do in routing tables,  you can express 
it as 3 cidr addresses..  0-64  MINUS 0 and MINUS 63.
i.e 10.0.0.0/18 minus 10.0.0.0/24 minus 10.0.63.0/24. However this is 
probably not of interest to your use case.
It however IS possible to do this in ipfw since it allows overlapping 
ranges of different widths. (you will note that the code pointed to 
does NOT do that. it would be an order of magnitude (or more) harder 
to do.)





I carefully tested the algorithm and a table that I pipe by the new geoip tool 
into ipfw is 100 % identical to the output of the ipfw command 'table N list'.

though that doesn't mean it is semantically identical to the original table due to 
'most specific rule wins" behaviour.

for example:
if you type in ;

1.2.3.0/24 -> A
and
1.2.3.0/26 -> B
then both rules will be listed the same as what you put in
but if you wanted to get all rules that point to A, without having rules that 
point to B, then you would have to export
1.2.3.64/26  -> A
1.2.3.128/25 -> A
  (i.e. TWO rules)

This is definitely not the usage case. The origin of the data to be passed to 
ipfw tables are RIR delegation statistics files, which is guaranteed to be 
consolidated, namely resolved overlaps and joined adjacencies, long before any 
tables for ipfw are generated. Each range entry got a well defined, i.e. fixed, 
i.e non-variable starting address, and anything that changes the starting 
address of the ranges renders the table useless. Every entry got a well defined 
range length, and that one also must not be changed, or the table would be 
useless as well.

I think you are misunderstanding what I meant.


In addition, we are talking about automatic generation of thousands of entries, 
and I never ever won't rely on something like 'most specific rule wins' 
behaviour, I want the behaviour as explicit as possible, and for this reason I 
am happy with 'INPUT is 100 % identical to the OUTPUT'.


I agree. I was saying that getting the same rules out of 

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 01.08.2016 um 03:17 schrieb Julian Elischer :
> On 30/07/2016 10:17 PM, Dr. Rolf Jansen wrote:
>> I finished the work on CIDR conformity of the IP ranges tables generated by 
>> the tool geoip. The main constraint is that the start and end address of an 
>> IP block given by the delegation files MUST BE PRESERVED during the 
>> transformation to a set of CIDR records. This target is achieved by:
>> 
>>  1. Finding the largest common netmask boundary of the start address 
>> utilizing
>> int(log2(addr_count)); then iteration like Euclid's algorithm in 
>> computing
>> a GCD.
>> 
>>  2. Output the CIDR with the given start address and the masklen belonging
>> to the found netmask.
>> 
>>  3. If the CIDR does not match the whole original IP range then set the start
>> address of the next CIDR block to the next boundary of the common 
>> netmask,
>> and loop over starting at 1. until the original range has been satisfied.
>> 
> check out the appletalk code I pointed out  to you.. I wrote that in 93 or so 
> but I remember sweating blood
> over it to get it right.

I read the description of the code and the following sentence made me 
suspicious that aa_dorangeroute() would guarantee the above mentioned main 
constraint  "start and end address of an IP block given by the delegation files 
MUST BE PRESERVED" can be matched. Start/end address are said to be anything 
(even undefined) but fixed in the description.

   ... 
   Split the range into two subranges such that the middle
   of the two ranges is the point where the highest bit of difference
   between the two addresses makes its transition.
   ...

I do not want this.

>> I carefully tested the algorithm and a table that I pipe by the new geoip 
>> tool into ipfw is 100 % identical to the output of the ipfw command 'table N 
>> list'.
> though that doesn't mean it is semantically identical to the original table 
> due to 'most specific rule wins" behaviour.
> 
> for example:
> if you type in ;
> 
> 1.2.3.0/24 -> A
> and
> 1.2.3.0/26 -> B
> then both rules will be listed the same as what you put in
> but if you wanted to get all rules that point to A, without having rules that 
> point to B, then you would have to export
> 1.2.3.64/26  -> A
> 1.2.3.128/25 -> A
>  (i.e. TWO rules)

This is definitely not the usage case. The origin of the data to be passed to 
ipfw tables are RIR delegation statistics files, which is guaranteed to be 
consolidated, namely resolved overlaps and joined adjacencies, long before any 
tables for ipfw are generated. Each range entry got a well defined, i.e. fixed, 
i.e non-variable starting address, and anything that changes the starting 
address of the ranges renders the table useless. Every entry got a well defined 
range length, and that one also must not be changed, or the table would be 
useless as well.

In addition, we are talking about automatic generation of thousands of entries, 
and I never ever won't rely on something like 'most specific rule wins' 
behaviour, I want the behaviour as explicit as possible, and for this reason I 
am happy with 'INPUT is 100 % identical to the OUTPUT'.

> you could also export
> 1.2.3.0/24 -> A
> 1.2.3.0/26 -> 0  (think of it as an "EXCEPT for these" rule)
> 
> which is ALSO two rules but you would need to be sure that the receiver knows 
> what to do with them.

This is simply a ridiculous example in the given respect, this sounds like you 
are suggesting fuzzying the input data in order to bring ipfw to its limits. 
This makes life less boring, doesn't it? No thanks.

>> It is worth to note, that already the original RIR delegation files contain 
>> 457 non CIDR conforming IPv4 ranges in a total of 165815 original records. I 
>> guess that this number will increase in the future because the RIR's ran 
>> empty on new IPv4 ranges and are urged to subdivide returned old ranges for 
>> new delegations. The above algorithm is ready for this.
>> 
>> Generally, CIDR conforming tables are more than twice as large as optimized 
>> (joined adjacencies) IP range tables. All said changes have been pushed to 
>> GitHup already.
>> 
> Unfortunately there is no way to specify (using cidr notation) a.b.1.x AND 
> a.b.2.x without including a.b.[03].x.
> 
> HOWEVER
> if you specified the FULL table you could use the "except" feature of routing 
> table behaviour where
> a.b.0.x/22  -> A
> a.b.0.x/24  -> B
> a.b.3.x/24  -> B
> gives you the same thing because of the 'most specific rule wins" nature of 
> routing table evaluation.
> I believe this is the case in the tables you imported.
> the trick is to be able to take an "optimised" table such as that above and 
> produce, given a required subset, just the required part, while changing the 
> rules as needed on the fly to "de-optimise" them enough to maintain 
> correctness.

Again, this is not the usage case.

>> I am still a little bit amazed how ipfw come to accept incorrect CIDR ranges 
>> and 

Re: Significant missing item in 11.0 release notes

[Ian Smith]

On Sun, 31 Jul 2016 12:28:06 -0700, Kevin Oberman wrote:

 > This morning I updated my min user system from 10.3-Stable to 11.0-BETA3.
 > In general, things went well, but I had two issues that prevented the
 > network from operating. the first is a lack of documentation in the Release
 > Notes and the second is a driver issue. Since they are in no way related,
 > I'll send the report of the driver issue later.
 > 
 > I use ipfw(8) tables in my firewall configuration. Unfortunately, 11.0 has
 > introduced a totally re-worked tables structure. The new structure is
 > awesome and I read about it at the time the changes were being planned and
 > implemented, but had forgotten. As a result the very first line in my
 > configuration, "table 1 flush" was no longer valid and the remainder of the
 > file was ignored.
 > 
 > I assumed that I had missed this in the release notes, but I can find no
 > reference to this significant change that simultaneously greatly enhanced
 > ipfw table functionality, but also broke my configuration. While the fix
 > was trivial, if the Release Notes had addressed this, I would not have had
 > the problem in the first place.

I don't see this as a Release Notes issue - though I guess it will be if 
it cannot be quickly fixed before 11.0-RELEASE - but as a very serious 
and -  as far as I know - unreported regression in ipfw(8).

In 18 years I cannot recall any addition of features, or additional 
options for existing features, that caused any breakage of existing 
rulesets.  What on earth could be invalid about "table 1 flush"?

cc'ing ipfw@, which is most likely where this should be discussed ..

cheers, Ian
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Julian Elischer]

On 30/07/2016 10:17 PM, Dr. Rolf Jansen wrote:

Am 29.07.2016 um 10:23 schrieb Dr. Rolf Jansen :

Am 29.07.2016 um 06:50 schrieb Julian Elischer :
On 29/07/2016 5:22 PM, Julian Elischer wrote:

On 29/07/2016 4:53 PM, Dr. Rolf Jansen wrote:

Am 28.07.2016 um 23:48 schrieb Lee Brown :

That makes sense to me.  Your /20 range encompasses 201.222.16.0 -
201.222.31.255.
If you want 201.222.20.0-201.222.31.255, you'll need 3 ranges:

201.222.20.0/22 (201.222.20.0-201.222.23.255)
201.222.24.0/22 (201.222.24.0-201.222.27.255)
201.222.28.0/22 (201.222.28.0-201.222.31.255)

Ian, Julian and Lee,

Thank you vary much for your responses. In order not bloat the thread, I answer 
only to one message.

I found the problem. As a matter of fact, the respective IP ranges in the 
LACNIC delegation statistics file are 3 adjacent blocks with 1024 addresses, 
i.e. those that you listed in your message above:

$grep 201.222.2 /usr/local/etc/ipdb/IPRanges/lacnic.dat
lacnic|BR|ipv4|201.222.20.0|1024|20140710|allocated|164725
lacnic|BR|ipv4|201.222.24.0|1024|20140630|allocated|138376
lacnic|BR|ipv4|201.222.28.0|1024|20140701|allocated|129095

However, my database compilation combines adjacent blocks with the same country 
code, and the ranges above turn into one block of 3072 addresses, which 
obviously doesn't have a valid netmask - log(3072) = 11,5849625.
...
..., it is not sufficient to forget about optimization but I need to check also 
whether, the delegation files contain already some non-CIDR ranges, which need 
to be broken down.

there is code to take ranges and produce cidr sets.

We used to have exactly that code in the appletalk code before we took it out. 
Appletalk uses ranges.
https://svnweb.freebsd.org/base/release/3.2.0/sys/netatalk/at_control.c?view=annotate#l703

though htat uassumes input in the form af an appletak sockaddr..
there is also this python module
https://pythonhosted.org/netaddr/tutorial_01.html#support-for-non-standard-address-ranges


maybe you can find other versions on the net.
however if you fully populate the table, you will get the correct result 
because more specific entries will
override less specific entries. To do that you would have to have a way to 
describe to your program what
value each table entry should output.
If you did what you do now, then you would specify the value for the required countries, 
and give a default falue for "all others".
aggregation of adjacent ranges with same value would be an optimisation.

Don't worry, breaking down an arbitrary IP-range into a CIDR conforming set of 
ranges, doesn't seem too difficult. ...
...
Once I come to a conclusion, I will post it to this mailing list.

I finished the work on CIDR conformity of the IP ranges tables generated by the 
tool geoip. The main constraint is that the start and end address of an IP 
block given by the delegation files MUST BE PRESERVED during the transformation 
to a set of CIDR records. This target is achieved by:

  1. Finding the largest common netmask boundary of the start address utilizing
 int(log2(addr_count)); then iteration like Euclid's algorithm in computing
 a GCD.

  2. Output the CIDR with the given start address and the masklen belonging
 to the found netmask.

  3. If the CIDR does not match the whole original IP range then set the start
 address of the next CIDR block to the next boundary of the common netmask,
 and loop over starting at 1. until the original range has been satisfied.


check out the appletalk code I pointed out  to you.. I wrote that in 
93 or so but I remember sweating blood

over it to get it right.


I carefully tested the algorithm and a table that I pipe by the new geoip tool 
into ipfw is 100 % identical to the output of the ipfw command 'table N list'.
though that doesn't mean it is semantically identical to the original 
table due to 'most specific rule wins" behaviour.


for example:
if you type in ;

1.2.3.0/24 -> A
and
1.2.3.0/26 -> B
then both rules will be listed the same as what you put in
but if you wanted to get all rules that point to A, without having 
rules that point to B, then you would have to export

1.2.3.64/26  -> A
1.2.3.128/25 -> A
 (i.e. TWO rules)

you could also export
1.2.3.0/24 -> A
1.2.3.0/26 -> 0  (think of it as an "EXCEPT for these" rule)

which is ALSO two rules but you would need to be sure that the 
receiver knows what to do with them.




It is worth to note, that already the original RIR delegation files contain 457 
non CIDR conforming IPv4 ranges in a total of 165815 original records. I guess 
that this number will increase in the future because the RIR's ran empty on new 
IPv4 ranges and are urged to subdivide returned old ranges for new delegations. 
The above algorithm is ready for this.

Generally, CIDR conforming tables are more than twice as large as optimized 
(joined adjacencies) IP range tables. All said changes have been pushed to 
GitHup 

Re: ipfw divert filter for IPv4 geo-blocking

[Julian Elischer]

On 30/07/2016 10:17 PM, Dr. Rolf Jansen wrote:


I am still a little bit amazed how ipfw come to accept incorrect CIDR ranges 
and arbitrarily moves the start/end addresses in order to achieve CIDR 
conformity, and that without any further notice, and that given that ipfw can 
be considered as being quite relevant to system security. Or, may I assume that 
ipfw knows always better than the user what should be allowed or denied. 
Otherwise, perhaps I am the only one ever who input incorrect CIDR ranges for 
processing by ipfw.
it's not so amazing when you think about it. The code comes from the 
routing table..


In this context  a.b.c.d/N means "the range of addresses containing 
a.b.c.d, masked to a length of N".  there is no specification that 
a.b.c.d is the first address of the range.  I have relied upon this 
behaviour many times.




Best regards

Rolf

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"



___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"