Re: Question that has dogged me for a while.

[Dr. Rolf Jansen]

Am 05.05.2017 um 21:14 schrieb Karl Denninger <k...@denninger.net>:
> On 5/5/2017 19:08, Dr. Rolf Jansen wrote:
>> Am 05.05.2017 um 20:53 schrieb Karl Denninger <k...@denninger.net>:
>>> On 5/5/2017 14:33, Julian Elischer wrote:
>>>> On 5/5/17 1:48 am, Dr. Rolf Jansen wrote:
>>>>> Resolving this with ipfw/NAT may easily become quite complicated, if
>>>>> not impossible if you want to run a stateful nat'ting firewall, which
>>>>> is usually the better choice.
>>>>> 
>>>>> IMHO a DNS based solution is much more effective.
>>>>> 
>>>>> On my gateway I have running the caching DNS resolver Unbound. Now
>>>>> let's assume, the second level domain name in question is
>>>>> example.com, and your web server would be accessed by
>>>>> www.example.com, while other services, e.g. mail are served from
>>>>> other sites on the internet.
>>>> I believe this is a much cleaner solution thanusing double NAT.
>>>> (see also my solution for if the server is also freebsd)
>>>> even though we have a nice set of new IPFW capabilities that can do
>>>> this, I still think double nat is an over complication of the system.
>>>> 
>>> Well, the DNS answer is one that works IF you control the zone in
>>> question every time. ...
>> I do not understand "control the zone ... every time".
>> 
>> I set up my transparent zones 5 years ago and never touched it again, and I 
>> don't see any "illegal" packets on my network caused by this either.
>> 
>> I understand that you actually didn't grasp the transparent zone technic.
>> 
>> Happy double nat'ting :-D
> On the contrary I do understand it (and how to do it), along with how to
> throw "off-network" packets at the other host.  Both ways work (unbound
> is arguably simpler than BIND, but it'll work in both cases) but the
> point is that you then must keep two things in sync rather than do one
> thing in one place.

With BIND you cannot setup a selectively transparent zone. You are talking 
about split DNS, and that's a different animal.


___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: Question that has dogged me for a while.

[Dr. Rolf Jansen]

Am 05.05.2017 um 20:53 schrieb Karl Denninger <k...@denninger.net>:
> On 5/5/2017 14:33, Julian Elischer wrote:
>> On 5/5/17 1:48 am, Dr. Rolf Jansen wrote:
>>> Resolving this with ipfw/NAT may easily become quite complicated, if
>>> not impossible if you want to run a stateful nat'ting firewall, which
>>> is usually the better choice.
>>> 
>>> IMHO a DNS based solution is much more effective.
>>> 
>>> On my gateway I have running the caching DNS resolver Unbound. Now
>>> let's assume, the second level domain name in question is
>>> example.com, and your web server would be accessed by
>>> www.example.com, while other services, e.g. mail are served from
>>> other sites on the internet.
>> 
>> I believe this is a much cleaner solution thanusing double NAT.
>> (see also my solution for if the server is also freebsd)
>> even though we have a nice set of new IPFW capabilities that can do
>> this, I still think double nat is an over complication of the system.
>> 
> Well, the DNS answer is one that works IF you control the zone in
> question every time. ...


I do not understand "control the zone ... every time".

I set up my transparent zones 5 years ago and never touched it again, and I 
don't see any "illegal" packets on my network caused by this either.

I understand that you actually didn't grasp the transparent zone technic.

Happy double nat'ting :-D

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: IPFW problem with passing IPSEC through in-kernel NAT

[Dr. Rolf Jansen]

> Am 09.12.2016 um 02:11 schrieb Karl Denninger :
> ...
> Some more information on this issue I suspect that something is
> getting mangled somewhere in the IP stack, perhaps related to hardware
> checksumming or similar -- or in the ipfw code.

I had always ran into IPsec-NAT-UDP checksumming issues since I started working 
with FreeBSD, that tim v8.0. With a rather simple change in the respective 
kernel source file at least my issue can be resolved. This may be related to 
your issue or even not, anyway, I guess it is worth to give it a try.

I am now running FreeBSD 11-RELEASE-p5. On line 462 of file 
/usr/src/sys/netinet/udp_usrreq.c, I replaced:

if (uh->uh_sum) {

with:

if (uh->uh_sum &&
uh->uh_dport != htons(1701) &&
uh->uh_dport != htons(4500)) {

This effectively skips extended UDP checksumming for certain UDP ports -- here 
the L2TP and IPsec-NAT-T ports. When I investigated the issue, I found in one 
related RFC, that IPsec-NAT-T isn't supposed to do UDP checksumming on the 
encapsulated packets anyway, and my patch enforces this behaviour.

Best regards

Rolf
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: Notice on upcoming ipdbtools 1.1.1

[Dr. Rolf Jansen]

> Am 14.08.2016 um 12:15 schrieb Dr. Rolf Jansen <r...@obsigna.com>:
> 
> As was noticed by the port maintainer, the initial release of ipdbtools 1.1.0 
> into the ports did not compile on i386 systems because the lack of the 
> __uint128_t data type on 32bit systems, and which was used for IPv6 computing.
> 
> In the meantime, I rolled in the necessary uint128 comparison, shift and 
> basic arithmetic operations that provide the missing built-in __uint128_t 
> operations on 32bit systems. The 64bit targets x86-64 and arm64 continue to 
> utilize the built-in operations.
> 
> The changes are ready on GitHub, and I will submit a changed port PR on 
> Monday (tomorrow) night, most of the post-mortem fixes since the initial 
> release are included -- I won't rename the tool 'ipup', though.

I just submitted the PR for updating the port of ipdbtools to v1.1.1.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211881

Best regards

Rolf
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Notice on upcoming ipdbtools 1.1.1

[Dr. Rolf Jansen]

As was noticed by the port maintainer, the initial release of ipdbtools 1.1.0 
into the ports did not compile on i386 systems because the lack of the 
__uint128_t data type on 32bit systems, and which was used for IPv6 computing.

In the meantime, I rolled in the necessary uint128 comparison, shift and basic 
arithmetic operations that provide the missing built-in __uint128_t operations 
on 32bit systems. The 64bit targets x86-64 and arm64 continue to utilize the 
built-in operations.

The changes are ready on GitHub, and I will submit a changed port PR on Monday 
(tomorrow) night, most of the post-mortem fixes since the initial release are 
included -- I won't rename the tool 'ipup', though.

Best regards

Rolf

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: your thoughts on a particualar ipfw action.

[Dr. Rolf Jansen]

> Am 11.08.2016 um 14:20 schrieb Ian Smith <smi...@nimnet.asn.au>:
> On Thu, 11 Aug 2016 10:09:24 -0300, Dr. Rolf Jansen wrote:
>>> Am 11.08.2016 um 08:06 schrieb Ian Smith <smi...@nimnet.asn.au>:
>>> On Wed, 10 Aug 2016 -0300, Dr. Rolf Jansen wrote:
>>> ...
>>> ...
>>>> I just submitted a PR asking to add the new port 'sysutils/ipdbtools'.
>>>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211744
>>> 
>>> Wonderful.
>> 
>> The port maintainers were really quick. The port has been accepted 
>> and has been already committed.
> 
> So it has, on refreshing the page.  Smooth and fast.
> 
> Re __uint128_t I _guess_ there may be macro/s to do that maths for i386?

Yeah, I am exploring the options. Comparisons, addition and subtraction are 
working already, multiplication, division and remainder operations are a tad 
more difficult, I must leave this for some weekend.

>>> ...
>>> A more tech-savvy article than ABC or other news media managed so far:
>>> https://www.theguardian.com/australia-news/2016/aug/10/computer-says-no-australian-census-shambles-explanation-depends-on-who-you-ask
>> 
>> Well, I tend to believe that this has nothing to do with DoS attacks, 
> 
> Some should have been expected, planned for, mitigation anticipated, as 
> well as expecting at least 5 times the legit connections/hr they tested 
> for, and as the guardian article pointed to, their DNS was screwed in 
> several ways: way too long TTL (can't move fast), hard-coded subdomain 
> in SSL cert (couldn't readily add load-sharing capacity?) and such.
> 
> But they admit the geo-blocking fell over - whether inline as firewall 
> or on another server fielding lookup requests not disclosed - but they 
> say that failure caused a/the/some router to fail (crash? explode? :)

Perhaps they did Geo-blocking in the way that I mentioned in the summary of the 
ipdbtool's manual to be a no-go:

...
Unfortunately, online database look-up is by far too slow for even think-
ing about being utilized on the firewall level, where IP packets need to
be processed in a microsecond time scale. Therefore, a locally maintained
IP Geo-location database is indispensable in the given respect.
...

> IBM, FFS! but they'll point to govt specs and disclaim hardware failure 
> but still it's not great product endorsement for their SoftLayer Cloud.

Natural but non-professional reaction. My mother always told us, if you point
with your index finger to others, three fingers are pointing back to you.
So IBM not only failed technically but also the PR devision did a bad job. 

>> I mean, of course it is DoS, but not caused by an attack. Exactly the 
>> same happens every year on 30th of April between 17:00 and 24:00 on 
>> the servers of the Federal Bureau of Finance here in Brazil. That is 
>> the deadline for the online-submission of the annual tax declaration 
>> of the Brazilian citizens. Seems that the bureaucrats all over the 
>> world share the same deficiency of creative problem solving.
> 
> Seems it's a requirement for the job, world wide.  Creativity is scary, 
> but you think they could guess that ~8 million households in the eastern 
> timezone were going to have dinner then do their census within ~2 hours.

Of course they could not guess this, because public servants are trained
to assume that the normal citizen does not meet her/his obligations, and
for sure they were (are) prepared to send out 8 million penalty notices
in 24 hours.

>> Who in the bureaucrats hell told them to go with one deadline for 
>> everybody? For the census in Australia, I would have told the 
>> citizens that everybody got an individual deadline which is his or 
>> her birthday in 2016 -- problem solved.
> 
> That'd be great load-balancing .. shall I let them know? :)

Doesn't cost anything giving it a try, however, you could as well slap an
ox on his horn - same effect.

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: your thoughts on a particualar ipfw action.

[Dr. Rolf Jansen]

> Am 11.08.2016 um 08:06 schrieb Ian Smith <smi...@nimnet.asn.au>:
> On Wed, 10 Aug 2016 -0300, Dr. Rolf Jansen wrote:
> 
> (just curious: whereabouts is -0300?  Brazil?)

Yes, I am a German living in Brazil for more than 10 years now. BTW, your mail 
provider is blocking my mails, perhaps, because the origin is Brazil, but I am 
using a German provider for my mail transport.

>>> Am 08.08.2016 um 18:46 schrieb Dr. Rolf Jansen <r...@obsigna.com>:
>>> I am almost finished with preparing the tools for geo-blocking and 
>>> geo-routing at the firewall for submission to the FreeBSD ports.
> 
>>> I created a man file for the tools, see: 
>>> https://cyclaero.github.io/ipdb/, and I added the recent suggestions 
>>> on rule number/action code per country code, namely, I changed the 
>>> formula for the x-flag to the suggestion of Ian (value = offset + 
>>> ((C1 - 'A')*26 + (C2 - 'A'))*10), and I added the idea of directly 
>>> assigning a number to a country code in the argument for the t-flag 
>>> ("CC=n:...").  Furthermore, I removed the divert filter daemon 
>>> from the Makefile. The source is still on GitHub, though, and can be 
>>> re-vamped if necessary. Now I am going to prepare the Makefile for
>>> the port.
> 
> Terrific work, Rolf!  Something for everyone, although I'm guessing the 
> pf people are going to want a piece of the action, if they need any more 
> than the -p option and a bit of scripting.

It is not that much work, to add other output options. The main obstacle for me 
is, that I won't be able to test it carefully together with pf. So, it would be 
good to do this in cooperation with someone who got a well running pf firewall 
-- the same holds for other possible applications as well.

>> I just submitted a PR asking to add the new port 'sysutils/ipdbtools'.
>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211744
> 
> Wonderful.

The port maintainers were really quick. The port has been accepted and has been 
already committed.

>> I needed to change the name of the geoip tool, because GeoIP® is a
>> registered trademark of MaxMind, Inc., see www.maxmind.com. The name 
> 
> I did wonder about that ..
> 
>> of the tool is now 'ipup' = abbreviated form of IP geo location table 
>> generation and look- UP , that is without the boring middle part :-D
>> 
>> Those, who used geoip already in some scripts, please excuse the
>> inconvenience of needing to change the name.
> 
>> With the great help of Julian, I was able to improve the man file and
>> the latest version can be read online:
>> 
>>  https://cyclaero.github.io/ipdb/
> 
> Nice manual and all.  A few typos noted below (niggly Virgo proofreader)

I was tempted to get these last changes into my PR, but I am sorry, it was too 
late for the initial release. I committed the corrected man file to the GitHub 
repository, though, it will automatically go into the next release of the 
ipdbtools, perhaps together with some additions for using it together with 
pf(8) and route(8).

> I must apologise for added exasperation earlier.  I was tending towards 
> conflating several other ipfw issues under discussion (named states, new 
> state actions, and this).  Sorry if I bumped you off course momentarily, 
> though I don't seem to have slowed you down too much ..

Nothing, to be sorry about. I like discussions.

> As a hopefully not unwelcome aside, it's a pity that IBM, of all people, 
> couldn't manage geo-blocking successfully for the Australian Census the 
> other night.  Next time around we can offer them a working geo-blocking 
> firewall/router for a good deal less than the AU$9.6M we've paid IBM :)
> 
> Census: How the Government says the website meltdown unfolded:
> http://www.abc.net.au/news/2016-08-10/census-night-how-the-shambles-unfolded/7712964
> 
> A more tech-savvy article than ABC or other news media managed so far:
> https://www.theguardian.com/australia-news/2016/aug/10/computer-says-no-australian-census-shambles-explanation-depends-on-who-you-ask

Well, I tend to believe that this has nothing to do with DoS attacks, I mean, 
of course it is DoS, but not caused by an attack. Exactly the same happens 
every year on 30th of April between 17:00 and 24:00 on the servers of the 
Federal Bureau of Finance here in Brazil. That is the deadline for the 
online-submission of the annual tax declaration of the Brazilian citizens. 
Seems that the bureaucrats all over the world share the same deficiency of 
creative problem solving.

Who in the bureaucrats hell told them to go with one deadline for everybody? 
For the census in Australia, I would have told the citizens that everybody got 
an individual deadline which is his or 

Re: your thoughts on a particualar ipfw action.

[Dr. Rolf Jansen]

> Am 08.08.2016 um 18:46 schrieb Dr. Rolf Jansen <r...@obsigna.com>:
> 
> I am almost finished with preparing the tools for geo-blocking and 
> geo-routing at the firewall for submission to the FreeBSD ports.
> 
> I created a man file for the tools, see: https://cyclaero.github.io/ipdb/, 
> and I added the recent suggestions on rule number/action code per country 
> code, namely, I changed the formula for the x-flag to the suggestion of Ian 
> (value = offset + ((C1 - 'A')*26 + (C2 - 'A'))*10), and I added the idea of 
> directly assigning a number to a country code in the argument for the t-flag 
> ("CC=n:...").
> 
> Furthermore, I removed the divert filter daemon from the Makefile. The source 
> is still on GitHub, though, and can be re-vamped if necessary.
> 
> Now I am going to prepare the Makefile for the port.

I just submitted a PR asking to add the new port 'sysutils/ipdbtools'.

   https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211744

I needed to change the name of the geoip tool, because GeoIP® is a registered 
trademark of MaxMind, Inc., see www.maxmind.com. The name of the tool is now 
'ipup' = abbreviated form of  IP  geo location table generation and look- UP , 
that is without the boring middle part :-D

Those, who used geoip already in some scripts, please excuse the inconvenience 
of needing to change the name.

With the great help of Julian, I was able to improve the man file and the 
latest version can be read online:

   https://cyclaero.github.io/ipdb/

Best regards

Rolf

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: your thoughts on a particualar ipfw action.

[Dr. Rolf Jansen]

I am almost finished with preparing the tools for geo-blocking and geo-routing 
at the firewall for submission to the FreeBSD ports.

I created a man file for the tools, see: https://cyclaero.github.io/ipdb/, and 
I added the recent suggestions on rule number/action code per country code, 
namely, I changed the formula for the x-flag to the suggestion of Ian (value = 
offset + ((C1 - 'A')*26 + (C2 - 'A'))*10), and I added the idea of directly 
assigning a number to a country code in the argument for the t-flag 
("CC=n:...").

Furthermore, I removed the divert filter daemon from the Makefile. The source 
is still on GitHub, though, and can be re-vamped if necessary.

Now I am going to prepare the Makefile for the port.

In the meantime, please can a native English speaker look at said man file (s. 
link above)? I know, that my English is lacking, and any corrections would be 
highly welcome.

Best regards

Rolf

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: your thoughts on a particualar ipfw action.

[Dr. Rolf Jansen]

> Am 05.08.2016 um 02:44 schrieb Julian Elischer <jul...@freebsd.org>:
> On 5/08/2016 2:22 AM, Dr. Rolf Jansen wrote:
>> I am completely free of passions on this CC encoding thingy. I won't use 
>> this feature anyway. Please, may I suggest that the experts of the ipfw 
>> community come to an agreement, and I then I will change the implementation 
>> accordingly.
>> 
>> Another possibility could be to attach the desired rule numbers directly to 
>> the country codes in the argument of the -t option, How about:
>> 
>>   geoip -t AU=5:RU=50010:US=50020:BR=50030
>> 
>> The present behaviour would be kept without attached numbers. Please let me 
>> know your choices. Furthermore, if the new ipfw allows for more 
>> sophisticated table construction directives, that could be beneficial for 
>> country code based table processing, please advice.
> 
> I can hear the exasperation in your writing :-)

Not exactly 'exasperation'. Moving targets are always kind of unpleasant - at 
least for me, perhaps I am not a sufficiently patient hunter :-)

> I've lost track..
> Was the present behaviour just a single value? or a generated number with -x 
> offset? (not sure if you actually added that or just described it).

I meant, geoip would continue allowing a -t option argument without numbers, 
for example -t AU:RU:US:BR, and in that case it would continue with its present 
behaviour (mutually exclusive either -v XOR -x):

-v  # default 0
-x   # in val = (C1-60)*1000 + C2*10 + offset

The -v and -x options as above are already on GitHub, the x-formula can be 
changed quickly. 

> your "US=50020" idea is nice but a lot of work I think for  you.

Not that much work. I like this one as well, because this is the most explicit 
way, that I can imagine, of associating a rule/action number with a country 
code.

I figure, that any kind of x-formula let people shoot themselves in the foot at 
one day or the other. Imagine you set up your sophisticated rule set in 2016 
and in 2017 a colleague is asked by the boss to add another country code. The 
trouble may start by she/he forgets to add an action rule for the implicitly 
generated table argument, and it does not end with a possible violation of the 
implicitly reserved rule number range.

> I guess you would do it with script
> geoip -t US=${LINE_US} |ipfw -q /dev/stdin
> ipfw add ${LINE_US} drop all tcp from any to any 80
> ipfw add $((${LINE_US} + 1)) skipto ${FINISH_UP}

Yes, why not? The nice thing of the "CC=n:..." feature is, that it is 
already useful as is, and that it is open for any further sophistication with 
shell script magics.  

> probably in a shell function
> it would also allow you to put 'action numbers' rather than line numbers as 
> it doesn't  interpret the values, just passes them through.
> 
> On the other hand the same thing can be achieved by embedding geoip in a loop 
> in a script.
> I think we should just let you get on with your life and be happy with what 
> you have given us.  mapping a set of country codes to a number. I can always 
> make more complicated setups using that and 15 minutes of shell script.

Yes, for us this is not a big deal. However, once this stuff is in the ports, I 
have to take into account the work of answering questions from the 
non-enlightened folks, on how to achieve the mapping between rule numbers and 
country codes. Perhaps, it is less hassle to simply add the "CC=n:..." 
feature and move on.


BTW:

In the course of preparing the packet for the ports, I am working now on a man 
file for the geoip db utilities (geoip, ipdb and ipdb-update.sh). I want to put 
it to section 1 - General Commands, OK?

Shall I remove the geod ipfw divert filter daemon from the distribution for the 
ports?

My initial incentive was Geo-blocking. However, I learned from your VPN usage 
example (in your other message), that these utilities may quite well serve for 
other objectives. Now, I am looking for a package title that does suggest a 
wider range of applications.

How about:

  "Utilities for IP based Geo-blocking/routing with IPFW"


Best regards

Rolf

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: your thoughts on a particualar ipfw action.

[Dr. Rolf Jansen]

> Am 04.08.2016 um 13:44 schrieb Ian Smith <smi...@nimnet.asn.au>:
>> On Wed, 3 Aug 2016 18:53:38 -0300, Dr. Rolf Jansen wrote:
>>>> Am 03.08.2016 um 11:13 schrieb Julian Elischer <jul...@freebsd.org>:
> 
> 'scuse savage reformatting, but I had to wrap it to read it .. and pine 
> has completely mangled the quoting levels too, dunno why.
> 
>>>>> looking for thoughts from people who know the new IPFW features well..
> 
> That's not me, but I'm having fun reading 11.0-RELEASE ipfw(8) ..
> 
>>>>> A recent addition to our armory is the geoip program that, given an 
>>>>> address can tell you what country it is in and given a country code, 
>>>>> can give an ipfw table that describes all the ip addresses in that 
>>>>> country.
>>>>> 
>>>>> SO I was thinking how to use this, and the obvious way would be to 
>>>>> have a set of rules for each country, and use the "skipto tablearg" 
>>>>> facility to skip to the right rules for each country. But the
>>>>> trouble is that a tablearg skipto is very inefficient. It's also a 
>>>>> hard thing to set up with a set of rules for each country (how many 
>>>>> countries are there in the internet allocation system?).
> 
> Julian, have you looked into Andrey's LINEAR_SKIPTO ?  How does it work?  
> Are there any disadvantages?  And if not, why isn't it the default? :)
> 
> Also, There's a particularly useful example in new ipfw(8), showing how 
> to set and use multiple tablearg values - the example uses skipto,fib 
> with a setfib tablearg followed by a skipto tablearg both from the same 
> table entry, and you can use, among others - some fully documented, some 
> yet to catch up - dscp values (0..63) setting or testing, and notably 
> tags 1..65534, set or test, which goes some way towards 'variables' you 
> were hoping for, no? :)  Also some netgraph stuff I won't understand ..
> 
>>>> As of today a total of 236 country codes are in use for IPv4 
>>>> delegations. If this helps for anything, a command line switch to the 
>>>> geoip tool could be added for letting it output the country code (as the 
>>>> hex encoded CC taken as a plain decimal integer) as the value for the 
>>>> given table entry. In the moment you can give one value for all entries 
>>>> generated by geoip, with this switch set, the output of geoip could look 
>>>> like:
>>>> 
>>>> $ geoip -t "DE:BR:US" -x
>>>> ...
>>>> table 0 add 93.157.48.0/21 4445
>>>> table 0 add 93.158.236.0/22 4252
>>>> table 0 add 93.159.96.0/19 4445
>>>> table 0 add 93.159.248.0/21 4445
>>>> table 0 add 93.180.72.0/21 4445
>>>> table 0 add 93.180.152.0/21 4445
>>>> table 0 add 93.181.0.0/18 4445
>>>> table 0 add 93.183.0.0/18 5553
>>>> ...
>>>> 
>>>> Given that ...
>>>> 0x4445 = 'DE'
>>>> 0x4252 = 'BR'
>>>> 0x5553 = 'US'
>>> 
>>> well it would have to be the decimal value so DE would be 6869, as 
>>> while 4445 works 444F is a problem :-)
>> 
>> Yes, you're right, I was taken away by the wave of enthusiasm, 
>> before thinking about the subject up to the end.
>> 
>>> 0x444F would work but we waste even more address space.  You'd have to 
>>> multiply the numbers by some scaler, because adjacent numbers would not 
>>> be enough for one rule to do something and another rule to skip on to 
>>> somewhere else. (well, you could have multiple rules at the same number 
>>> but that has its limitations. > The idea would certainly work. it would 
>>> mean setting aside all the rules between 6565 and 9090 however. > A more 
>>> compact encoding could be something like
>>> 
>>> ((name[0]-'A') * 32)+(name[1]-'A')) multiplied by some 'step' (maybe 
>>> 10 by default) and offset by a given offset.
>>> 
>>> so AF (Afghanistan) would be the first 0*32+5 * 10 would give an 
>>> offset of 50.. plus a user supplied offset turns it into say, 15050..
>> 
>> I understand the reasons, however, any complicated encoding will 
>> defeat the idea of the value can be sort of numeric mnemonic for the 
>> country code ÿÿ well, so it is. I elaborated on your idea and came-up 
>> with the following formula:  val = (C1-60)*1000 + C2*10 + offset. The 
>> offset can be given as the parameter to the -x flag.
>> 
>> $ geoip -t "DE:BR:US" -4 -x 3
>> ...
>

Re: your thoughts on a particualar ipfw action.

[Dr. Rolf Jansen]

> Am 03.08.2016 um 11:13 schrieb Julian Elischer <jul...@freebsd.org>:
> 
> On 2/08/2016 8:50 PM, Dr. Rolf Jansen wrote:
>>> Am 02.08.2016 um 05:08 schrieb Julian Elischer <jul...@freebsd.org>:
>>> 
>>> looking for thoughts from people who know the new IPFW features well..
>>> 
>>> 
>>> A recent addition to our armory is the geoip program that, given an address 
>>> can tell you what country it is in and given a country code, can give an 
>>> ipfw table that describes all the ip addresses in that country.
>>> 
>>> SO I was thinking how to use this, and the obvious way would be to have a 
>>> set of rules for each country, and use the "skipto tablearg" facility to 
>>> skip to the right rules for each country. But the trouble is that a 
>>> tablearg skipto is very inefficient. It's also a hard thing to set up with 
>>> a set of rules for each country (how many countries are there in the 
>>> internet allocation system?).
>> 
>> As of today a total of 236 country codes are in use for IPv4 delegations. If 
>> this helps for anything, a command line switch to the geoip tool could be 
>> added for letting it output the country code (as the hex encoded CC taken as 
>> a plain decimal integer) as the value for the given table entry. In the 
>> moment you can give one value for all entries generated by geoip, with this 
>> switch set, the output of geoip could look like:
>> 
>> $ geoip -t "DE:BR:US" -x
>> ...
>> table 0 add 93.157.48.0/21 4445
>> table 0 add 93.158.236.0/22 4252
>> table 0 add 93.159.96.0/19 4445
>> table 0 add 93.159.248.0/21 4445
>> table 0 add 93.180.72.0/21 4445
>> table 0 add 93.180.152.0/21 4445
>> table 0 add 93.181.0.0/18 4445
>> table 0 add 93.183.0.0/18 5553
>> ...
>> 
>> Given that ...
>> 0x4445 = 'DE'
>> 0x4252 = 'BR'
>> 0x5553 = 'US'
> 
> well it would have to be the decimal value so DE would be 6869, as while 4445 
> works 444F is a problem :-)

Yes, you're right, I was taken away by the wave of enthusiasm, before thinking 
about the subject up to the end.

> 0x444F would work but we waste even more address space.  You'd have to 
> multiply the numbers by some scaler, because adjacent numbers would not be 
> enough for one rule to do something and another rule to skip on to somewhere 
> else. (well, you could have multiple rules at the same number but that has 
> its limitations.
> The idea would certainly work. it would mean setting aside all  the rules  
> between 6565 and 9090 however.
> A more compact encoding could be something like ((name[0]-'A') * 
> 32)+(name[1]-'A')) multiplied by some 'step' (maybe 10 by default) and offset 
> by a given offset.
> 
> so AF (Afghanistan) would be the first 0*32+5  * 10 would give an offset of 
> 50.. plus a user supplied offset turns it into say, 15050..

I understand the reasons, however, any complicated encoding will defeat the 
idea of the value can be sort of numeric mnemonic for the country code – well, 
so it is. I elaborated on your idea and came-up with the following formula:  
val = (C1-60)*1000 + C2*10 + offset. The offset can be given as the parameter 
to the -x flag.

$ geoip -t "DE:BR:US" -4 -x 3
...
table 0 add 93.157.48.0/21 38690
table 0 add 93.158.236.0/22 36820
table 0 add 93.159.96.0/19 38690
table 0 add 93.159.248.0/21 38690
table 0 add 93.180.72.0/21 38690
table 0 add 93.180.152.0/21 38690
table 0 add 93.181.0.0/18 38690
table 0 add 93.183.0.0/18 55830
...

The limits (without offset) are:
AA = 5650  -- actually AD = 5680
ZZ = 30900

With this formula, the offset must be less than 34735. Although, this wastes a 
considerable amount of rule number space, the advantage is that the numbers are 
still accessible by mental arithmetic, and the other constraint of having a 
step > 1 is fulfilled as well. Anyway, this one is not graved in stone, we can 
agree on another one.

> or there could be a translation into iso3166 numeric codes where Afghanistan 
> is 004, but then you have the worry of keeping the data up to date as they 
> add new country codes, which in my opinion makes it a bad solution.

Agreed, too much dependencies, let's forget the numeric iso codes.


BTW: The ipdb tools are now IPv6 aware.

$ geoip 2001:1900:2254:206a::50:5
2001:1900:2254:206a::50:5 in 2001:1900:0:0:0:0:0:0 - 
2001:1900:::::: in US

$ geoip -t "DE:BR:US" -p
...
...
217.194.64.0/20
217.194.224.0/20
217.195.0.0/20
217.195.32.0/20
217.197.80.0/20
217.198.128.0/20
217.198.240.0/20
217.199.64.0/20
217.199.192.0/20
217.224.0.0/11
2001:400:0:0:0:0:0:0/32
2001:408:0:0:0:0:0:0/32
2001:418:0:0:0:0:0:0/32
2001:420:0:0:0:0:0:0/32

Re: your thoughts on a particualar ipfw action.

[Dr. Rolf Jansen]

> Am 02.08.2016 um 05:08 schrieb Julian Elischer :
> 
> looking for thoughts from people who know the new IPFW features well..
> 
> 
> A recent addition to our armory is the geoip program that, given an address 
> can tell you what country it is in and given a country code, can give an ipfw 
> table that describes all the ip addresses in that country.
> 
> SO I was thinking how to use this, and the obvious way would be to have a set 
> of rules for each country, and use the "skipto tablearg" facility to skip to 
> the right rules for each country. But the trouble is that a tablearg skipto 
> is very inefficient. It's also a hard thing to set up with a set of rules for 
> each country (how many countries are there in the internet allocation 
> system?).

As of today a total of 236 country codes are in use for IPv4 delegations. If 
this helps for anything, a command line switch to the geoip tool could be added 
for letting it output the country code (as the hex encoded CC taken as a plain 
decimal integer) as the value for the given table entry. In the moment you can 
give one value for all entries generated by geoip, with this switch set, the 
output of geoip could look like:

$ geoip -t "DE:BR:US" -x
...
table 0 add 93.157.48.0/21 4445
table 0 add 93.158.236.0/22 4252
table 0 add 93.159.96.0/19 4445
table 0 add 93.159.248.0/21 4445
table 0 add 93.180.72.0/21 4445
table 0 add 93.180.152.0/21 4445
table 0 add 93.181.0.0/18 4445
table 0 add 93.183.0.0/18 5553
...

Given that ...
0x4445 = 'DE'
0x4252 = 'BR'
0x5553 = 'US'

..., IT people who know by heart the low ASCII table like chemists (are 
supposed to) know the periodic table of the elements, this should be not too 
hard to remember.

> Another way would be to just put 'action numbers' in the tablearg field and 
> have a few actions, shared by countries, but the trouble comes when you want 
> to  change the action for  a country, you need to rewrite potentially 
> thousands of entries (USA has over 15800 allocations).

Two or more geoip commands can be used for populating ipfw tables for different 
utilization in ipfw directives:

# Europe
geoip -t "FR:IT:DE:NL:BE:GB:..." -n 1 -x | ipfw -q > /dev/stdin

# North America
geoip -t "US:CA" -n 2 -x | ipfw -q > /dev/stdin

# South America
geoip -t "AR:BR:UR:CL:PY:BO:PE..." -n 3 -x | ipfw -q > /dev/stdin

...

> A second way woudl be to somehow map the tablearg of the country, into a 
> table of actions. effectively doing two levels of lookup.
> 
> The first table converting IP addresses to a country number and a second 
> lookup converting that to an action.
> 
> the only trouble is that I don't know of a way to do that.  If the new 
> changes allow that, and anyone knows how, please let me know :-).

Looking-up a given IP in the totally balanced binary search tree takes on a 
decent system on average about 10-20 nanoseconds. So in theory 50 to 100 
million packets per second could be filtered by this algorithm. In order to 
come more close to this performance in reality, it might be an option to move 
the search algorithm into ipfw.

Best regards

Rolf

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 01.08.2016 um 03:17 schrieb Julian Elischer <jul...@freebsd.org>:
> On 30/07/2016 10:17 PM, Dr. Rolf Jansen wrote:
>> I finished the work on CIDR conformity of the IP ranges tables generated by 
>> the tool geoip. The main constraint is that the start and end address of an 
>> IP block given by the delegation files MUST BE PRESERVED during the 
>> transformation to a set of CIDR records. This target is achieved by:
>> 
>>  1. Finding the largest common netmask boundary of the start address 
>> utilizing
>> int(log2(addr_count)); then iteration like Euclid's algorithm in 
>> computing
>> a GCD.
>> 
>>  2. Output the CIDR with the given start address and the masklen belonging
>> to the found netmask.
>> 
>>  3. If the CIDR does not match the whole original IP range then set the start
>> address of the next CIDR block to the next boundary of the common 
>> netmask,
>> and loop over starting at 1. until the original range has been satisfied.
>> 
> check out the appletalk code I pointed out  to you.. I wrote that in 93 or so 
> but I remember sweating blood
> over it to get it right.

I read the description of the code and the following sentence made me 
suspicious that aa_dorangeroute() would guarantee the above mentioned main 
constraint  "start and end address of an IP block given by the delegation files 
MUST BE PRESERVED" can be matched. Start/end address are said to be anything 
(even undefined) but fixed in the description.

   ... 
   Split the range into two subranges such that the middle
   of the two ranges is the point where the highest bit of difference
   between the two addresses makes its transition.
   ...

I do not want this.

>> I carefully tested the algorithm and a table that I pipe by the new geoip 
>> tool into ipfw is 100 % identical to the output of the ipfw command 'table N 
>> list'.
> though that doesn't mean it is semantically identical to the original table 
> due to 'most specific rule wins" behaviour.
> 
> for example:
> if you type in ;
> 
> 1.2.3.0/24 -> A
> and
> 1.2.3.0/26 -> B
> then both rules will be listed the same as what you put in
> but if you wanted to get all rules that point to A, without having rules that 
> point to B, then you would have to export
> 1.2.3.64/26  -> A
> 1.2.3.128/25 -> A
>  (i.e. TWO rules)

This is definitely not the usage case. The origin of the data to be passed to 
ipfw tables are RIR delegation statistics files, which is guaranteed to be 
consolidated, namely resolved overlaps and joined adjacencies, long before any 
tables for ipfw are generated. Each range entry got a well defined, i.e. fixed, 
i.e non-variable starting address, and anything that changes the starting 
address of the ranges renders the table useless. Every entry got a well defined 
range length, and that one also must not be changed, or the table would be 
useless as well.

In addition, we are talking about automatic generation of thousands of entries, 
and I never ever won't rely on something like 'most specific rule wins' 
behaviour, I want the behaviour as explicit as possible, and for this reason I 
am happy with 'INPUT is 100 % identical to the OUTPUT'.

> you could also export
> 1.2.3.0/24 -> A
> 1.2.3.0/26 -> 0  (think of it as an "EXCEPT for these" rule)
> 
> which is ALSO two rules but you would need to be sure that the receiver knows 
> what to do with them.

This is simply a ridiculous example in the given respect, this sounds like you 
are suggesting fuzzying the input data in order to bring ipfw to its limits. 
This makes life less boring, doesn't it? No thanks.

>> It is worth to note, that already the original RIR delegation files contain 
>> 457 non CIDR conforming IPv4 ranges in a total of 165815 original records. I 
>> guess that this number will increase in the future because the RIR's ran 
>> empty on new IPv4 ranges and are urged to subdivide returned old ranges for 
>> new delegations. The above algorithm is ready for this.
>> 
>> Generally, CIDR conforming tables are more than twice as large as optimized 
>> (joined adjacencies) IP range tables. All said changes have been pushed to 
>> GitHup already.
>> 
> Unfortunately there is no way to specify (using cidr notation) a.b.1.x AND 
> a.b.2.x without including a.b.[03].x.
> 
> HOWEVER
> if you specified the FULL table you could use the "except" feature of routing 
> table behaviour where
> a.b.0.x/22  -> A
> a.b.0.x/24  -> B
> a.b.3.x/24  -> B
> gives you the same thing because of the 'most specific rule wins" nature of 
> routing table evaluation.
> I believe this is the case in the tab

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 31.07.2016 um 15:38 schrieb Ian Smith <smi...@nimnet.asn.au>:
> On Sat, 30 Jul 2016 11:17:13 -0300, Dr. Rolf Jansen wrote:
>> I finished the work on CIDR conformity of the IP ranges tables 
>> generated by the tool geoip. The main constraint is that the start 
>> and end address of an IP block given by the delegation files MUST BE 
>> PRESERVED during the transformation to a set of CIDR records. This 
>> target is achieved by:
>> 
>> 1. Finding the largest common netmask boundary of the start address utilizing
>>int(log2(addr_count)); then iteration like Euclid's algorithm in computing
>>a GCD.
>> 
>> 2. Output the CIDR with the given start address and the masklen belonging
>>to the found netmask.
>> 
>> 3. If the CIDR does not match the whole original IP range then set the start
>>address of the next CIDR block to the next boundary of the common netmask,
>>and loop over starting at 1. until the original range has been satisfied.
>> 
>> I carefully tested the algorithm and a table that I pipe by the new 
>> geoip tool into ipfw is 100 % identical to the output of the ipfw 
>> command 'table N list'.
> 
> Great.  I suppose that caters for some of the odd delegations one sees, 
> such as perhaps a /16 then a /15 (ie 3/4 of a /14) followed by maybe a 
> /12, maybw with another /15 tacked on the end .. but I'm unsure if that 
> applies to country allocations as much as it does within countries.
> 
>> It is worth to note, that already the original RIR delegation files 
>> contain 457 non CIDR conforming IPv4 ranges in a total of 165815 
>> original records. I guess that this number will increase in the 
>> future because the RIR's ran empty on new IPv4 ranges and are urged 
>> to subdivide returned old ranges for new delegations. The above 
>> algorithm is ready for this.
> 
> Yes, and just as well.  I'm surprised it's as few as 457 .. I looked 
> into it a bit back when 115.70/17 was first allocated to my ISP, after 
> previously having been, as I recall, in China .. so of course we fell 
> foul of a number of (probably perennially) out-of-date geoip blockers, 
> for months in some cases .. malevolent beasts if not kept well fed :)
> 
>> Generally, CIDR conforming tables are more than twice as large as 
>> optimized (joined adjacencies) IP range tables. All said changes have 
>> been pushed to GitHup already.
> 
> So how many table entries does 'the world' come to, around 400,000?

No, it is not that bad. The total number of original entries in the delegation 
statistics files of all 5 RIR's is about 166000. The ipdb tool which compiles 
these ranges into a consolidated sorted binary table, that is suitable for 
loading it directly into a binary search tree, reduces the number of entries to 
a bit more than one half, namely ca. 83500.

Consolidation primarily means, resolving of overlaps, because these could not 
be handled in a meaningful way by a binary search tree. Only as an additional 
benefit in the same go, that routine combines adjacencies with the same country 
code, although, skipping the combination is technically not a show stopper for 
the BST, this is only to increase the performance.

The geoip tool which generates the tables of CIDR ranges per country code out 
of the consolidated tables would output a count of 167500 entries for all 
countries. That is a little bit more than the original count, however this 
table is still optimized, because original ranges that when combined form a new 
valid CIDR are not broken down again, but the combined CIDR is passed.

>> I am still a little bit amazed how ipfw come to accept incorrect CIDR 
>> ranges and arbitrarily moves the start/end addresses in order to 
>> achieve CIDR conformity, and that without any further notice, and 
>> that given that ipfw can be considered as being quite relevant to 
>> system security. Or, may I assume that ipfw knows always better than 
>> the user what should be allowed or denied. Otherwise, perhaps I am 
>> the only one ever who input incorrect CIDR ranges for processing by 
>> ipfw.
> 
> You've lost me here, Rolf.  Do you mean that ipfw adds incorrect table 
> entries for a given IPv4 address and mask length?  Or that it c/should 
> itself accept IP ranges and generate the needed CIDR entries to match?

Perhaps an example may explain it better. Remember that the first incarnation 
of geoip passed the incorrect range 201.222.20.0/20 to ipfw. This is an 
incorrect CIDR because the start address does not match a mask boundary defined 
by the given masklen. The point now is that this error is caused by EITHER the 
masklen is incorrect OR the start address is incorrect. ipfw can determine only 
that the CIDR is i

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 29.07.2016 um 10:23 schrieb Dr. Rolf Jansen <r...@obsigna.com>:
>> Am 29.07.2016 um 06:50 schrieb Julian Elischer <jul...@freebsd.org>:
>> On 29/07/2016 5:22 PM, Julian Elischer wrote:
>>> On 29/07/2016 4:53 PM, Dr. Rolf Jansen wrote:
>>>>> Am 28.07.2016 um 23:48 schrieb Lee Brown <l...@ratnaling.org>:
>>>>> 
>>>>> That makes sense to me.  Your /20 range encompasses 201.222.16.0 -
>>>>> 201.222.31.255.
>>>>> If you want 201.222.20.0-201.222.31.255, you'll need 3 ranges:
>>>>> 
>>>>> 201.222.20.0/22 (201.222.20.0-201.222.23.255)
>>>>> 201.222.24.0/22 (201.222.24.0-201.222.27.255)
>>>>> 201.222.28.0/22 (201.222.28.0-201.222.31.255)
>>>> 
>>>> Ian, Julian and Lee,
>>>> 
>>>> Thank you vary much for your responses. In order not bloat the thread, I 
>>>> answer only to one message.
>>>> 
>>>> I found the problem. As a matter of fact, the respective IP ranges in the 
>>>> LACNIC delegation statistics file are 3 adjacent blocks with 1024 
>>>> addresses, i.e. those that you listed in your message above:
>>>> 
>>>> $grep 201.222.2 /usr/local/etc/ipdb/IPRanges/lacnic.dat
>>>> lacnic|BR|ipv4|201.222.20.0|1024|20140710|allocated|164725
>>>> lacnic|BR|ipv4|201.222.24.0|1024|20140630|allocated|138376
>>>> lacnic|BR|ipv4|201.222.28.0|1024|20140701|allocated|129095
>>>> 
>>>> However, my database compilation combines adjacent blocks with the same 
>>>> country code, and the ranges above turn into one block of 3072 addresses, 
>>>> which obviously doesn't have a valid netmask - log(3072) = 11,5849625.
>>>> ...
>>>> ..., it is not sufficient to forget about optimization but I need to check 
>>>> also whether, the delegation files contain already some non-CIDR ranges, 
>>>> which need to be broken down.
>>> 
>>> there is code to take ranges and produce cidr sets.
>>> 
>>> We used to have exactly that code in the appletalk code before we took it 
>>> out. Appletalk uses ranges.
>>> https://svnweb.freebsd.org/base/release/3.2.0/sys/netatalk/at_control.c?view=annotate#l703
>>>  
>> 
>> though htat uassumes input in the form af an appletak sockaddr..
>> there is also this python module
>> https://pythonhosted.org/netaddr/tutorial_01.html#support-for-non-standard-address-ranges
>> 
>>> maybe you can find other versions on the net.
>>> however if you fully populate the table, you will get the correct result 
>>> because more specific entries will
>>> override less specific entries. To do that you would have to have a way to 
>>> describe to your program what
>>> value each table entry should output.
>>> If you did what you do now, then you would specify the value for the 
>>> required countries, and give a default falue for "all others".
>>> aggregation of adjacent ranges with same value would be an optimisation.
> 
> Don't worry, breaking down an arbitrary IP-range into a CIDR conforming set 
> of ranges, doesn't seem too difficult. ...
> ...
> Once I come to a conclusion, I will post it to this mailing list.

I finished the work on CIDR conformity of the IP ranges tables generated by the 
tool geoip. The main constraint is that the start and end address of an IP 
block given by the delegation files MUST BE PRESERVED during the transformation 
to a set of CIDR records. This target is achieved by:

 1. Finding the largest common netmask boundary of the start address utilizing
int(log2(addr_count)); then iteration like Euclid's algorithm in computing
a GCD.

 2. Output the CIDR with the given start address and the masklen belonging
to the found netmask.

 3. If the CIDR does not match the whole original IP range then set the start
address of the next CIDR block to the next boundary of the common netmask,
and loop over starting at 1. until the original range has been satisfied.

I carefully tested the algorithm and a table that I pipe by the new geoip tool 
into ipfw is 100 % identical to the output of the ipfw command 'table N list'.

It is worth to note, that already the original RIR delegation files contain 457 
non CIDR conforming IPv4 ranges in a total of 165815 original records. I guess 
that this number will increase in the future because the RIR's ran empty on new 
IPv4 ranges and are urged to subdivide returned old ranges for new delegations. 
The above algorithm is ready for this.

Generally, CIDR conforming table

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 29.07.2016 um 06:50 schrieb Julian Elischer <jul...@freebsd.org>:
> On 29/07/2016 5:22 PM, Julian Elischer wrote:
>> On 29/07/2016 4:53 PM, Dr. Rolf Jansen wrote:
>>>> Am 28.07.2016 um 23:48 schrieb Lee Brown <l...@ratnaling.org>:
>>>> 
>>>> That makes sense to me.  Your /20 range encompasses 201.222.16.0 -
>>>> 201.222.31.255.
>>>> If you want 201.222.20.0-201.222.31.255, you'll need 3 ranges:
>>>> 
>>>> 201.222.20.0/22 (201.222.20.0-201.222.23.255)
>>>> 201.222.24.0/22 (201.222.24.0-201.222.27.255)
>>>> 201.222.28.0/22 (201.222.28.0-201.222.31.255)
>>> 
>>> Ian, Julian and Lee,
>>> 
>>> Thank you vary much for your responses. In order not bloat the thread, I 
>>> answer only to one message.
>>> 
>>> I found the problem. As a matter of fact, the respective IP ranges in the 
>>> LACNIC delegation statistics file are 3 adjacent blocks with 1024 
>>> addresses, i.e. those that you listed in your message above:
>>> 
>>> $grep 201.222.2 /usr/local/etc/ipdb/IPRanges/lacnic.dat
>>> lacnic|BR|ipv4|201.222.20.0|1024|20140710|allocated|164725
>>> lacnic|BR|ipv4|201.222.24.0|1024|20140630|allocated|138376
>>> lacnic|BR|ipv4|201.222.28.0|1024|20140701|allocated|129095
>>> 
>>> However, my database compilation combines adjacent blocks with the same 
>>> country code, and the ranges above turn into one block of 3072 addresses, 
>>> which obviously doesn't have a valid netmask - log(3072) = 11,5849625.
>>> ...
>>> ..., it is not sufficient to forget about optimization but I need to check 
>>> also whether, the delegation files contain already some non-CIDR ranges, 
>>> which need to be broken down.
>> 
>> there is code to take ranges and produce cidr sets.
>> 
>> We used to have exactly that code in the appletalk code before we took it 
>> out. Appletalk uses ranges.
>> https://svnweb.freebsd.org/base/release/3.2.0/sys/netatalk/at_control.c?view=annotate#l703
>>  
> 
> though htat uassumes input in the form af an appletak sockaddr..
> there is also this python module
> https://pythonhosted.org/netaddr/tutorial_01.html#support-for-non-standard-address-ranges
> 
>> maybe you can find other versions on the net.
>> however if you fully populate the table, you will get the correct result 
>> because more specific entries will
>> override less specific entries. To do that you would have to have a way to 
>> describe to your program what
>> value each table entry should output.
>> If you did what you do now, then you would specify the value for the 
>> required countries, and give a default falue for "all others".
>> aggregation of adjacent ranges with same value would be an optimisation.

Don't worry, breaking down an arbitrary IP-range into a CIDR conforming set of 
ranges, doesn't seem too difficult. I quickly hacked into the   for() loop for 
table generation in geoip.c a nested do while() for if necessary breaking down 
the given range:

for (i = 0; i < n; i++)
{
   if (!*ccList || findCC(CCTable, sortedIPSets[i][2]))
   {
  ipv4_lo = sortedIPSets[i][0];
  do
  {
 m = intlb(sortedIPSets[i][1] - ipv4_lo  + 1);
 while (ipv4_lo  % (k = (int)lround(exp2(m m--;
 printRange(ipv4_lo, 32 - m, table_number, table_value);
 
  }
  while ((ipv4_lo += k) < sortedIPSets[i][1]);
   }
}

This seems to work as expected, however, I need to check this more carefully 
(in the course of the weekend) until pushing it to GitHub. At the first glance, 
the tables become quite large. For example, the table for Brazil of 824 joined 
entries is bloated to 6621 CIDR records.

Once I came to a conclusion, I will post it to this mailing list.

Best regards

Rolf

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 27.07.2016 um 12:31 schrieb Julian Elischer <jul...@freebsd.org>:
> On 27/07/2016 9:36 PM, Dr. Rolf Jansen wrote:
>>> Am 26.07.2016 um 23:03 schrieb Julian Elischer <jul...@freebsd.org>:
>>> On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote:
>>>> There is another tool called geoip , that I uploaded to GitHub, and that I 
>>>> use for looking up country codes by IP addresses on the command line.
>>>> 
>>>> https://github.com/cyclaero/ipdb/blob/master/geoip.c
>>>> 
>>>> This one could easily be extended to produce sorted IP ranges per CC that 
>>>> could be fed into tables of ipfw. I am thinking of adding a command line 
>>>> option for specifying CC's for which the IP ranges should be exported, 
>>>> something like:
>>>> 
>>>>geoip -e DE:BR:US:IT:FR:ES
>>>> 
>>>> And this could print sorted IP-Ranges belonging to the listed countries. 
>>>> For this purpose, what would be the ideal format for directly feeding the 
>>>> produced output into ipfw tables?
>>> The format for using tables directly is the same as that used for routing 
>>> tables.
>>> …
>>> table 5 add 1.1.1.0/32 1000
>>> …
>>> your application becomes an application for configuring the firewall.
>>> (which you do by feeding commands down a pipe to ipfw, which is started as 
>>> 'ipfw -q /dev/stdin')
>> I finished adding a second usage form for the geoip tool, namely generation 
>> of ipfw table construction directives filtered by country codes.
> wow, wonderful!
> 
> with that tool, and ipfw tables we have a fully functional geo 
> blocking/munging solution in about 4 lines of shell script.

Unfortunately, I finally discovered that ipfw tables as they are, are 
unsuitable for the given purpose, because for some reason ipfw mangles about 20 
% of the passed IP address/masklen pairs.

For example:

# ipfw table 1 add 201.222.20.0/20
# ipfw table 1 list
-->  201.222.16.0/20 0

$ geoip 201.222.20.1
--> 201.222.20.1 in 201.222.20.0-201.222.31.255 in BR

$ geoip 201.222.16.1
--> 201.222.16.1 in 201.222.16.0-201.222.19.255 in AR

Effectively, I asked ipfw to add an IP-range of Brazil to table 1, but it 
actually added another one which belongs to Argentina. This doesn't make too 
much sense, does it?

For the time being I switched my servers back to geo-blocking with the divert 
filter daemon.

Best regards

Rolf

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 27.07.2016 um 17:08 schrieb olli hauer <oha...@gmx.de>:
> On 2016-07-27 15:36, Dr. Rolf Jansen wrote:
>> 
>> I finished adding a second usage form for the geoip tool, namely generation 
>> of ipfw table construction directives filtered by country codes.
>> 
>> __
>> $ geoip -h
>> geoip v1.0.1 (16), Copyright © 2016 Dr. Rolf Jansen
>> 
>> Usage:
>> 
>> 1) look-up the country code belonging to an IPv4 address given by the last 
>> command line argument:
>> 
>>   geoip [-r bstfile] [-h] 
>>  a dotted IPv4 address to be looked-up.
>> 
>> 2) generate a sorted list of IPv4 address/masklen pairs per country code, 
>> formatted as ipfw table construction directives:
>> 
>>   geoip -t [CC:DD:EE:..] [-n table number] [-v table value] [-r bstfile] [-h]
>> 
>>  -t [CC:DD:EE:..]  output all IPv4 address/masklen pairs belonging to 
>> the listed countries, given by 2 letter
>>capital country codes, separated by colon. An empty 
>> CC list means any country code.
>>  -n table number   the ipfw table number between 0 and 65534 [default: 
>> 0].
>>  -v table valuethe 32-bit unsigned value of the ipfw table entry 
>> [default: 0].
>> 
>> valid arguments in both usage forms:
>> 
>>  -r bstfilethe path to the binary file with the consolidated IP 
>> ranges that has been.
>>generated by the 'ipdb' tool [default: 
>> /usr/local/etc/ipdb/IPRanges/ipcc.bst].
>>  -hshow these usage instructions.
>> __
>> 
>> With that, the ipfw configuration script may contain something alike:
>> 
>>…
>># allow only web access from DE, BR, US:
>>/usr/local/bin/geoip -t DE:BR:US -n 7 | /sbin/ipfw -q /dev/stdin
>>/sbin/ipfw -q add 70 deny tcp from not table\(7\) to any 80,443 in recv 
>> WAN_if setup
>>…
>> 
>> OR, the other way around:
>>…
>># deny web access from certain disgraceful regions:
>>/usr/local/bin/geoip -t KO:TR:SA:RU:GB -n 66 | /sbin/ipfw -q /dev/stdin
>>/sbin/ipfw -q add 70 allow tcp from not table\(66\) to any 80,443 in recv 
>> WAN_if setup
>>…
>> 
> 
> Nice work :)
> 
> Now it is also possible to use geoip to create files usable for pf.
> (just pipe the output through sed -e 's/table 0 add //')
> 
> Perhaps the following diff for Makefile is useful.
> - use PREFIX instead hard coded path
> - use "install -s" instead "strip -x -o"
> - use "install -m" instead "cp ; chmod"

I changed the Makefile according to your suggestions, and I added another 
command line option to the geoip tool:

…
  -p   plain IP table generation, i.e. without ipfw construction 
directives, -n and -v flags are ignored.
…

The changes are already uploaded to GitHub.

Best regards

Rolf

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 26.07.2016 um 23:03 schrieb Julian Elischer <jul...@freebsd.org>:
> On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote:
>> There is another tool called geoip , that I uploaded to GitHub, and that I 
>> use for looking up country codes by IP addresses on the command line.
>> 
>> https://github.com/cyclaero/ipdb/blob/master/geoip.c
>> 
>> This one could easily be extended to produce sorted IP ranges per CC that 
>> could be fed into tables of ipfw. I am thinking of adding a command line 
>> option for specifying CC's for which the IP ranges should be exported, 
>> something like:
>> 
>>geoip -e DE:BR:US:IT:FR:ES
>> 
>> And this could print sorted IP-Ranges belonging to the listed countries. For 
>> this purpose, what would be the ideal format for directly feeding the 
>> produced output into ipfw tables?
> The format for using tables directly is the same as that used for routing 
> tables.
> …
> table 5 add 1.1.1.0/32 1000
> …
> your application becomes an application for configuring the firewall.
> (which you do by feeding commands down a pipe to ipfw, which is started as 
> 'ipfw -q /dev/stdin')

I finished adding a second usage form for the geoip tool, namely generation of 
ipfw table construction directives filtered by country codes.

__
$ geoip -h
geoip v1.0.1 (16), Copyright © 2016 Dr. Rolf Jansen

Usage:

1) look-up the country code belonging to an IPv4 address given by the last 
command line argument:

   geoip [-r bstfile] [-h] 
  a dotted IPv4 address to be looked-up.

2) generate a sorted list of IPv4 address/masklen pairs per country code, 
formatted as ipfw table construction directives:

   geoip -t [CC:DD:EE:..] [-n table number] [-v table value] [-r bstfile] [-h]

  -t [CC:DD:EE:..]  output all IPv4 address/masklen pairs belonging to the 
listed countries, given by 2 letter
capital country codes, separated by colon. An empty CC 
list means any country code.
  -n table number   the ipfw table number between 0 and 65534 [default: 0].
  -v table valuethe 32-bit unsigned value of the ipfw table entry 
[default: 0].

valid arguments in both usage forms:

  -r bstfilethe path to the binary file with the consolidated IP 
ranges that has been.
generated by the 'ipdb' tool [default: 
/usr/local/etc/ipdb/IPRanges/ipcc.bst].
  -hshow these usage instructions.
__

With that, the ipfw configuration script may contain something alike:

…
# allow only web access from DE, BR, US:
/usr/local/bin/geoip -t DE:BR:US -n 7 | /sbin/ipfw -q /dev/stdin
/sbin/ipfw -q add 70 deny tcp from not table\(7\) to any 80,443 in recv 
WAN_if setup
…

OR, the other way around:
…
# deny web access from certain disgraceful regions:
/usr/local/bin/geoip -t KO:TR:SA:RU:GB -n 66 | /sbin/ipfw -q /dev/stdin
/sbin/ipfw -q add 70 allow tcp from not table\(66\) to any 80,443 in recv 
WAN_if setup
…



Best regards

Rolf


___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 26.07.2016 um 13:23 schrieb Julian Elischer <jul...@freebsd.org>:
> On 26/07/2016 1:41 AM, Dr. Rolf Jansen wrote:
>> Once a week, the IP ranges are compiled from original sources into a binary 
>> sorted table, containing as of today 83162 consolidated range/cc pairs. On 
>> starting-up, the divert daemon reads the binary file in one block and stores 
>> the ranges into a totally balanced binary search tree. Looking-up a country 
>> code for a given IPv4 address in the BST takes on average 20 nanoseconds on 
>> an AWS-EC2 micro instance. I don't know the overhead of diverting, though. I 
>> guess this may be one or two orders of magnitudes higher. Even though, I 
>> won't see any performance issues.
> 
> yes the diversion to user space is not a fast operation. When we wrote it, 
> fast was 10Mbits/sec.
> The firewall tables use a radix tree (*) and might be slower than what you 
> have, but possibly it might be made up for by not having to do the divert 
> logic. it's not entorely clear from your description why you look up  a 
> country rather than just a pass/block result, but maybe different sources can 
> access different countries?.

The basic idea was to develop a facility for ipfw for filtering IPv4 packets by 
country code - see: https://github.com/cyclaero/ipdb

I simply put into /etc/rc.conf:

   geod_enable="YES"
   geod_flags="-a DE:BR:US"

The -a flag tells, that source IP addresses only from these countries are 
allowed (i.e. passed through the filter). I added also a -d flag, which means 
deny (i.e. drop packets) from the given list of countries.

With that in place, I need to add a respective divert rule to the ipfw ruleset 
(the divert port of the geod daemon is 8669, remembering that 8668 is the port 
of the natd daemon):

ipfw -q add 70 divert 8669 tcp from any to any 80,443 in recv WAN_if setup

> I did similar once using ipfw tables but couldn't find a reliable source of 
> data.

The IP/CC database is compiled from downloads of the daily published delegation 
statistics files of the 5 RIR's. I consider the RIR's being the authoritative 
source. Anyway, on my systems the IP/CC-database is updated only weekly, 
although, daily updating would be possible. I wrote a shell script for this, 
that can be executed by a cron job.

https://github.com/cyclaero/ipdb/blob/master/ipdb-update.sh 
<https://github.com/cyclaero/ipdb/blob/master/ipdb-update.sh>

There is another tool called geoip , that I uploaded to GitHub, and that I use 
for looking up country codes by IP addresses on the command line.

https://github.com/cyclaero/ipdb/blob/master/geoip.c

This one could easily be extended to produce sorted IP ranges per CC that could 
be fed into tables of ipfw. I am thinking of adding a command line option for 
specifying CC's for which the IP ranges should be exported, something like:

   geoip -e DE:BR:US:IT:FR:ES 

And this could print sorted IP-Ranges belonging to the listed countries. For 
this purpose, what would be the ideal format for directly feeding the produced 
output into ipfw tables?

>> Independent from the actual usage case (geo-blocking), let's talk about 
>> divert filtering in general. The original question which is still unanswered 
>> can be generalized to, whether "dropping/denying" a package simply means 
>> 'forget about it' or whether the divert filter is required to do something 
>> more involved, e.g. communicate the situation somehow to ipfw.
> 
> there is no residual information about the packet in the kernel once it has 
> been passed to the  user process.
> so just "forgetting to hand it back" is sufficient to drop it.

OK, many thanks, that just answers my original doubt. At least technically, my 
daemon handles package dropping correctly, although, more elegant ways can be 
imagined to do the same thing.

Best regards

Rolf 
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 25.07.2016 um 12:47 schrieb Michael Sierchio :
> 
> Writing a divert daemon is a praiseworthy project, but I think you could do
> this without sending packets to user land.
> 
> You could use tables - …


> Am 25.07.2016 um 14:01 schrieb Jan Bramkamp :
> 
> I would use a set of IPFW tables with skipto/call tablearg rules instead …

Michael and Jan, many thanks for your suggestions.

As everybody knows, 'Many roads lead to Rome.', and I am already there. I don't 
feel alike going all the way back only for the sake of trying out other routes.

Once a week, the IP ranges are compiled from original sources into a binary 
sorted table, containing as of today 83162 consolidated range/cc pairs. On 
starting-up, the divert daemon reads the binary file in one block and stores 
the ranges into a totally balanced binary search tree. Looking-up a country 
code for a given IPv4 address in the BST takes on average 20 nanoseconds on an 
AWS-EC2 micro instance. I don't know the overhead of diverting, though. I guess 
this may be one or two orders of magnitudes higher. Even though, I won't see 
any performance issues.

Independent from the actual usage case (geo-blocking), let's talk about divert 
filtering in general. The original question which is still unanswered can be 
generalized to, whether "dropping/denying" a package simply means 'forget about 
it' or whether the divert filter is required to do something more involved, 
e.g. communicate the situation somehow to ipfw.

Best regards

Rolf
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

I have written a ipfw divert filter daemon for IPv4 geo-blocking. It is working 
flawlessly on two server installations since a week.

Anyway, I am still in doubt whether I do the blocking in the correct way. Once 
the filter receives a packet from the respective divert socket it looks up the 
country code of the source IP in the IP-Ranges database, and if the country 
code shall be allowed then it returns the unaltered packet via said socket, 
otherwise, the filter does no further processing, so the packet is effectively 
gone, lost, dropped, discarded, or whatever would be the correct terminology. 
Is this the really the correct way of denying a packet, or is it necessary to 
inform ipfw somehow about the circumstances, so it can run a proper dropping 
procedure?

I uploaded the filter + accompanying tools to GitHub

   https://github.com/cyclaero/ipdb

Many thnaks for any advices in advance.

Best regards

Rolf

 
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"