Re: ipfw divert filter for IPv4 geo-blocking

2016-07-26 Thread Julian Elischer 

On 26/07/2016 1:41 AM, Dr. Rolf Jansen wrote:

Am 25.07.2016 um 12:47 schrieb Michael Sierchio :

Writing a divert daemon is a praiseworthy project, but I think you could do
this without sending packets to user land.

You could use tables - …



Am 25.07.2016 um 14:01 schrieb Jan Bramkamp :

I would use a set of IPFW tables with skipto/call tablearg rules instead …

Michael and Jan, many thanks for your suggestions.

As everybody knows, 'Many roads lead to Rome.', and I am already there. I don't 
feel alike going all the way back only for the sake of trying out other routes.

and I personally am responsible for at least parts of several of them ;-)
(parts of ipdivert, netgraph, and various ipfw bits).




Once a week, the IP ranges are compiled from original sources into a binary 
sorted table, containing as of today 83162 consolidated range/cc pairs. On 
starting-up, the divert daemon reads the binary file in one block and stores 
the ranges into a totally balanced binary search tree. Looking-up a country 
code for a given IPv4 address in the BST takes on average 20 nanoseconds on an 
AWS-EC2 micro instance. I don't know the overhead of diverting, though. I guess 
this may be one or two orders of magnitudes higher. Even though, I won't see 
any performance issues.


yes the diversion to user space is not a fast operation. When we wrote 
it, fast was 10Mbits/sec.
The firewall tables use a radix tree (*) and might be slower than what 
you have, but possibly it might be made up for by not having to do the 
divert logic. it's not entorely clear from your description why you 
look up  a country rather than just a pass/block result, but maybe 
different sources can access different countries?.


I did similar once using ipfw tables but couldn't find a reliable 
source of data.




Independent from the actual usage case (geo-blocking), let's talk about divert filtering 
in general. The original question which is still unanswered can be generalized to, 
whether "dropping/denying" a package simply means 'forget about it' or whether 
the divert filter is required to do something more involved, e.g. communicate the 
situation somehow to ipfw.


there is no residual information about the packet in the kernel once 
it has been passed to the  user process.

so just "forgetting to hand it back" is sufficient to drop it.



Best regards

Rolf
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"




___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

2016-07-26 Thread Julian Elischer 

On 26/07/2016 1:01 AM, Jan Bramkamp wrote:



On 25/07/16 16:28, Dr. Rolf Jansen wrote:
I have written a ipfw divert filter daemon for IPv4 geo-blocking. 
It is working flawlessly on two server installations since a week.


Anyway, I am still in doubt whether I do the blocking in the 
correct way. Once the filter receives a packet from the respective 
divert socket it looks up the country code of the source IP in the 
IP-Ranges database, and if the country code shall be allowed then 
it returns the unaltered packet via said socket, otherwise, the 
filter does no further processing, so the packet is effectively 
gone, lost, dropped, discarded, or whatever would be the correct 
terminology. Is this the really the correct way of denying a 
packet, or is it necessary to inform ipfw somehow about the 
circumstances, so it can run a proper dropping procedure?


I uploaded the filter + accompanying tools to GitHub

   https://github.com/cyclaero/ipdb

Many thnaks for any advices in advance.


I would use a set of IPFW tables with skipto/call tablearg rules 
instead. 


Use the daemon to maintain the IPFW tables. I assume your database 
is a list of of (CIDR, country code) pairs. In that case the daemon 
config should probably map from sets of country codes to table 
values e.g.


table 1 { DE, NL } -> 1,
{ US, UK } -> 10100
table 2 { CN, KO, TR } -> 2

why multiple tables?
if you load the table at once you can assign a country code as the 
tablearg for every run of addresses. all in one table.




Next the daemon would calculate the minimal set of table entries to 
match these policies exactly and patch the kernel table contents if 
the database changes.


This design avoids the userspace<->kernel copies without losing 
flexibility.

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"



___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

2016-07-26 Thread Michael Sierchio 
On Tue, Jul 26, 2016 at 9:26 AM, Julian Elischer  wrote:

table 1 { DE, NL } -> 1,
>> { US, UK } -> 10100
>> table 2 { CN, KO, TR } -> 2
>>
> why multiple tables?
> if you load the table at once you can assign a country code as the
> tablearg for every run of addresses. all in one table.


I mentioned that in my earlier response - but if the point is to block
entire countries (or any collection of CIDR blocks, for that matter), it's
sufficient to have a whitelist table and a blacklist table. The table arg
could also be a skipto rule number, right? And you can do policy-based
routing, with the table arg as a FIB number.

Passing the packet to userland via divert sockets was a brilliant idea in
2003. natd was pretty much the first NAT mechanism to properly handle ICMP
error responses, too.

-- 
"Well," Brahma said, "even after ten thousand explanations, a fool is no
wiser, but an intelligent man requires only two thousand five hundred."

- The Mahābhārata
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

2016-07-26 Thread Julian Elischer 

On 27/07/2016 1:40 AM, Michael Sierchio wrote:

On Tue, Jul 26, 2016 at 9:26 AM, Julian Elischer  wrote:

 table 1 { DE, NL } -> 1,

 { US, UK } -> 10100
 table 2 { CN, KO, TR } -> 2


why multiple tables?
if you load the table at once you can assign a country code as the
tablearg for every run of addresses. all in one table.


I mentioned that in my earlier response - but if the point is to block
entire countries (or any collection of CIDR blocks, for that matter), it's
sufficient to have a whitelist table and a blacklist table. The table arg
could also be a skipto rule number, right? And you can do policy-based
routing, with the table arg as a FIB number.

Passing the packet to userland via divert sockets was a brilliant idea in
2003. natd was pretty much the first NAT mechanism to properly handle ICMP
error responses, too.

2003?  nahh we wrote it and divert in 96 :-)





___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

2016-07-26 Thread Dr. Rolf Jansen 
> Am 26.07.2016 um 13:23 schrieb Julian Elischer :
> On 26/07/2016 1:41 AM, Dr. Rolf Jansen wrote:
>> Once a week, the IP ranges are compiled from original sources into a binary 
>> sorted table, containing as of today 83162 consolidated range/cc pairs. On 
>> starting-up, the divert daemon reads the binary file in one block and stores 
>> the ranges into a totally balanced binary search tree. Looking-up a country 
>> code for a given IPv4 address in the BST takes on average 20 nanoseconds on 
>> an AWS-EC2 micro instance. I don't know the overhead of diverting, though. I 
>> guess this may be one or two orders of magnitudes higher. Even though, I 
>> won't see any performance issues.
> 
> yes the diversion to user space is not a fast operation. When we wrote it, 
> fast was 10Mbits/sec.
> The firewall tables use a radix tree (*) and might be slower than what you 
> have, but possibly it might be made up for by not having to do the divert 
> logic. it's not entorely clear from your description why you look up  a 
> country rather than just a pass/block result, but maybe different sources can 
> access different countries?.

The basic idea was to develop a facility for ipfw for filtering IPv4 packets by 
country code - see: https://github.com/cyclaero/ipdb

I simply put into /etc/rc.conf:

   geod_enable="YES"
   geod_flags="-a DE:BR:US"

The -a flag tells, that source IP addresses only from these countries are 
allowed (i.e. passed through the filter). I added also a -d flag, which means 
deny (i.e. drop packets) from the given list of countries.

With that in place, I need to add a respective divert rule to the ipfw ruleset 
(the divert port of the geod daemon is 8669, remembering that 8668 is the port 
of the natd daemon):

ipfw -q add 70 divert 8669 tcp from any to any 80,443 in recv WAN_if setup

> I did similar once using ipfw tables but couldn't find a reliable source of 
> data.

The IP/CC database is compiled from downloads of the daily published delegation 
statistics files of the 5 RIR's. I consider the RIR's being the authoritative 
source. Anyway, on my systems the IP/CC-database is updated only weekly, 
although, daily updating would be possible. I wrote a shell script for this, 
that can be executed by a cron job.

https://github.com/cyclaero/ipdb/blob/master/ipdb-update.sh 


There is another tool called geoip , that I uploaded to GitHub, and that I use 
for looking up country codes by IP addresses on the command line.

https://github.com/cyclaero/ipdb/blob/master/geoip.c

This one could easily be extended to produce sorted IP ranges per CC that could 
be fed into tables of ipfw. I am thinking of adding a command line option for 
specifying CC's for which the IP ranges should be exported, something like:

   geoip -e DE:BR:US:IT:FR:ES 

And this could print sorted IP-Ranges belonging to the listed countries. For 
this purpose, what would be the ideal format for directly feeding the produced 
output into ipfw tables?

>> Independent from the actual usage case (geo-blocking), let's talk about 
>> divert filtering in general. The original question which is still unanswered 
>> can be generalized to, whether "dropping/denying" a package simply means 
>> 'forget about it' or whether the divert filter is required to do something 
>> more involved, e.g. communicate the situation somehow to ipfw.
> 
> there is no residual information about the packet in the kernel once it has 
> been passed to the  user process.
> so just "forgetting to hand it back" is sufficient to drop it.

OK, many thanks, that just answers my original doubt. At least technically, my 
daemon handles package dropping correctly, although, more elegant ways can be 
imagined to do the same thing.

Best regards

Rolf 
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

2016-07-26 Thread Julian Elischer 

On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote:

Am 26.07.2016 um 13:23 schrieb Julian Elischer :
On 26/07/2016 1:41 AM, Dr. Rolf Jansen wrote:

Once a week, the IP ranges are compiled from original sources into a binary 
sorted table, containing as of today 83162 consolidated range/cc pairs. On 
starting-up, the divert daemon reads the binary file in one block and stores 
the ranges into a totally balanced binary search tree. Looking-up a country 
code for a given IPv4 address in the BST takes on average 20 nanoseconds on an 
AWS-EC2 micro instance. I don't know the overhead of diverting, though. I guess 
this may be one or two orders of magnitudes higher. Even though, I won't see 
any performance issues.

yes the diversion to user space is not a fast operation. When we wrote it, fast 
was 10Mbits/sec.
The firewall tables use a radix tree (*) and might be slower than what you 
have, but possibly it might be made up for by not having to do the divert 
logic. it's not entorely clear from your description why you look up  a country 
rather than just a pass/block result, but maybe different sources can access 
different countries?.

The basic idea was to develop a facility for ipfw for filtering IPv4 packets by 
country code - see: https://github.com/cyclaero/ipdb

I simply put into /etc/rc.conf:

geod_enable="YES"
geod_flags="-a DE:BR:US"

The -a flag tells, that source IP addresses only from these countries are 
allowed (i.e. passed through the filter). I added also a -d flag, which means 
deny (i.e. drop packets) from the given list of countries.

With that in place, I need to add a respective divert rule to the ipfw ruleset 
(the divert port of the geod daemon is 8669, remembering that 8668 is the port 
of the natd daemon):

 ipfw -q add 70 divert 8669 tcp from any to any 80,443 in recv WAN_if setup


I did similar once using ipfw tables but couldn't find a reliable source of 
data.

The IP/CC database is compiled from downloads of the daily published delegation 
statistics files of the 5 RIR's. I consider the RIR's being the authoritative 
source. Anyway, on my systems the IP/CC-database is updated only weekly, 
although, daily updating would be possible. I wrote a shell script for this, 
that can be executed by a cron job.

 https://github.com/cyclaero/ipdb/blob/master/ipdb-update.sh 


There is another tool called geoip , that I uploaded to GitHub, and that I use 
for looking up country codes by IP addresses on the command line.

 https://github.com/cyclaero/ipdb/blob/master/geoip.c

This one could easily be extended to produce sorted IP ranges per CC that could 
be fed into tables of ipfw. I am thinking of adding a command line option for 
specifying CC's for which the IP ranges should be exported, something like:

geoip -e DE:BR:US:IT:FR:ES

And this could print sorted IP-Ranges belonging to the listed countries. For 
this purpose, what would be the ideal format for directly feeding the produced 
output into ipfw tables?
The format for using tables directly is the same as that used for 
routing tables.
so imagine that you had to generate a routing table that sent packets 
to two different routers depending on their source.


here's a simple rule set that filters web traffic by such a 'routing 
table'
except it's routing to two different rules. It also sorts OUTGOING web 
traffic to the same rules.


ipfw -q /dev/stdin <<-DONE
# we hate this guy
table 5 add 1.1.1.0/32 1000
# but all ow our people to visit everyone else in that subnet
table 5 add 1.1.0.0/24 2000
# we block 1.1.2.0 through 1.1.3.255
table 5 add 1.1.2.0/23 1000
# but we allow 1.1.4.0 through to 1.1.7.255
table 5 add 1.1.4.0/22 2000
# etc
table 5 add 1.1.8.0/21 1000
table 5 add 1.2.0.0/16 1000
table 5 add 0.0.0.0/0 2000 # default
check-state  # If we already decided what to do,  do it
# select out only external traffic, into direction specific rules.
add 400 skipto 500 ip from any to any in recv WAN_if
add 410 skipto 700 ip from any to any out xmit WAN_If
add 320 skipto 1
# Incoming packets
add 500 skipto tablearg tcp from table(5) to any 80,443 setup 
keep-state  # sort tcp setup packets between rules 1000 and 2000

add 600 skipto 1
# outgoing packets
add 700 skipto tablearg tcp from any  to table(5) 80,443 setup 
keep-state  # sort tcp setup packets between rules 100 and 2000

add 800 skipto 1
add 1000 drop ip from any to any
add 2000 allow ip from any to any
# further processing
add 1 .. # further processing for non tcp
DONE

for full configurability you could have a rule for each country, and a 
number for it in the table:

table 5 add 150.101.0.0/16 10610 # Australia
[...]
add 10610 block tcp from any to any 445  # only allow non encrypted 
web to those Aussie scum.

add 10611 allow ip from any to any

then by changing the rules at that location you could change the 
policy for a country without changing everything else.
(t