Re: [Bloat] netperf server news

[Kenneth Porter]

--On Wednesday, October 07, 2020 3:23 PM -0400 Rich Brown 
 wrote:



I'm also aware of ipset, which I take to be an optimized alternative to
searching a long set of iptables rules (true?) I don't believe that my
OpenVZ VPS has kernel support for this, so as long as the
long-list-of-rules seems to work well, I'm going to leave it alone.


A quick google of "OpenVZ ipset" turned up a thread from 3 years ago 
suggesting it's in their kernel:




Note that ipset operates in addition to iptables. Other kernel subsystems 
can also use them. iptables has a module to query an ipset.


500 rules is a lot to search linearly. I'd think a hash table would give 
much superior performance. Note that every "good" packet has to check ALL 
the blocking rules to be approved.


I use ipsets to block probes to my mail servers from outside the country 
and from cloud services. I've managed to find a few sources of lists for 
those. I also use ipset with fail2ban.


The only complicated part is how to handle reboots or other service 
restarts. I use firewalld which does its own ipset management so I put the 
permanent lists there. (I have scripts to convert the cloud lists to a 
firewalld's XML format for its ipset storage.) fail2ban keeps its own block 
database in a sqlite file and tears down and recreates its ipsets on 
restart.


___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] netperf server news

[Rich Brown]

> 
> 
> --
> 
> Message: 1
> Date: Tue, 06 Oct 2020 19:39:54 -0700
> From: Kenneth Porter 
> To: bloat 
> Subject: Re: [Bloat] netperf server news
> Message-ID: <38F0B196CFEA470FEEBE0520@[172.27.17.193]>
> Content-Type: text/plain; charset=us-ascii; format=flowed
> 
> --On Tuesday, October 06, 2020 7:52 AM -0400 Rich Brown 
>  wrote:
> 
>> 3) I would be pleased to get comments on the set of scripts. I'm a newbie
>> at iptables, so it wouldn't hurt to have someone else check the rules I
>> devised. See the README at https://github.com/richb-hanover/netperfclean
> 
> A couple of alternatives to custom scripts are fail2ban and the 
> rate-limiting modules available for iptables such as hashlimit and recent. 
> I haven't used fail2ban for rate-limiting so I'm not sure if it's the right 
> tool for that but it monitors log files to add iptables rules for 
> short-term banning. It's not hard to add your own log monitoring rule. I 
> haven't used the iptables modules but they look like a natural solution for 
> this.
> 
> <https://poorlydocumented.com/2017/08/understanding-iptables-hashlimit-module/>
> 
> <https://serverfault.com/questions/682045/source-ip-rate-limiting-in-iptables-hashlimit-vs-recent>
> 
> Instead of using a unique iptables rule for each blocklist member, I 
> suggest using an ipset. (I use firewalld as a front-end to iptables so I 
> let it manage my ipsets, but you can also install ipset's service for use 
> with raw iptables to save and restore the sets across boots.) Your block 
> rule could be as simple as this:
> 
> iptables -I INPUT 1 -p tcp --dport netperf -m set --match-set 
> NetPerfAbusers src -m conntrack --ctstate NEW -j DROP

Thanks for these thoughts. I looked briefly at rate-limiting schemes, but 
didn't see a good way for them to distinguish good users from bad:

- Good users (who are setting up their SQM, or testing various algorithms) run 
a test (that creates 10 connections in ~10-60 seconds), tweak a parameter, then 
re-run that test, repeating until they're happy.

- Bad users who test every five minutes 24x7 create 10 connections every 300 
seconds - a slower "rate" of establishing new connections than the good guys.

The primary characteristic that distinguishes the good guys from the bad is 
that good guys *stop.*

So, my reasoning goes, I need to look at a longer time window and set a limit 
on the number of connections over the course of a day or two (not minutes or 
hours). And that's the genesis of my question to the group:

What is *your* pattern of testing? How many successive tests are you 
likely to make over the course of a day? 

I'm also aware of ipset, which I take to be an optimized alternative to 
searching a long set of iptables rules (true?) I don't believe that my OpenVZ 
VPS has kernel support for this, so as long as the long-list-of-rules seems to 
work well, I'm going to leave it alone.

That's my thinking, but please let me know if I'm missing something. Thanks 
again.



___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] netperf server news

[Kenneth Porter]

--On Tuesday, October 06, 2020 7:52 AM -0400 Rich Brown 
 wrote:



3) I would be pleased to get comments on the set of scripts. I'm a newbie
at iptables, so it wouldn't hurt to have someone else check the rules I
devised. See the README at https://github.com/richb-hanover/netperfclean


A couple of alternatives to custom scripts are fail2ban and the 
rate-limiting modules available for iptables such as hashlimit and recent. 
I haven't used fail2ban for rate-limiting so I'm not sure if it's the right 
tool for that but it monitors log files to add iptables rules for 
short-term banning. It's not hard to add your own log monitoring rule. I 
haven't used the iptables modules but they look like a natural solution for 
this.






Instead of using a unique iptables rule for each blocklist member, I 
suggest using an ipset. (I use firewalld as a front-end to iptables so I 
let it manage my ipsets, but you can also install ipset's service for use 
with raw iptables to save and restore the sets across boots.) Your block 
rule could be as simple as this:


iptables -I INPUT 1 -p tcp --dport netperf -m set --match-set 
NetPerfAbusers src -m conntrack --ctstate NEW -j DROP



___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] netperf server news

[Dave Collier-Brown]

Number?

One or two that were complete and successful.  Maybe five unsuccessful tries.

More of a sequence than a time-period: u{1-5}s{1-2}

--daev


On 2020-10-06 4:40 p.m., Rich Brown wrote:

Thanks for the feedback. Some responses:

1) I'm glad that people are seeing reasonable speeds from the VPS. (I don't 
know what I can do to make it go faster, so I'm relieved...)

2) I don't think I posed the right question for the number-of-tests threshold. (Most of 
the responses were like, "Sure, that sounds like enough..." Let me reframe the 
question:

   In your normal testing/troubleshooting process, what is the maximum 
number of tests YOU might need to run in any two-day period?

3) If you can't get through to netperf.bufferbloat.net, send me your IP address 
because it might have been blacklisted.

Thanks!

Rich




On Oct 6, 2020, at 6:52 AM, Rich Brown 
 wrote:

To the Bloat list,

I had some time, so I looked into what it might take to keep the netperf.bufferbloat.net 
server on-line in the face of an unwitting "DDoS" attack - automated scripts 
that run tests every 5 minutes 24x7. The problem was that these tests would blow through 
my 4TB/month bandwidth allocation in a few days.

In the past, I had been irregularly running a set of scripts to count incoming 
netperf connections and blacklist (in iptables) those whose counts were too 
high. This wasn't good enough: it wasn't keeping up with the tidal wave of 
connections.

Last week, I revised those scripts to work as a cron job. The current parameters 
are: run the script every hour; process the last two days' of kern.log files; look 
for > 500 connections; drop those addresses in iptables.

There are currently 479 addresses blacklisted in iptables (that explains why 
the bandwidth was being consumed so quickly). There are only a few new 
addresses being added per day, so it seems that we have flushed out most of the 
abusers.

My questions for this august group:

1) The server at netperf.bufferbloat.net is up and running. I get full rate 
speed from my 7mbps DSL circuit, but that's not much of a test. I would be 
interested to hear your results.

2) The current threshold comes from this estimate: most speed tests use 10 connections: 5 
connections up and 5 down. So 500 connections would permit about 50 tests over the course 
of two days. Is that enough for "real research"? (If you need more, I can add 
your address to my whitelist file...)

3) I would be pleased to get comments on the set of scripts. I'm a newbie at 
iptables, so it wouldn't hurt to have someone else check the rules I devised. 
See the README at https://github.com/richb-hanover/netperfclean

Thanks.

Rich




___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat


--
David Collier-Brown, | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
dave.collier-br...@indexexchange.com
 |  -- Mark Twain



CONFIDENTIALITY NOTICE AND DISCLAIMER : This telecommunication, including any 
and all attachments, contains confidential information intended only for the 
person(s) to whom it is addressed. Any dissemination, distribution, copying or 
disclosure is strictly prohibited and is not a waiver of confidentiality. If 
you have received this telecommunication in error, please notify the sender 
immediately by return electronic mail and delete the message from your inbox 
and deleted items folders. This telecommunication does not constitute an 
express or implied agreement to conduct transactions by electronic means, nor 
does it constitute a contract offer, a contract amendment or an acceptance of a 
contract offer. Contract terms contained in this telecommunication are subject 
to legal review and the completion of formal documentation and are not binding 
until same is confirmed in writing and has been signed by an authorized 
signatory.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] netperf server news

[Rich Brown]

Thanks for the feedback. Some responses:

1) I'm glad that people are seeing reasonable speeds from the VPS. (I don't 
know what I can do to make it go faster, so I'm relieved...)

2) I don't think I posed the right question for the number-of-tests threshold. 
(Most of the responses were like, "Sure, that sounds like enough..." Let me 
reframe the question: 

In your normal testing/troubleshooting process, what is the maximum 
number of tests YOU might need to run in any two-day period?

3) If you can't get through to netperf.bufferbloat.net, send me your IP address 
because it might have been blacklisted.

Thanks!

Rich


> On Oct 6, 2020, at 6:52 AM, Rich Brown  wrote:
> 
> To the Bloat list,
> 
> I had some time, so I looked into what it might take to keep the 
> netperf.bufferbloat.net server on-line in the face of an unwitting "DDoS" 
> attack - automated scripts that run tests every 5 minutes 24x7. The problem 
> was that these tests would blow through my 4TB/month bandwidth allocation in 
> a few days.
> 
> In the past, I had been irregularly running a set of scripts to count 
> incoming netperf connections and blacklist (in iptables) those whose counts 
> were too high. This wasn't good enough: it wasn't keeping up with the tidal 
> wave of connections.
> 
> Last week, I revised those scripts to work as a cron job. The current 
> parameters are: run the script every hour; process the last two days' of 
> kern.log files; look for > 500 connections; drop those addresses in iptables.
> 
> There are currently 479 addresses blacklisted in iptables (that explains why 
> the bandwidth was being consumed so quickly). There are only a few new 
> addresses being added per day, so it seems that we have flushed out most of 
> the abusers.
> 
> My questions for this august group:
> 
> 1) The server at netperf.bufferbloat.net is up and running. I get full rate 
> speed from my 7mbps DSL circuit, but that's not much of a test. I would be 
> interested to hear your results.
> 
> 2) The current threshold comes from this estimate: most speed tests use 10 
> connections: 5 connections up and 5 down. So 500 connections would permit 
> about 50 tests over the course of two days. Is that enough for "real 
> research"? (If you need more, I can add your address to my whitelist file...)
> 
> 3) I would be pleased to get comments on the set of scripts. I'm a newbie at 
> iptables, so it wouldn't hurt to have someone else check the rules I devised. 
> See the README at https://github.com/richb-hanover/netperfclean
> 
> Thanks.
> 
> Rich
> 

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] netperf server news

[Colin Dearborn]

1): This hits what I'd expect my connection to be (or close enough anyway):
./betterspeedtest.sh
2020-10-06 12:53:54 Testing against netperf.bufferbloat.net (ipv4) with 5 
simultaneous sessions while pinging gstatic.com (60 seconds in each direction)
.
 Download: 926.27 Mbps
  Latency: (in msec, 61 pings, 0.00% packet loss)
  Min: 23.200
10pct: 25.500
   Median: 31.100
  Avg: 30.503
90pct: 34.100
  Max: 38.800
...
   Upload: 103.50 Mbps
  Latency: (in msec, 63 pings, 0.00% packet loss)
  Min: 22.700
10pct: 23.900
   Median: 28.600
  Avg: 30.233
90pct: 33.400
  Max: 112.000

2) That sounds like plenty to me.


-Original Message-
From: Bloat  On Behalf Of Rich Brown
Sent: October 6, 2020 4:53 AM
To: bloat 
Cc: Richard E. Brown 
Subject: [Bloat] netperf server news

  CAUTION: This email is from an external source. Do not click links or open 
attachments unless you recognize the sender and know the content is safe.

To the Bloat list,

I had some time, so I looked into what it might take to keep the 
netperf.bufferbloat.net server on-line in the face of an unwitting "DDoS" 
attack - automated scripts that run tests every 5 minutes 24x7. The problem was 
that these tests would blow through my 4TB/month bandwidth allocation in a few 
days.

In the past, I had been irregularly running a set of scripts to count incoming 
netperf connections and blacklist (in iptables) those whose counts were too 
high. This wasn't good enough: it wasn't keeping up with the tidal wave of 
connections.

Last week, I revised those scripts to work as a cron job. The current 
parameters are: run the script every hour; process the last two days' of 
kern.log files; look for > 500 connections; drop those addresses in iptables.

There are currently 479 addresses blacklisted in iptables (that explains why 
the bandwidth was being consumed so quickly). There are only a few new 
addresses being added per day, so it seems that we have flushed out most of the 
abusers.

My questions for this august group:

1) The server at netperf.bufferbloat.net is up and running. I get full rate 
speed from my 7mbps DSL circuit, but that's not much of a test. I would be 
interested to hear your results.

2) The current threshold comes from this estimate: most speed tests use 10 
connections: 5 connections up and 5 down. So 500 connections would permit about 
50 tests over the course of two days. Is that enough for "real research"? (If 
you need more, I can add your address to my whitelist file...)

3) I would be pleased to get comments on the set of scripts. I'm a newbie at 
iptables, so it wouldn't hurt to have someone else check the rules I devised. 
See the README at https://github.com/richb-hanover/netperfclean

Thanks.

Rich

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] netperf server news

[Sebastian Moeller]

Dear Rich,

first, thanks for supplying that service.

> On Oct 6, 2020, at 12:52, Rich Brown  wrote:
> 
> To the Bloat list,
> 
> I had some time, so I looked into what it might take to keep the 
> netperf.bufferbloat.net server on-line in the face of an unwitting "DDoS" 
> attack - automated scripts that run tests every 5 minutes 24x7. The problem 
> was that these tests would blow through my 4TB/month bandwidth allocation in 
> a few days.
> 
> In the past, I had been irregularly running a set of scripts to count 
> incoming netperf connections and blacklist (in iptables) those whose counts 
> were too high. This wasn't good enough: it wasn't keeping up with the tidal 
> wave of connections.
> 
> Last week, I revised those scripts to work as a cron job. The current 
> parameters are: run the script every hour; process the last two days' of 
> kern.log files; look for > 500 connections; drop those addresses in iptables.
> 
> There are currently 479 addresses blacklisted in iptables (that explains why 
> the bandwidth was being consumed so quickly). There are only a few new 
> addresses being added per day, so it seems that we have flushed out most of 
> the abusers.
> 
> My questions for this august group:
> 
> 1) The server at netperf.bufferbloat.net is up and running. I get full rate 
> speed from my 7mbps DSL circuit, but that's not much of a test. I would be 
> interested to hear your results.

From work:

bash-3.2$ ./betterspeedtest.sh
2020-10-06 14:46:19 Testing against netperf.bufferbloat.net (ipv4) with 5 
simultaneous sessions while pinging gstatic.com (60 seconds in each direction)
.
 Download:  Mbps
  Latency: (in msec, 1 pings, 0.00% packet loss)
  Min: 6.868 
10pct: 0.000 
   Median: 0.000 
  Avg: 6.868 
90pct: 0.000 
  Max: 6.868
.
   Upload:  309.67 Mbps
  Latency: (in msec, 61 pings, 0.00% packet loss)
  Min: 6.644 
10pct: 6.730 
   Median: 7.289 
  Avg: 7.385 
90pct: 7.941 
  Max: 9.980
Press any key to continue...

bash-3.2$ ./betterspeedtest.sh
2020-10-06 14:49:33 Testing against netperf.bufferbloat.net (ipv4) with 5 
simultaneous sessions while pinging gstatic.com (60 seconds in each direction)

 Download:  0 Mbps
  Latency: (in msec, 80 pings, 0.00% packet loss)
  Min: 6.583 
10pct: 6.637 
   Median: 6.674 
  Avg: 6.694 
90pct: 6.743 
  Max: 7.204

   Upload:  0 Mbps
  Latency: (in msec, 80 pings, 0.00% packet loss)
  Min: 6.555 
10pct: 6.622 
   Median: 6.667 
  Avg: 6.687 
90pct: 6.742 
  Max: 7.218
Press any key to continue...

So there seems to be an issue with the Download test, from home I currently get 
0/0 for both Upload/download Maybe I just made it on the block list (not 
that I remember trying to reach that server in the last weeks at all).

Running flent's rrul_cs8 manually against netperf.bufferbloat.net gave me 
around 80/25 which seems  believable.


> 
> 2) The current threshold comes from this estimate: most speed tests use 10 
> connections: 5 connections up and 5 down. So 500 connections would permit 
> about 50 tests over the course of two days. Is that enough for "real 
> research"? (If you need more, I can add your address to my whitelist file...)

I think 50 tests is quite generous, that is more than one test every 
hour for two days ;)

> 
> 3) I would be pleased to get comments on the set of scripts. I'm a newbie at 
> iptables, so it wouldn't hurt to have someone else check the rules I devised. 
> See the README at https://github.com/richb-hanover/netperfclean

Outside of my area of expertise

Best Reards
Sebastian

> 
> Thanks.
> 
> Rich
> 
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

[Bloat] netperf server news

[Rich Brown]

To the Bloat list,

I had some time, so I looked into what it might take to keep the 
netperf.bufferbloat.net server on-line in the face of an unwitting "DDoS" 
attack - automated scripts that run tests every 5 minutes 24x7. The problem was 
that these tests would blow through my 4TB/month bandwidth allocation in a few 
days.

In the past, I had been irregularly running a set of scripts to count incoming 
netperf connections and blacklist (in iptables) those whose counts were too 
high. This wasn't good enough: it wasn't keeping up with the tidal wave of 
connections.

Last week, I revised those scripts to work as a cron job. The current 
parameters are: run the script every hour; process the last two days' of 
kern.log files; look for > 500 connections; drop those addresses in iptables.

There are currently 479 addresses blacklisted in iptables (that explains why 
the bandwidth was being consumed so quickly). There are only a few new 
addresses being added per day, so it seems that we have flushed out most of the 
abusers.

My questions for this august group:

1) The server at netperf.bufferbloat.net is up and running. I get full rate 
speed from my 7mbps DSL circuit, but that's not much of a test. I would be 
interested to hear your results.

2) The current threshold comes from this estimate: most speed tests use 10 
connections: 5 connections up and 5 down. So 500 connections would permit about 
50 tests over the course of two days. Is that enough for "real research"? (If 
you need more, I can add your address to my whitelist file...)

3) I would be pleased to get comments on the set of scripts. I'm a newbie at 
iptables, so it wouldn't hurt to have someone else check the rules I devised. 
See the README at https://github.com/richb-hanover/netperfclean

Thanks.

Rich

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat