Re: [tor-relays] Detecting Network Attack [re: exit synflooded]
> From: igor.n.mitrofa...@gmail.com > Sorry for the spam. One more link to a tuning guide that I have found Contributions are good but since we're on the same list how about trimming quoted messages? ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Detecting Network Attack [re: exit synflooded]
On Sat, Nov 25, 2017 at 5:15 PM, teor wrote: > need a privacy-preserving aggregation scheme > (Otherwise, anyone who can remotely trigger a rare protocol > violation can find out which relays a client or onion service is using.) The above don't necessarily lead to each other. > scheme in Tor so we can do these counts That's thinking of 'in tor' code, which is good way and project to see some things only visible there, and way to count and submit them over tor. I'm more thinking using external tools to watch the network interface itself... Attackers will read / fuzz the source code till they exploit via tor's open ports anyway. Though it could still be good to instrument those ports with both tor protocol analyzer, and a raw packet statistical analyzer / classifier to see what's incoming. Instrumenting the IP itself to look for debilitating inbound packet bursts from the internet indicating node pruning segmentation attacks. Would be interesting discovery. Though attackers might find the method redundant given already ways to deanon hidden services and fewer to deanon users. And all the usual IDS type of tools that could be deployed and collected to see who / what is probing away at the network itself and how. Might want to look for modulation patterns in OR traffic proving existance of certain known attack methods. Not talking about content of exit traffic in any of this. It's exposing attacks from clearnet, not users of tor. Operators could opt in. Prebuilt tool packages could be created. Someone with a handful of relays could always do the research project on their own, and like silent attackers, may already be. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Detecting Network Attack [re: exit synflooded]
> kernel: nf_conntrack: table full, dropping packet If rules are dropping exit traffic based on other than traffic content, it's very hard to say other users are not adversly affected with the same, likely quite unsophisticated, hammer. And doing it based on content usually comes with major legal hurdles, besides being arbitrary. And both ways can get you dropped with badexit flag. Further, kernel dropping of packets is not signaled back into tor daemons for exitpolicy management therein, much less back to clients to avoid the censorship. And dropped packets hurts performance. Exitpolicy reject is the preferred method. Don't like the exit traffic, don't advertise to clients that you will carry it outbound in the first place. Please move this talk about tor exit traffic to a new thread or put it back to the first one where it came from. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Detecting Network Attack [re: exit synflooded]
There are often silly filters and SSH WARNING OMG type of things people do individually that are generally moot. That was probably the context. No time to cover various scenarios therein. Here, we're considering aggregating the entire network as one to see about what may be going on as a whole. Because tor has no vision there right now. Dogs that wake you up and a janitor to ask might be nice byproducts. Though being able to do anything about a discovery, given the potential dark makeup of any attacker, other than expose it's existance, seems less likely. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Detecting Network Attack [re: exit synflooded]
Sorry for the spam. One more link to a tuning guide that I have found useful: https://access.redhat.com/sites/default/files/attachments/20150325_network_performance_tuning.pdf On Sat, Nov 25, 2017 at 10:04 PM, Igor Mitrofanov wrote: > After reading every paper and post on sysctl.conf and iptables tuning > I could find, and reading some kernel code, I have come to a > conclusion that, while there are a few settings to tune (can share > mine, but your mileage *will* vary), most of the defaults are actually > not broken in the latest kernels, while a lot of tuning > recommendations online are outdated. > > I enjoyed this guide a lot: > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html-single/performance_tuning_guide/index. > It is written for Red Hat, but contains quite a few insights > applicable to all Linux distributions. > > - Igor > > > On Sat, Nov 25, 2017 at 3:14 PM, Dhalgren Tor wrote: >>>Well, it's still going on, and is pretty much ruining Libero :( . Running >>>CentOS 6, here. >>> >>>When it's happening it can look like this: >>> >>># netstat -n | grep -c SYN >>>17696 >> >> I run a fast exit and can offer some advice: >> >> 1) mitigate bug #18580 (also #21394); is a DNS denial-of-service and >> could be the problem. either upgrade to 0.3.2.4+ or edit resolve.conf >> per >> https://trac.torproject.org/projects/tor/wiki/doc/DnsResolver#TuningeventdnscomponentofTorDaemon >> >> also check out https://arthuredelstein.net/exits/ >> >> 2) if you continue to experience excessive outbound scanning SYNs, >> I've found that simply enabling connection tracking helps by >> implicitly limiting the rate a which connections can originate. If an >> iptables "-m state --state ESTABLISHED,RELATED -j ACCEPT" rule exists >> it will turn it on or you could modprobe the module if you don't want >> to configure incoming connection rules. >> >> some useful sysctl.conf changes, run sysctl -p after or reboot >> >> # Tor Exit tuning. >> net.ipv4.ip_local_port_range = 1025 65535 >> net.ipv4.tcp_synack_retries = 2 >> net.ipv4.tcp_max_tw_buckets = 4194304 >> net.ipv4.tcp_tw_recycle = 1 >> net.ipv4.tcp_fin_timeout = 30 >> net.ipv4.tcp_keepalive_time = 300 >> net.ipv4.tcp_keepalive_intvl = 10 >> net.ipv4.tcp_keepalive_probes = 3 >> net.netfilter.nf_conntrack_tcp_timeout_established = 1000 >> net.netfilter.nf_conntrack_checksum = 0 >> >> you might see many messages: >> >> kernel: nf_conntrack: table full, dropping packet >> >> which indicates no connection table slots were available for an >> outbound connection and that rate limiting is effected >> >> state of connection tracking appears in /proc/net/nf_conntrack >> ___ >> tor-relays mailing list >> tor-relays@lists.torproject.org >> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Detecting Network Attack [re: exit synflooded]
After reading every paper and post on sysctl.conf and iptables tuning I could find, and reading some kernel code, I have come to a conclusion that, while there are a few settings to tune (can share mine, but your mileage *will* vary), most of the defaults are actually not broken in the latest kernels, while a lot of tuning recommendations online are outdated. I enjoyed this guide a lot: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html-single/performance_tuning_guide/index. It is written for Red Hat, but contains quite a few insights applicable to all Linux distributions. - Igor On Sat, Nov 25, 2017 at 3:14 PM, Dhalgren Tor wrote: >>Well, it's still going on, and is pretty much ruining Libero :( . Running >>CentOS 6, here. >> >>When it's happening it can look like this: >> >># netstat -n | grep -c SYN >>17696 > > I run a fast exit and can offer some advice: > > 1) mitigate bug #18580 (also #21394); is a DNS denial-of-service and > could be the problem. either upgrade to 0.3.2.4+ or edit resolve.conf > per > https://trac.torproject.org/projects/tor/wiki/doc/DnsResolver#TuningeventdnscomponentofTorDaemon > > also check out https://arthuredelstein.net/exits/ > > 2) if you continue to experience excessive outbound scanning SYNs, > I've found that simply enabling connection tracking helps by > implicitly limiting the rate a which connections can originate. If an > iptables "-m state --state ESTABLISHED,RELATED -j ACCEPT" rule exists > it will turn it on or you could modprobe the module if you don't want > to configure incoming connection rules. > > some useful sysctl.conf changes, run sysctl -p after or reboot > > # Tor Exit tuning. > net.ipv4.ip_local_port_range = 1025 65535 > net.ipv4.tcp_synack_retries = 2 > net.ipv4.tcp_max_tw_buckets = 4194304 > net.ipv4.tcp_tw_recycle = 1 > net.ipv4.tcp_fin_timeout = 30 > net.ipv4.tcp_keepalive_time = 300 > net.ipv4.tcp_keepalive_intvl = 10 > net.ipv4.tcp_keepalive_probes = 3 > net.netfilter.nf_conntrack_tcp_timeout_established = 1000 > net.netfilter.nf_conntrack_checksum = 0 > > you might see many messages: > > kernel: nf_conntrack: table full, dropping packet > > which indicates no connection table slots were available for an > outbound connection and that rate limiting is effected > > state of connection tracking appears in /proc/net/nf_conntrack > ___ > tor-relays mailing list > tor-relays@lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Detecting Network Attack [re: exit synflooded]
Do not enable net.ipv4.tcp_tw_recycle: https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux#netipv4tcp_tw_recycle For ip_local_port_range, make one number even and the other one odd (i.e. 1024 and 65535). Not sure if this is still required, but won't hurt to include port 1024. Consider blacklisting the nf_conntrack module altogether and running a completely stateless firewall: https://javapipe.com/ddos/blog/iptables-ddos-protection/ https://strongarm.io/blog/linux-stateless-firewall/ On Sat, Nov 25, 2017 at 3:14 PM, Dhalgren Tor wrote: >>Well, it's still going on, and is pretty much ruining Libero :( . Running >>CentOS 6, here. >> >>When it's happening it can look like this: >> >># netstat -n | grep -c SYN >>17696 > > I run a fast exit and can offer some advice: > > 1) mitigate bug #18580 (also #21394); is a DNS denial-of-service and > could be the problem. either upgrade to 0.3.2.4+ or edit resolve.conf > per > https://trac.torproject.org/projects/tor/wiki/doc/DnsResolver#TuningeventdnscomponentofTorDaemon > > also check out https://arthuredelstein.net/exits/ > > 2) if you continue to experience excessive outbound scanning SYNs, > I've found that simply enabling connection tracking helps by > implicitly limiting the rate a which connections can originate. If an > iptables "-m state --state ESTABLISHED,RELATED -j ACCEPT" rule exists > it will turn it on or you could modprobe the module if you don't want > to configure incoming connection rules. > > some useful sysctl.conf changes, run sysctl -p after or reboot > > # Tor Exit tuning. > net.ipv4.ip_local_port_range = 1025 65535 > net.ipv4.tcp_synack_retries = 2 > net.ipv4.tcp_max_tw_buckets = 4194304 > net.ipv4.tcp_tw_recycle = 1 > net.ipv4.tcp_fin_timeout = 30 > net.ipv4.tcp_keepalive_time = 300 > net.ipv4.tcp_keepalive_intvl = 10 > net.ipv4.tcp_keepalive_probes = 3 > net.netfilter.nf_conntrack_tcp_timeout_established = 1000 > net.netfilter.nf_conntrack_checksum = 0 > > you might see many messages: > > kernel: nf_conntrack: table full, dropping packet > > which indicates no connection table slots were available for an > outbound connection and that rate limiting is effected > > state of connection tracking appears in /proc/net/nf_conntrack > ___ > tor-relays mailing list > tor-relays@lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Detecting Network Attack [re: exit synflooded]
grarpamp wrote: > On Fri, Nov 24, 2017 at 6:23 PM, wrote: > > Was anyone else's exit being synflooded yesterday and today? > > There could be a combined monitoring array deployed > among all nodes that might start to answer these questions. > And further alert on all sorts of interesting network attacks > launched at Tor. Software exploits... segmentation / modulation > attacks... etc. This is perimeter security 101, and Tor > currently has no fence, no guard dogs, not even a janitor > you can ask the next day. That's beyond bad. > And big oppurtunity for new research project. Not that long ago, grarpamp, you were vehemently--to the point of hysterics--opposed to fences, even with gate passes for authorized entry. Now you complain about the lack fences (without mention of gate passes, however), guard dogs, etc. What changed your mind? Scott Bennett, Comm. ASMELG, CFIAG ** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * ** * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * *-- Gov. John Hancock, New York Journal, 28 January 1790 * ** ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Detecting Network Attack [re: exit synflooded]
>Well, it's still going on, and is pretty much ruining Libero :( . Running >CentOS 6, here. > >When it's happening it can look like this: > ># netstat -n | grep -c SYN >17696 I run a fast exit and can offer some advice: 1) mitigate bug #18580 (also #21394); is a DNS denial-of-service and could be the problem. either upgrade to 0.3.2.4+ or edit resolve.conf per https://trac.torproject.org/projects/tor/wiki/doc/DnsResolver#TuningeventdnscomponentofTorDaemon also check out https://arthuredelstein.net/exits/ 2) if you continue to experience excessive outbound scanning SYNs, I've found that simply enabling connection tracking helps by implicitly limiting the rate a which connections can originate. If an iptables "-m state --state ESTABLISHED,RELATED -j ACCEPT" rule exists it will turn it on or you could modprobe the module if you don't want to configure incoming connection rules. some useful sysctl.conf changes, run sysctl -p after or reboot # Tor Exit tuning. net.ipv4.ip_local_port_range = 1025 65535 net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_max_tw_buckets = 4194304 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_fin_timeout = 30 net.ipv4.tcp_keepalive_time = 300 net.ipv4.tcp_keepalive_intvl = 10 net.ipv4.tcp_keepalive_probes = 3 net.netfilter.nf_conntrack_tcp_timeout_established = 1000 net.netfilter.nf_conntrack_checksum = 0 you might see many messages: kernel: nf_conntrack: table full, dropping packet which indicates no connection table slots were available for an outbound connection and that rate limiting is effected state of connection tracking appears in /proc/net/nf_conntrack ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Detecting Network Attack [re: exit synflooded]
> On 26 Nov 2017, at 07:14, grarpamp wrote: > > The subject of this new thread is detecting network > attack upon tor network / relays itself. Nick Mathewson has mentioned wanting to do this for Tor protocol violations. But we need a privacy-preserving aggregation scheme in Tor so we can do these counts safely. (Otherwise, anyone who can remotely trigger a rare protocol violation can find out which relays a client or onion service is using.) When we create this list, we will also think about what other kinds of attacks on the network we can reliably detect and monitor. We're limited in the number of counters we can create for these events, and they must track integer counts. Do you have a "top 5" list of attacks we could detect this way? T ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Detecting Network Attack [re: exit synflooded]
The subject of this new thread is detecting network attack upon tor network / relays itself. You report is users using tor's exits / exit traffic from relays, which would be excluded from such monitoring, most absolutely in any identifiable manner. If the exit traffic bothers you, exitpolicy reject or whatever else makes up the issue. It should go away after time for new descriptor to pass around. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Detecting Network Attack [re: exit synflooded]
I went and added a reject for exit to port and HUPed the process. Maybe this is the fix! :) ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Detecting Network Attack [re: exit synflooded]
Well, it's still going on, and is pretty much ruining Libero :( . Running CentOS 6, here. Actually, I think from what I'm seeing that it may not exactly be a synflood targeting Libero. I think Libero may be being (ab)used to do massive portscanning or similar. Image should be visible below - it's normal statistics for the 18th through say, part of the 21st, messed up 22ed - 24th and on the 24th me trying different things in an attempt to mitigate (graph is from yesterday). Look at the SYN_SENT2 maximum. When it's happening it can look like this: # netstat -n | grep -c SYN 17696 # Also one tiny part of the netstat looked like this: tcp0 1 64.113.32.29:33354 munged>:81SYN_SENT tcp0 1 64.113.32.29:39659 munged>: SYN_SENT tcp0 1 64.113.32.29:44247 munged>:87SYN_SENT tcp0 1 64.113.32.29:42038 munged>: SYN_SENT tcp0 1 64.113.32.29:42077 munged>:83 SYN_SENT tcp0 1 64.113.32.29:36282 munged>: SYN_SENT tcp0 1 64.113.32.29:46023 munged>: SYN_SENT Port is supposedly opened up for listen by a virus. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] Detecting Network Attack [re: exit synflooded]
On Fri, Nov 24, 2017 at 6:23 PM, wrote: > Was anyone else's exit being synflooded yesterday and today? There could be a combined monitoring array deployed among all nodes that might start to answer these questions. And further alert on all sorts of interesting network attacks launched at Tor. Software exploits... segmentation / modulation attacks... etc. This is perimeter security 101, and Tor currently has no fence, no guard dogs, not even a janitor you can ask the next day. That's beyond bad. And big oppurtunity for new research project. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
