Re: [tor-relays] upcoming directory authority changes

[Toralf Förster]

On 12/6/22 19:44, Roger Dingledine wrote:

But it seems
like this role separation never quite matches up well to the security
issues that arise in practice, whereas it definitely adds complexity
both to the design and to operation. This piece of the design could use
some new ideas.


So the concept of off-line master keys doesn't help too much nowadays?

--
Toralf

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] upcoming directory authority changes

[Toralf Förster]

On 12/6/22 19:44, Roger Dingledine wrote:

  We
could start by encouraging directory authority operators to participate
in the monthly virtual relay operator meetups.


I'd appreciate it.

--
Toralf



OpenPGP_signature
Description: OpenPGP digital signature
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] upcoming directory authority changes

[Roger Dingledine]

Hello fellow relay operators,

Later today (Tuesday) we plan to do a synchronized shift where we
make two configuration changes on the directory authorities. The goal
will be to make these changes while maintaining the right threshold of
signatures so relays and users still get a safe network status consensus
that they trust.

For background, Tor uses a threshold consensus design, where as long as
a majority of directory authorities are behaving properly, all users get
the same accurate view of the Tor network. You can learn more about the
design in the bottom half of
https://support.torproject.org/about/key-management/.

Specifically, we plan to make these changes:

(1) Temporarily remove Faravahar from the set of directory authorities,
until Sina finds a new good home for it. We weren't all comfortable with
its previous hosting location inside Team Cymru's network (see the bottom
of https://blog.torproject.org/role-tor-project-board-conflicts-interest/
for more details), so to be extra safe we are making it no longer
participate in the consensus process until it is reborn in its future
home. You can read more about Sina and why we are excited to keep him
involved in the Tor community at
https://blog.torproject.org/volunteer-spotlight-sina-rabbani-helps-activists-avoid-government-censorship/.

(2) Rotate to fresh identity keys for moria1, the directory authority
that I run. In early November 2022 there was a remote break-in to the
computer running moria1. Based on the evidence and the type of attack,
I believe it was a standard automated attack -- that is, I think they
weren't targeting the directory authority and also they never realized it
*was* a directory authority. But to be extra safe, we decided to rotate
to a fresh set of keys. I was also in the middle of a planned move to
better hardware, so overall it was good timing for a fresh new start.

I should note that for both moria1 and Faravahar, we have no evidence
of any misuse of their keys. But more importantly, the threshold
consensus design keeps Tor users safe *even if* we are underestimating
what happened. Tor users and relays demand a consensus signed by a
majority of the directory authorities, and right now that's five out of
eight. Tor's security/privacy/anonymity properties remained safe before,
during, and after these changes.

Some closing thoughts and details:

* We actually already removed Faravahar in the Tor git repository and
in the Tor release 0.4.7.11 (which came out Nov 10 and has been in
Tor Browser since Nov 22), so modern clients and relays are already
demanding signatures from five authorities that aren't Faravahar. The
change we'll do today is to make the other directory authorities stop
incorporating its vote and signature into the consensus. Doing these
changes at different times opened the window for surprising but harmless
log warnings like https://bugs.torproject.org/tpo/core/tor/40725. We
could see similar issues with the new moria1 key, and if so they should
go away once there's a new release with the new key and you have upgraded.

* In my opinion there is a good argument for every directory authority
making fresh keys every few years anyway, especially in a world with
sophisticated attackers who might try to obtain keys and not leave any
traces. So for example when we bring Faravahar back I think it should
be with entirely new keys.

* Secure consensus designs have become much better in the past decade,
in large part from all the Bitcoin enthusiasm. Our current design was
always intended to be a placeholder until we got something better. And
our friends in the PETS research community have been exploring edge
cases where evil directory authorities can create mischief even when
they're slightly less than a full majority. So while we're triaging
the usual fires and priorities, we shouldn't forget about improving the
directory design.

* Directory authority keys already have a notion of an offline long-term
identity with shorter-lifetime online keys that expire periodically,
with the goal of limiting the future impact of a compromise. But it seems
like this role separation never quite matches up well to the security
issues that arise in practice, whereas it definitely adds complexity
both to the design and to operation. This piece of the design could use
some new ideas.

* Lastly, the directory authority operator community is a community too,
and the security of the Tor network relies on social trust and cooperation
between these fine people. We used to have directory authority operator
meetups in person at security conferences and at Tor dev meetings, to
maintain the social and community connections. Covid put a stop to that
along with so many other things, and we should work to bring it back. We
could start by encouraging directory authority operators to participate
in the monthly virtual relay operator meetups.

Thanks for reading all the way to the end / hope this level of detail helps,
--Roger

Re: [tor-relays] inet_csk_bind_conflict

[Chris]

Excellent. Thank you.

Yes a blanket iotables rule is not going to work well in this set up as
it pools all connections to all IP addresses into one. So if we accept 4
connections to port 443, a blanket iptables rules accepts 4 connections
to all IP addresses combined and drops everything else and of course
that brings your server to a halt.

In another thread in this mailing list, they had the same situation and
I put a script together yesterday that you're welcome to try if you
wish. Not sure if they've tried it yet or what the result has been. But
the script is set up to apply the rules to two IP addresses at a time
and leave the rest alone. So you can apply to two addresses on your
server, assess the result and then either expand to the rest or stop
altogether.

The script makes a back up of your existing iptables rules. All you have
to do is restore it and everything goes back to how it was without
having to reboot. It also specifically uses the mangle table and
PREROUTING and it won't interfere with your existing rules. That should
reduce the number of used ports as well. Flushing the mangle table will
also get rid of these rules and you're back to how it was before.

You can get it here:

https://raw.githubusercontent.com/Enkidu-6/tor-ddos/dev/multiple/multi-addr.sh

Simply choose two of your IP addresses and the ORPort for each and run
the script.

If it does what you expect it to do, all you have to do is to change the
IP Addresses and run the script again until all your addresses are
covered. Please save the iptables backup somewhere else as the second
time you run the script, the original back up will be overwritten.

If one of your IP addresses has two ORPorts, the above script won't work
and you should use the script below:

https://raw.githubusercontent.com/Enkidu-6/tor-ddos/dev/multiple/two-or.sh

Best of luck and I hope this helps.

On 12/5/2022 3:48 PM, Christopher Sheats wrote:
>> May I ask what your set up is?
>> Are you running your relays on separate VMs on the main system or are
>> you using a different set up like having all IP addresses on the same OS
>> and using OutboundBindAddress , routing, etc... to separate them? If I
>> know more, I might be able to make a script specific to your set up.
>
> Thank you. Yes, of course.
>
> Ubuntu server 22.04 runs on bare metal. Ansible-relayor manages 20
> exit relays on each system. Netplan has each IP individually listed
> (sub-divided as a /25 per server from within a dedicated /24,
> similarly for v6 addresses). I believe an available IP is randomly
> picked by ansible-relayor and used statically in each torrc file.
>
> Here is an example torrc:
>
> # ansible-relayor generated torrc configuration file
>
> # Note: manual changes will be OVERWRITTEN on the next
> ansible-playbook run
>
>
> OfflineMasterKey 1
>
> RunAsDaemon 0
>
> Log notice syslog
>
> OutboundBindAddress 23.129.64.130
>
> SocksPort 0
>
> User _tor-23.129.64.130_443
>
> DataDirectory /var/lib/tor-instances/23.129.64.130_443
>
> ORPort 23.129.64.130:443
>
> ORPort [2620:18c:0:192::130]:443
>
> OutboundBindAddress [2620:18c:0:192::130]
>
>
> DirPort 23.129.64.130:80
>
> Address 23.129.64.130
>
>
> SyslogIdentityTag 23.129.64.130_443
>
>
> ControlSocket /var/run/tor-instances/23.129.64.130_443/control
> GroupWritable RelaxDirModeCheck
>
>
> Nickname ageis
>
> ContactInfo url:emeraldonion.org proof:uri-rsa ciissversion:2
> t...@emeraldonion.org
>
>
> Sandbox 1
>
> NoExec 1
>
>
> # we are an exit relay!
>
> ExitRelay 1
>
> IPv6Exit 1
>
> DirPort [2620:18c:0:192::130]:80 NoAdvertise
>
> DirPortFrontPage /etc/tor/instances/tor-exit-notice.html
>
>
>
> ExitPolicy reject 23.129.64.128/25:*,reject6
> [2613:18c:0:192::]/64:*,accept *:*,accept6 *:*
>
>
>
> MyFamily 
>
> # end of torrc
>
>
>
> --
> Christopher Sheats (yawnbox)
> Executive Director
> Emerald Onion
> Signal: +1 206.739.3390
> Website: https://emeraldonion.org/
> Mastodon: https://digitalcourage.social/@EmeraldOnion/
>
>
>
>
>> On Dec 4, 2022, at 10:08 PM, Chris  wrote:
>>
>> Sorry to hear it wasn't much help. Even though the additions I suggested
>> didn't help they certainly couldn't cause any harm and can't be
>> responsible for the drops in traffic.
>>
>> As for the torutils scripts, I'm sure toralf would be able to better
>> investigate that but I have a feeling you have a certain set up that
>> might not have worked with the script. May I ask what your set up is?
>> Are you running your relays on separate VMs on the main system or are
>> you using a different set up like having all IP addresses on the same OS
>> and using OutboundBindAddress , routing, etc... to separate them? If I
>> know more, I might be able to make a script specific to your set up.
>>
>> On 12/3/2022 2:07 PM, Christopher Sheats wrote:
>>> Hello,
>>>
>>> Thank you for this information. After 24-hours of testing, these
>>> configurations brought Tor to a halt.
>>>
>>> At first I started with the sysctl modifications. After a few hours
>>> with just 

Re: [tor-relays] inet_csk_bind_conflict

[Christopher Sheats]

> May I ask what your set up is?
> Are you running your relays on separate VMs on the main system or are
> you using a different set up like having all IP addresses on the same OS
> and using OutboundBindAddress , routing, etc... to separate them? If I
> know more, I might be able to make a script specific to your set up.

Thank you. Yes, of course.

Ubuntu server 22.04 runs on bare metal. Ansible-relayor manages 20 exit relays 
on each system. Netplan has each IP individually listed (sub-divided as a /25 
per server from within a dedicated /24, similarly for v6 addresses). I believe 
an available IP is randomly picked by ansible-relayor and used statically in 
each torrc file.

Here is an example torrc:

# ansible-relayor generated torrc configuration file
# Note: manual changes will be OVERWRITTEN on the next ansible-playbook run

OfflineMasterKey 1
RunAsDaemon 0
Log notice syslog
OutboundBindAddress 23.129.64.130
SocksPort 0
User _tor-23.129.64.130_443
DataDirectory /var/lib/tor-instances/23.129.64.130_443
ORPort 23.129.64.130:443
ORPort [2620:18c:0:192::130]:443
OutboundBindAddress [2620:18c:0:192::130]

DirPort 23.129.64.130:80
Address 23.129.64.130

SyslogIdentityTag 23.129.64.130_443

ControlSocket /var/run/tor-instances/23.129.64.130_443/control GroupWritable 
RelaxDirModeCheck

Nickname ageis
ContactInfo url:emeraldonion.org proof:uri-rsa ciissversion:2 
t...@emeraldonion.org

Sandbox 1
NoExec 1

# we are an exit relay!
ExitRelay 1
IPv6Exit 1
DirPort [2620:18c:0:192::130]:80 NoAdvertise
DirPortFrontPage /etc/tor/instances/tor-exit-notice.html


ExitPolicy reject 23.129.64.128/25:*,reject6 [2613:18c:0:192::]/64:*,accept 
*:*,accept6 *:*


MyFamily 
# end of torrc


--
Christopher Sheats (yawnbox)
Executive Director
Emerald Onion
Signal: +1 206.739.3390
Website: https://emeraldonion.org/
Mastodon: https://digitalcourage.social/@EmeraldOnion/




> On Dec 4, 2022, at 10:08 PM, Chris  wrote:
> 
> Sorry to hear it wasn't much help. Even though the additions I suggested
> didn't help they certainly couldn't cause any harm and can't be
> responsible for the drops in traffic.
> 
> As for the torutils scripts, I'm sure toralf would be able to better
> investigate that but I have a feeling you have a certain set up that
> might not have worked with the script. May I ask what your set up is?
> Are you running your relays on separate VMs on the main system or are
> you using a different set up like having all IP addresses on the same OS
> and using OutboundBindAddress , routing, etc... to separate them? If I
> know more, I might be able to make a script specific to your set up.
> 
> On 12/3/2022 2:07 PM, Christopher Sheats wrote:
>> Hello,
>> 
>> Thank you for this information. After 24-hours of testing, these
>> configurations brought Tor to a halt.
>> 
>> At first I started with the sysctl modifications. After a few hours
>> with just that, there was no improvement in ~75%
>> inet_csk_bind_conflict utilization. I then installed Torutils for both
>> IPv4 and IPv6. After only a couple of hours, Tor dropped to below 15
>> Mbps across both servers (40 relays). 16 hours later, Tor dropped
>> below 2 Mbps.
>> 
>> I've removed all of these new settings and restarted.
>> 
>> --
>> Christopher Sheats (yawnbox)
>> Executive Director
>> Emerald Onion
>> Signal: +1 206.739.3390
>> Website: https://emeraldonion.org/
>> Mastodon: https://digitalcourage.social/@EmeraldOnion/
>> 
>> 
>> 
>> 
>>> On Dec 2, 2022, at 7:30 AM, Chris  wrote:
>>> 
>>> Hi,
>>> 
>>> As I'm sure you've already gathered, your system is maxing out trying to
>>> deal with all the connection requests. When inet_csk_get_port is called
>>> and the port is found to be occupied then inet_csk_bind_conflict is
>>> called to resolve the conflict. So in normal circumstances you shouldn't
>>> see it in perf top much less at 79%. There are two ways to deal with it,
>>> and each method should be complimented by the other. One way is to try
>>> to increase the number of ports and reduce the wait time which you have
>>> somehow tried. I would add the following:
>>> 
>>> net.ipv4.tcp_fin_timeout = 20
>>> 
>>> net.ipv4.tcp_max_tw_buckets = 1200
>>> 
>>> net.ipv4.tcp_keepalive_time = 1200
>>> 
>>> net.ipv4.tcp_syncookies = 1
>>> 
>>> net.ipv4.tcp_max_syn_backlog = 8192
>>> 
>>> The complimentary method to the above is to lower the number of
>>> connection requests by removing the frivolous connection requests out of
>>> the equation using a few iptables rules.
>>> 
>>> I'm assuming the increased load you're experiencing is due to the
>>> current DDos attacks and I'm not sure if you're using anything to
>>> mitigate that but you should consider it.
>>> 
>>> You may find something useful at the following  links
>>> 
>>> [1](https://github.com/Enkidu-6/tor-ddos)
>>> 
>>> [2](https://github.com/toralf/torutils)
>>> 
>>> [background](https://gitlab.torproject.org/tpo/community/support/-/issues/40093)
>>> 
>>> Cheers.
>>> 
>>> On 12/1/2022 3:35 PM, Christopher Sheats 

Re: [tor-relays] inet_csk_bind_conflict

[Christopher Sheats]

server1:~$ ss -s
Total: 454644
TCP:   465840 (estab 368011, closed 36634, orphaned 7619, timewait 11466)

Transport Total IPIPv6
RAW   0 0 0
UDP   48480
TCP   42920641381515391
INET  42925441386315391
FRAG  0 0 0

81% inet_csk_bind_conflict

server2:~$ ss -s
Total: 460089
TCP:   477026 (estab 367786, closed 42817, orphaned 7456, timewait 17239)

Transport Total IPIPv6
RAW   0 0 0
UDP   71710
TCP   43420941823515974
INET  43428041830615974
FRAG  1 1 0

80% inet_csk_bind_conflict

(total combined throughput at the time of measurement was ~650 Mbps symmetrical 
per transit provider metrics, this low throughput volume is common when 
inet_csk_bind_conflict is this high)

Re OutboundBindAddress - yes, for both v4 and v6

Re kernel version - 5.15.0-56-generic (jammy). Foundation for Applied Privacy 
recommended that we try the nightly repo which apparently includes the 
IP_BIND_ADDRESS_NO_PORT change. However that merge request mentions a 
workaround of modifying net.ipv4.ip_local_port_range, which we've already 
performed.

--
Christopher Sheats (yawnbox)
Executive Director
Emerald Onion
Signal: +1 206.739.3390
Website: https://emeraldonion.org/
Mastodon: https://digitalcourage.social/@EmeraldOnion/




> On Dec 3, 2022, at 3:02 AM, Anders Trier Olesen 
>  wrote:
> 
> Hi Christopher
> 
> How many open connections do you have? (`ss -s`)
> Do you happen to use OutboundBindAddress in your torrc?
> 
> What I think we need is for the Tor developers to include this PR in a 
> release: https://gitlab.torproject.org/tpo/core/tor/-/merge_requests/579
> Once that has happened, I think the problem should go away, as long as you 
> run a recent enough Linux kernel that supports IP_BIND_ADDRESS_NO_PORT (since 
> Linux 4.2).
> 
> - Anders
> 
> 
> 
> 
> fre. 2. dec. 2022 kl. 09.24 skrev Christopher Sheats 
> mailto:yawn...@emeraldonion.org>>:
>> Hello tor-relays,
>> 
>> We are using Ubuntu server currently for our exit relays. Occasionally, exit 
>> throughput will drop from ~4Gbps down to ~200Mbps and the only observable 
>> data point that we have is a significant increase in inet_csk_bind_conflict, 
>> as seen via 'perf top', where it will hit 85% [kernel] utilization.
>> 
>> A while back we thought we solved with with two /etc/sysctl.conf settings:
>> net.ipv4.ip_local_port_range = 1024 65535
>> net.ipv4.tcp_tw_reuse = 1
>> 
>> However we are still experiencing this problem.
>> 
>> Both of our (currently, two) relay servers suffer from the same problem, at 
>> the same time. They are AMD Epyc 7402P bare-metal servers each with 96GB 
>> RAM, each has 20 exit relays on them. This issue persists after upgrading to 
>> 0.4.7.11.
>> 
>> Screenshots of perf top are shared here: 
>> https://digitalcourage.social/@EmeraldOnion/109440197076214023
>> 
>> Does anyone have experience troubleshooting and/or fixing this problem?
>> 
>> Cheers,
>> 
>> --
>> Christopher Sheats (yawnbox)
>> Executive Director
>> Emerald Onion
>> Signal: +1 206.739.3390
>> Website: https://emeraldonion.org/
>> Mastodon: https://digitalcourage.social/@EmeraldOnion/
>> 
>> 
>> 
>> 
>> ___
>> tor-relays mailing list
>> tor-relays@lists.torproject.org 
>> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
> ___
> tor-relays mailing list
> tor-relays@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays



signature.asc
Description: Message signed with OpenPGP
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays