Re: [tor-relays] Declining Relay Usage

[Micah Elizabeth Scott]

On 12/17/23 21:16, likogan via tor-relays wrote:


My exit relay has seen steadily decreasing traffic from 8MBps to 6MBps
over the span of three weeks. It averages a load of ~50% CPU usage and
~65% RAM usage. It's rated network capacity is 17Mbps on a 10GB link.
Why would traffic decrease if I have plenty of spare resources? Are
there ways I can configure my server to boost traffic?





Hello!

In general it isn't possible to arbitrarily request more traffic; it's 
bad if an adversary can easily spin up many new relays and get as much 
traffic as they want directed their way, for example. It's normal for 
relays with low uptime to get less traffic. If you aren't seeing CPU 
usage hit 100% of a single core that is not necessarily a problem. Extra 
capacity is useful for providing lower-latency circuits. If you want 
more traffic, you might just need to be patient and let your relay run 
without interruption.


-beth


___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Worse throughput with 0.4.8.x, on a slow CPU

[Micah Elizabeth Scott]

On 12/13/23 06:15, Roman Mamedov wrote:

Hello,

Ater upgrading from Tor 0.4.7.13 to 0.4.8.9, I get a much worse bandwidth
numbers.


Hello!

I'm not aware of any changes in that interval that should affect relays. 
Conflux and proof of work both arrived in that time period, but neither 
of these features affect relays.


If you can indeed generate this change in throughput reproducibly by 
switching versions, it would be useful to know more. Do you have access 
to a sampling profiler like linux-perf "perf top"? If you could compare 
both the bandwidth and the location of top CPU usage across both 
versions, that would be ideal.


It's possible that there's a bottleneck in system calls or cache 
footprint that is especially noticeable on a CPU like the Atom. That's 
the kind of question that may be answerable by comparing CPU counters 
between the two runs.


We haven't done this type of profiling on C-tor recently and we are 
still early in defining the specific performance characteristics that we 
expect from Arti. If you can help us identify the actual bottlenecks 
your relay is hitting that may prove instructive for any future design 
choices we make in Arti!


That said, it may be better for the network to also encourage everyone 
to run their relays below 100% CPU load. Relays with lower load are 
going to be lower latency, and raw bandwidth numbers aren't necessarily 
what users will find the most beneficial.


Thanks for running a relay, and for any additional performance data you 
can gather from your system.


--beth


The CPU is Atom C2338 with two cores at 1.75 Ghz. Multiple Tor instances are
running to take advantage of both cores.

On the older version it gets about 80+80 Mbit total in+out. On the new one the
average is at most 45+45 Mbit. There are frequent periods where the bandwidth
drops to 5-10 Mbit for 3-5 seconds, while all Tor processes continue to use
100% of both CPUs, then gradually climbs back up.

Does anyone notice anything similar?

Here's how it looks for a few days on 0.4.8 then a roll-back:

+-+-+---
 today461.94 GiB |  470.84 GiB |  932.78 GiB |  153.98 Mbit/s 

  12/12/23462.81 GiB |  475.67 GiB |  938.48 GiB |   91.12 Mbit/s 
###
  12/11/23434.72 GiB |  443.08 GiB |  877.80 GiB |   85.23 Mbit/s 
##
  12/10/23446.13 GiB |  459.56 GiB |  905.70 GiB |   87.93 Mbit/s 
###
  12/09/23464.79 GiB |  473.17 GiB |  937.96 GiB |   91.07 Mbit/s 
###
  12/08/23454.67 GiB |  463.48 GiB |  918.16 GiB |   89.14 Mbit/s 
###
  12/07/23463.82 GiB |  472.84 GiB |  936.65 GiB |   90.94 Mbit/s 
###
  12/06/23670.01 GiB |  680.55 GiB |1.32 TiB |  131.13 Mbit/s 
##
  12/05/23808.50 GiB |  817.32 GiB |1.59 TiB |  157.85 Mbit/s 
#
  12/04/23854.09 GiB |  866.43 GiB |1.68 TiB |  167.05 Mbit/s 
###
  12/03/23782.20 GiB |  799.00 GiB |1.54 TiB |  153.52 Mbit/s 

  12/02/23805.18 GiB |  817.91 GiB |1.59 TiB |  157.59 Mbit/s 
#
  12/01/23733.80 GiB |  745.35 GiB |1.44 TiB |  143.61 Mbit/s 
#
  11/30/23742.48 GiB |  756.56 GiB |1.46 TiB |  145.54 Mbit/s 
#
  11/29/23657.90 GiB |  673.66 GiB |1.30 TiB |  129.28 Mbit/s 
#

In general, are there any tweaks to reduce relay CPU usage on a slow processor?
I did seemingly most of what is possible, ethtool, iptables, sysctl, etc.

How long before the Rust-based Tor will be ready for use on relays?


___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Declining Relay Usage

[Roman Mamedov]

On Mon, 18 Dec 2023 16:57:37 +
George Hartley  wrote:

> Please read the code, not only Tor's code, but also OpenSSL's code.
> 
> Yes, AES is not displayed as engine itself, however, it still does not seem
> to use aes-ni instructions unless told to initialize engines via the code I
> deducted.
> 
> If this proves anything, I ran an Exit Relay in 2013 before my host forced
> me to switch to a Guard one because of excessive abuse, and even though my
> VM supported aesni instructions, OpenSSL would not actually use them until I
> added the config parameter, the peak CPU usage (single core) dropped from
> 88-95% avg to around 23% avg once I added it.
> 
> Maybe some developer can comment on the deeper workings of OpenSSL and Tor,
> and terminology might get weird between the Tor and OpenSSL (both very big
> code-bases).
> 
>  Also, regarding the e-mail, I post regularly on here and tor-dev, so no
> worries :)
> 
> Let's just end this pointless discussion here, I will do some more research
> the next few days because I actually want to know, but to me everything
> seems pretty clear (from the code I've YET SEEN, not the one I DID NOT YET
> SEE).

My main point though: it would be an insanity and beyond belief, if Tor would
not use the universally-available AES-NI instruction sets on modern CPUs,
unless the user somehow knows to find and set a non-default config option.

Moreover, just recently in my experience leading to the message "[tor-relays]
Worse throughput with 0.4.8.x, on a slow CPU"[1] I tried ALL sorts of the
possible tweaks I could find, which included adding "HardwareAccel 1" to the
config (it was absent before), and it made no difference in CPU use.

[1] https://lists.torproject.org/pipermail/tor-relays/2023-December/021407.html

-- 
With respect,
Roman
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Declining Relay Usage

[Roman Mamedov]

On Mon, 18 Dec 2023 15:58:52 +
George Hartley  wrote:

> I had a quick look at the manual, and it stated:
> 
> > HardwareAccel 0|1
> 
> > If non-zero, try to use built-in (static) crypto hardware acceleration > 
> > when available. Can not be changed while tor is running. (Default: 0)
> 
> A quick look at the source code tells me that Tor relies entirely on OpenSSL.
> 
> The call-chain here is as follows:
> 
> crypto_set_options first determines whether to enable any available OpenSSL 
> engines based on if the variable mentioned above is non-zero or if an 
> accelerator name has been set:
> 
> > const bool hardware_accel = options->HardwareAccel || options->AccelName;
> 
> This bool is then passed into crypto_global_init, where it is the first 
> argument, fittingly named "useAccel".
> 
> useAccel is then passed into crypto_openssl_late_init, where if HardwareAccel 
> is the default (0) or no engine name has been specified, OpenSSL will make no 
> attempt to load any acceleration engines.
> 
> Here is a permalink to that last relevant function in the call chain:
> 
> https://gitlab.torproject.org/tpo/core/tor/-/blob/main/src/lib/crypt_ops/crypto_openssl_mgt.c?ref_type=heads#L382
> 
> So yes, I think it is pretty safe to assume that if you do not set either 
> configuration option, no OpenSSL engine will be used.
> 
> Thank you for questioning me though, thanks to you I learned some more about 
> Tor's inner workings, and you hopefully too :)

It is not entirely clear to me what conclusion you came to after this
research. If you mean that HardwareAccel is needed, I would still disagee.

If I'm not mistaken the AES-NI support is implemented in OpenSSL not via an
"engine" that you have to "use", it is just built-in internally on some deeper
level. For a proof you can run "openssl engine" in the console of any
AES-supporting machine, and you will not see any loadable engines there, aside
from rdrand, which is unrelated, and "dynamic" which just means it can load
some acceleration engines if it had any. And for instance VIA Padlock would
show up as "padlock" in that list.

Please use reply to all the mailing list. Sorry for bringing out your mail
into the public, but it didn't seem to be strictly private in any case.

-- 
With respect,
Roman
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Declining Relay Usage

[Roman Mamedov]

On Mon, 18 Dec 2023 10:03:01 +
George Hartley via tor-relays  wrote:

> lscpu | grep aes
> 
> If the command returns nothing, sadly your CPU does not support hardware AES
> acceleration, or if you run your OS in a VM, then the VM operator likely did
> not set "host" as CPU model.
> 
> If however the command does output a list of flags, with aes highlighted in
> red (depends on SSH client), then you can safely add the following line to
> your nodes configuration file:
> 
> > HardwareAccel 1

I don't think this is necessary to make it use AES. It would use that
automatically. HardwareAccel is needed only to enable some external OpenSSL
acceleration engines, such as the defunct "VIA Padlock".

-- 
With respect,
Roman
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Declining Relay Usage

[George Hartley via tor-relays]

Hello Likogan (you did not specify a name, so I just took your domain name).

First, lets look at issue number one:

If your Tor Exit is using ~50% of the entire CPU (VM or dedicated server?) 
while only routing 6 Mbps, then you are likely not using hardware AES 
acceleration (aesni).

For example, my Tor Exit node only uses 15-25% of a single core while easily 
routing 10 to 12 Megabytes per second.

All on the following CPU:

Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz

with a maximum boost clock of 2.50 GHz.

Try the following command:

lscpu | grep aes

If the command returns nothing, sadly your CPU does not support hardware AES 
acceleration, or if you run your OS in a VM, then the VM operator likely did 
not set "host" as CPU model.

If however the command does output a list of flags, with aes highlighted in red 
(depends on SSH client), then you can safely add the following line to your 
nodes configuration file:

> HardwareAccel 1

General specs about your server, including the full output from lscpu would 
also be nice, if you are on a 10GbE link, then I assume it is a dedicated 
server, and a relatively new one (hardware wise) at that.

Now lets look at your traffic provider, or it's AS number:

https://metrics.torproject.org/rs.html#search/as:AS53667

We can see right away that this host is very congested with Tor nodes already 
(around 230 nodes in their datacenters right now), and thus the Tor authorities 
might route less traffic through it in general - decentralization is ALWAYS 
better!

I actually don't know if the Tor authorities act that way, maybe someone with 
more knowledge can chime in.

So yes, here is a too long, didn't read for you:

Check for aesni support as explained above, if it exists, please add the 
mentioned config entry, and just to make sure, the NumCPUs variable with the 
amount of your logical CPU cores.

Also, even if Tor's code base is mostly single-threaded, there are a few tasks 
that can be offloaded to different cores, such as onionskin decryption, zlib 
compression, etc.

If you have some spare CPU cores, please let Tor offload as much work as 
possible by, as said above, adding the

> NumCPUs 

variable to your nodes configuration.

This generally is not necessary as Tor will try to detect the amount of cores 
automatically, but in a locked down environment, such as mine, it wouldn't work 
:)

Hope this helps you or others,
George

P.S: My e-mail web-client always auto-attaches my PGP public key, so if you (or 
others) want to talk to me privately, that option exists too, however it 
shouldn't be needed in this case.

All the best and thank you so much for hosting an exit relay!

> My exit relay has seen steadily decreasing traffic from 8MBps to 6MBps
> over the span of three weeks. It averages a load of ~50% CPU usage and
> ~65% RAM usage. It's rated network capacity is 17Mbps on a 10GB link.
> Why would traffic decrease if I have plenty of spare resources? Are
> there ways I can configure my server to boost traffic?
> 

> https://metrics.torproject.org/rs.html#details/292FCACE773DC259B799914A23BE65A6A6178E8F
> 

> 

> 

> ___
> tor-relays mailing list
> tor-relays@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

publickey - hartley_george@proton.me - 0xAEE8E00F.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Tor: Possible bug on 0.4.8.9 exit relay.

[trinity pointard]

Hi,

this looks related to TROVE-2023-007 /
https://gitlab.torproject.org/tpo/core/tor/-/issues/40897 . It should
no longer appear after you upgrade.
If it still does, please do come back.

Regards,
trinity-1686a

On Mon, 18 Dec 2023 at 07:05, George Hartley via tor-relays
 wrote:
>
> Hi,
>
> while going through journalctl I noticed the following entries from my exit 
> relay and wanted to report this non-fatal assertion.
>
> I also host a Guard relay on the same VM and IP, and it did not yet assert 
> that message.
>
> The full assert() with the stack-trace is as follows:
>
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] 
> tor_bug_occurred_(): Bug: src/core/or/conflux.c:565: conflux_pick_first_leg: 
> Non-fatal assertion !(smartlist_len(cfx->legs) <= 0) failed. (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: Tor 0.4.8.9: 
> Non-fatal assertion !(smartlist_len(cfx->legs) <= 0) failed in 
> conflux_pick_first_leg at src/core/or/conflux.c:565. Stack trace: (on Tor 
> 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/bin/tor(log_backtrace_impl+0x5d) [0x5fdf47b6a9ad] (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/bin/tor(tor_bug_occurred_+0x194) [0x5fdf47b81764] (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/bin/tor(conflux_decide_next_circ+0x3fe) [0x5fdf47c2031e] (on Tor 0.4.8.9 
> )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/bin/tor(circuit_get_package_window+0x6d) [0x5fdf47c285ed] (on Tor 
> 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/bin/tor(+0x9bc59) [0x5fdf47b15c59] (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/bin/tor(connection_edge_package_raw_inbuf+0xa9) [0x5fdf47b15e39] (on Tor 
> 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/bin/tor(connection_edge_process_inbuf+0x76) [0x5fdf47c42866] (on Tor 
> 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/bin/tor(+0x1c0218) [0x5fdf47c3a218] (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/bin/tor(+0x6e72d) [0x5fdf47ae872d] (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/lib/libevent-2.1.so.7(+0x22c45) [0x7d444a49cc45] (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/lib/libevent-2.1.so.7(event_base_loop+0x4ff) [0x7d444a49e3af] (on Tor 
> 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/bin/tor(do_main_loop+0x104) [0x5fdf47ae8f44] (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/bin/tor(tor_run_main+0x215) [0x5fdf47aed165] (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/bin/tor(tor_main+0x5e) [0x5fdf47aed5ee] (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/bin/tor(main+0x1d) [0x5fdf47adf08d] (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/lib/libc.so.6(+0x27cd0) [0x7d4449a7ecd0] (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/lib/libc.so.6(__libc_start_main+0x8a) [0x7d4449a7ed8a] (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] Bug: 
> /usr/bin/tor(_start+0x25) [0x5fdf47adf0e5] (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] 
> conflux_pick_first_leg(): Bug: Matching client sets: (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] 
> conflux_log_set(): Bug: Conflux 21CEFB4FACA11A02: 0 linked, 0 launched. 
> Delivered: 7272; teardown: 0; Current: (nil), Previous: (nil) (on Tor 0.4.8.9 
> )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] 
> conflux_pick_first_leg(): Bug: Matching server sets: (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] 
> conflux_log_set(): Bug: Conflux 21CEFB4FACA11A02: 0 linked, 0 launched. 
> Delivered: 7272; teardown: 0; Current: (nil), Previous: (nil) (on Tor 0.4.8.9 
> )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] 
> conflux_pick_first_leg(): Bug: End conflux set dump (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] 
> circuit_get_package_window(): Bug: Conflux has no circuit to send on. Circuit 
> 0x5fdf581d3460 idx 4524 marked at line src/core/or/command.c:663 (on Tor 
> 0.4.8.9 )
> Dec 14 16:29:36 matrix tor[575]: Dec 14 16:29:36.000 [warn] 
> tor_bug_occurred_(): Bug: src/core/or/conflux.c:565: conflux_pick_first_leg: 
> Non-fatal assertion !(smartlist_len(cfx->legs) <= 0) failed. (on Tor 0.4.8.9 )
> Dec 14 16:29:36 matrix 

[tor-relays] Tor relay operator meetups

[telekobold]

Hi,

will there be a Tor relay operators meetup @37C3 [*]?

Also, there were apparently no meetups in November and December this year?

Kind regards
telekobold

[*] https://events.ccc.de/congress/2023/infos/startpage.html
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Issue with Relay Status Discrepancy on Tor Metrics

[Georg Koppen]

Jan:

Dear Tor Metrics Support Team,

I hope this message finds you well.

I am writing to report an issue regarding the status of my relay on Tor
Metrics. According to Nyx, my relay appears to be running without any
problems. However, Tor Metrics indicates that it is offline.

Relay Details:

Name:BtcToTheMoon
IP Address: 116.203.227.107
Contact Email: janthor...@gmail.com

I have verified through Nyx that the relay is operational, yet the status
on Tor Metrics remains unchanged. I am concerned about this discrepancy and
would appreciate any assistance in resolving this matter.

Thank you for your attention to this issue. Your help in rectifying this
discrepancy would be greatly appreciated.


It seems this issue got solved in the mean time as your relay is showing 
up as running on our relay search portal right now:


https://metrics.torproject.org/rs.html#details/D8952E93A489B7C2320B2A9896B3355F2DAB6096

Let us know if you have more questions, we are happy to help. :)

Thanks for running relays,
Georg


Sincerely,
Michal


___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays




OpenPGP_signature.asc
Description: OpenPGP digital signature
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] Issue with Relay Status Discrepancy on Tor Metrics

[Jan]

Dear Tor Metrics Support Team,

I hope this message finds you well.

I am writing to report an issue regarding the status of my relay on Tor
Metrics. According to Nyx, my relay appears to be running without any
problems. However, Tor Metrics indicates that it is offline.

Relay Details:

Name:BtcToTheMoon
IP Address: 116.203.227.107
Contact Email: janthor...@gmail.com

I have verified through Nyx that the relay is operational, yet the status
on Tor Metrics remains unchanged. I am concerned about this discrepancy and
would appreciate any assistance in resolving this matter.

Thank you for your attention to this issue. Your help in rectifying this
discrepancy would be greatly appreciated.

Sincerely,
Michal
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays