The tracer should be advertising its own address in ZK. By default, the server listens on `0.0.0.0`, unless `-a` or `--address` is specified on the command-line when it is started. Most server types use a utility class that will use `InetAddress.getLocalHost().getCanonicalHostName()` for the advertisement address if it sees that it is listening on `0.0.0.0`. However, it looks like the tracer doesn't do this.
I would try to verify that `0.0.0.0` actually appears as the advertised address in ZK for the tracer service, just be sure it's not a ZooTraceClient bug on the tablet server side (but I'm pretty sure it's not, after looking at the code). As a workaround, I would start up the tracer service using a script that sets `--address accumulo_tracer` on the command-line, where "accumulo_tracer" is the hostname or IP address that you want the tracer listening on and advertising to other servers. This address should be bindable where the tracer is running, and reachable from wherever the tservers are (so make sure it's not a private address only available inside the container running the tracer). The reason for this is that the hostname will resolve to an IP address inside the tracer process, and then the tracer will advertise the IP address. It won't advertise the original hostname, if you specified a hostname. I hope that helps. On Fri, Jul 29, 2022 at 12:18 PM kma <k...@phemi.com> wrote: > Nice, > > Turn on debug really help. > > 2022-07-29 12:12:14,235 [tracer.ZooTraceClient] DEBUG: Scanning trace > hosts in zookeeper: /tracers > 2022-07-29 12:12:14,240 [tracer.ZooTraceClient] DEBUG: Trace hosts: > [0.0.0.0:12234, 0.0.0.0:12234] > 2022-07-29 12:12:14,240 [tracer.ZooTraceClient] DEBUG: Successfully > initialized tracer hosts from ZooKeeper > > [root@accumulo_gc /]# curl 0.0.0.0:12234 > curl: (7) Failed connect to 0.0.0.0:12234; Connection refused > > [root@accumulo_gc /]# curl accumulo_tracer:12234 > curl: (52) Empty reply from server > > Is there a way to set Trace hosts to accumulo_tracer:12234 > > Cheers!... > ...Keith > > > On 2022/07/28 23:40:18 Christopher wrote: > > Do all servers have the same configuration? > > > > I would investigate the tablet server debug logs to determine if it's > > having trouble setting up tracing. It should be able to locate the > tracer > > service by talking to ZooKeeper and observing its service address > > advertisement, similar to how other servers register themselves in > > ZooKeeper. I'm not a docker network expert, but whatever service address > > the tracer service is advertising there should be routable from the > tablet > > servers. > > > > On Thu, Jul 28, 2022 at 7:31 PM kma <km...@phemi.com> wrote: > > > > > Thanks again Christopher, > > > > > > Our environment is a little different from usual. > > > > > > We have accumulo tracer running in it's own docker container and the > > > other accumulo services (e.g. gc, master, tserver, etc.) are also > > > running in different docker containers. We also have kerberos enabled. > > > > > > The following is our trace related configurations > > > > > > default | trace.port.client ................. | 12234 > > > default | trace.span.receivers .......... | > > > org.apache.accumulo.tracer.ZooTraceClient > > > default | trace.table ......................... | trace > > > default | trace.token.type ................ | > > > org.apache.accumulo.core.client.security.tokens.PasswordToken > > > site | @override ...................... | > > > org.apache.accumulo.core.client.security.tokens.KerberosToken > > > default | trace.user .......................... | root > > > site | @override ...................... | > > > accumulo-tra...@dev.phemi.com > > > default | trace.zookeeper.path ........ | /tracers > > > > > > And we can confirm that `trace on` and `trace off` works well in the > > > accumulo tracer container where the tracer process is running. > > > > > > However, `trace on` and `trace off` does not work in any other > > > containers. This is probably why we don't see compaction trace > messages > > > in the trace table. And it's likely because these containers don't > know > > > where the tracer service is running ? > > > > > > Question: How does the other accumulo services know where tracer > service > > > is running ? > > > Question: Is there a way to configure the tracer host where the tracer > > > service is running ? > > > > > > Cheers!... > > > ...Keith > > > > > > > > > On 2022/07/22 14:58:07 Christopher wrote: > > > > I would double check your trace credentials in your Accumulo > > > configuration > > > > file. And double check that the user credentials used to write to > the > > > trace > > > > table have permission to write to that table, and that you're > running > > > the > > > > accumulo-tracer service. > > > > > > > > On Fri, Jul 22, 2022 at 10:45 AM kma <km...@phemi.com> wrote: > > > > > > > > > Thank you Christopher, > > > > > > > > > > We can confirm that trace on and trace off are working fine in > > > accumulo > > > > > shell. We can see those trace messages in the trace table and > tracer > > > > > service is running. > > > > > > > > > > However, we are not seeing trace messages for MajC / MinC > > > > > > > > > > We do see these warnings in Monitor > > > > > > > > > > "Tracing spans are being dropped because there are already 5000 > spans > > > > > queued for delivery. This does not affect performance, security or > > > data > > > > > integrity, but distributed tracing information is being lost." > > > > > > > > > > Any suggestions ? > > > > > > > > > > Cheers!... > > > > > ...Keith > > > > > > > > > > > > > > > On 2022/07/12 19:42:07 Christopher wrote: > > > > > > By default, the tracer service that collects traces and writes > > > them to > > > > > > the trace table does not run. Generally, it is also > responsible for > > > > > > creating the trace table. If you have an empty trace table, > was it > > > > > > created by the tracer service, or did you create it manually? > > > > > > > > > > > > Assuming tracing is working as intended and there is a tracer > > > service > > > > > > running, there's only a few operations that trace > automatically, and > > > > > > even they do so at a small probability by default (like > > > compactions). > > > > > > If your system is just started and you're using default > settings, it > > > > > > may not have had a chance to create any traces yet. > > > > > > > > > > > > Some operations can be traced by explicitly turning on > tracing once > > > > > > everything is set up. You can test this by trying to turn on > tracing > > > > > > in the shell, using the "trace on" command, and then doing a > scan, > > > > > > then doing "trace off". I believe this still works as expected > in > > > > > > 2.0.1. This should trace client side operations through to the > > > server, > > > > > > initiated from the shell. This is also possible to be done in > > > > > > user-code, but there is no public API for it. > > > > > > > > > > > > A lot of the behavior of tracing has had to change in the > last few > > > > > > versions because of the dependencies we previously used for > tracing > > > > > > are no longer supported. Everything should still work like it > did in > > > > > > previous versions, but it's possible 2.0.1 does have issues. For > > > 2.1, > > > > > > we're expecting to use OpenTelemetry, which should make > tracing much > > > > > > easier and more reliable to configure. However, we do not > provide a > > > > > > built-in OpenTelemetry sink in 2.1 to write tracing > information to a > > > > > > table in Accumulo, like we did in previous versions for HTrace. > > > Such a > > > > > > thing could be added by the user or a 3rd party, though. > > > > > > > > > > > > On Tue, Jul 12, 2022 at 3:25 PM kma <km...@phemi.com> wrote: > > > > > > > > > > > > > > Hi Team, > > > > > > > > > > > > > > In one of out latest deployment of Accumulo 2.0.1, the trace > > > table is > > > > > > > empty. > > > > > > > > > > > > > > Question: is tracing automatically enabled by default ? If > not, > > > how to > > > > > > > enable it ? > > > > > > > > > > > > > > -- > > > > > > > ______________________________ > > > > > > > Keith Ma > > > > > > > Software Developer > > > > > > > *PHEMI Systems* > > > > > > > Suite 600 – 777 Hornby Street > > > > > > > Vancouver, BC V6Z 1S4 > > > > > > > 604-336-1119 > > > > > > > > > > > > > > website <http://www.phemi.com/> twitter > > > > > > > <https://twitter.com/PHEMISystems>linkedin > > > > > > > > > > > > < > > > > > > > > > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > ______________________________ > > > > > Keith Ma > > > > > Software Developer > > > > > *PHEMI Systems* > > > > > Suite 600 – 777 Hornby Street > > > > > Vancouver, BC V6Z 1S4 > > > > > 604-336-1119 > > > > > > > > > > website <http://www.phemi.com/> twitter > > > > > <https://twitter.com/PHEMISystems>linkedin > > > > > < > > > > > > > > > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > > > > > > > > > > > -- > > > ______________________________ > > > Keith Ma > > > Software Developer > > > *PHEMI Systems* > > > Suite 600 – 777 Hornby Street > > > Vancouver, BC V6Z 1S4 > > > 604-336-1119 > > > > > > website <http://www.phemi.com/> twitter > > > <https://twitter.com/PHEMISystems>linkedin > > > < > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > -- > ______________________________ > Keith Ma > Software Developer > *PHEMI Systems* > Suite 600 – 777 Hornby Street > Vancouver, BC V6Z 1S4 > 604-336-1119 > > website <http://www.phemi.com/> twitter > <https://twitter.com/PHEMISystems>linkedin > < > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > >
