https://github.com/apache/accumulo/pull/2119 fixed a bug in 2.1 where the -a argument was not correctly setting the hostname. It looks like 2.0.1 is affected by this too.
On Fri, Jul 29, 2022 at 12:57 PM Christopher <ctubb...@apache.org> wrote: > The tracer should be advertising its own address in ZK. By default, the > server listens on `0.0.0.0`, unless `-a` or `--address` is specified on the > command-line when it is started. Most server types use a utility class that > will use `InetAddress.getLocalHost().getCanonicalHostName()` for the > advertisement address if it sees that it is listening on `0.0.0.0`. > However, it looks like the tracer doesn't do this. > > I would try to verify that `0.0.0.0` actually appears as the advertised > address in ZK for the tracer service, just be sure it's not a > ZooTraceClient bug on the tablet server side (but I'm pretty sure it's not, > after looking at the code). > > As a workaround, I would start up the tracer service using a script that > sets `--address accumulo_tracer` on the command-line, where > "accumulo_tracer" is the hostname or IP address that you want the tracer > listening on and advertising to other servers. This address should be > bindable where the tracer is running, and reachable from wherever the > tservers are (so make sure it's not a private address only available inside > the container running the tracer). The reason for this is that the hostname > will resolve to an IP address inside the tracer process, and then the > tracer will advertise the IP address. It won't advertise the original > hostname, if you specified a hostname. > > I hope that helps. > > > On Fri, Jul 29, 2022 at 12:18 PM kma <k...@phemi.com> wrote: > > > Nice, > > > > Turn on debug really help. > > > > 2022-07-29 12:12:14,235 [tracer.ZooTraceClient] DEBUG: Scanning trace > > hosts in zookeeper: /tracers > > 2022-07-29 12:12:14,240 [tracer.ZooTraceClient] DEBUG: Trace hosts: > > [0.0.0.0:12234, 0.0.0.0:12234] > > 2022-07-29 12:12:14,240 [tracer.ZooTraceClient] DEBUG: Successfully > > initialized tracer hosts from ZooKeeper > > > > [root@accumulo_gc /]# curl 0.0.0.0:12234 > > curl: (7) Failed connect to 0.0.0.0:12234; Connection refused > > > > [root@accumulo_gc /]# curl accumulo_tracer:12234 > > curl: (52) Empty reply from server > > > > Is there a way to set Trace hosts to accumulo_tracer:12234 > > > > Cheers!... > > ...Keith > > > > > > On 2022/07/28 23:40:18 Christopher wrote: > > > Do all servers have the same configuration? > > > > > > I would investigate the tablet server debug logs to determine if it's > > > having trouble setting up tracing. It should be able to locate the > > tracer > > > service by talking to ZooKeeper and observing its service address > > > advertisement, similar to how other servers register themselves in > > > ZooKeeper. I'm not a docker network expert, but whatever service > address > > > the tracer service is advertising there should be routable from the > > tablet > > > servers. > > > > > > On Thu, Jul 28, 2022 at 7:31 PM kma <km...@phemi.com> wrote: > > > > > > > Thanks again Christopher, > > > > > > > > Our environment is a little different from usual. > > > > > > > > We have accumulo tracer running in it's own docker container and the > > > > other accumulo services (e.g. gc, master, tserver, etc.) are also > > > > running in different docker containers. We also have kerberos > enabled. > > > > > > > > The following is our trace related configurations > > > > > > > > default | trace.port.client ................. | 12234 > > > > default | trace.span.receivers .......... | > > > > org.apache.accumulo.tracer.ZooTraceClient > > > > default | trace.table ......................... | trace > > > > default | trace.token.type ................ | > > > > org.apache.accumulo.core.client.security.tokens.PasswordToken > > > > site | @override ...................... | > > > > org.apache.accumulo.core.client.security.tokens.KerberosToken > > > > default | trace.user .......................... | root > > > > site | @override ...................... | > > > > accumulo-tra...@dev.phemi.com > > > > default | trace.zookeeper.path ........ | /tracers > > > > > > > > And we can confirm that `trace on` and `trace off` works well in the > > > > accumulo tracer container where the tracer process is running. > > > > > > > > However, `trace on` and `trace off` does not work in any other > > > > containers. This is probably why we don't see compaction trace > > messages > > > > in the trace table. And it's likely because these containers don't > > know > > > > where the tracer service is running ? > > > > > > > > Question: How does the other accumulo services know where tracer > > service > > > > is running ? > > > > Question: Is there a way to configure the tracer host where the > tracer > > > > service is running ? > > > > > > > > Cheers!... > > > > ...Keith > > > > > > > > > > > > On 2022/07/22 14:58:07 Christopher wrote: > > > > > I would double check your trace credentials in your Accumulo > > > > configuration > > > > > file. And double check that the user credentials used to write to > > the > > > > trace > > > > > table have permission to write to that table, and that you're > > running > > > > the > > > > > accumulo-tracer service. > > > > > > > > > > On Fri, Jul 22, 2022 at 10:45 AM kma <km...@phemi.com> wrote: > > > > > > > > > > > Thank you Christopher, > > > > > > > > > > > > We can confirm that trace on and trace off are working fine in > > > > accumulo > > > > > > shell. We can see those trace messages in the trace table and > > tracer > > > > > > service is running. > > > > > > > > > > > > However, we are not seeing trace messages for MajC / MinC > > > > > > > > > > > > We do see these warnings in Monitor > > > > > > > > > > > > "Tracing spans are being dropped because there are already 5000 > > spans > > > > > > queued for delivery. This does not affect performance, security > or > > > > data > > > > > > integrity, but distributed tracing information is being lost." > > > > > > > > > > > > Any suggestions ? > > > > > > > > > > > > Cheers!... > > > > > > ...Keith > > > > > > > > > > > > > > > > > > On 2022/07/12 19:42:07 Christopher wrote: > > > > > > > By default, the tracer service that collects traces and writes > > > > them to > > > > > > > the trace table does not run. Generally, it is also > > responsible for > > > > > > > creating the trace table. If you have an empty trace table, > > was it > > > > > > > created by the tracer service, or did you create it manually? > > > > > > > > > > > > > > Assuming tracing is working as intended and there is a tracer > > > > service > > > > > > > running, there's only a few operations that trace > > automatically, and > > > > > > > even they do so at a small probability by default (like > > > > compactions). > > > > > > > If your system is just started and you're using default > > settings, it > > > > > > > may not have had a chance to create any traces yet. > > > > > > > > > > > > > > Some operations can be traced by explicitly turning on > > tracing once > > > > > > > everything is set up. You can test this by trying to turn on > > tracing > > > > > > > in the shell, using the "trace on" command, and then doing a > > scan, > > > > > > > then doing "trace off". I believe this still works as expected > > in > > > > > > > 2.0.1. This should trace client side operations through to the > > > > server, > > > > > > > initiated from the shell. This is also possible to be done in > > > > > > > user-code, but there is no public API for it. > > > > > > > > > > > > > > A lot of the behavior of tracing has had to change in the > > last few > > > > > > > versions because of the dependencies we previously used for > > tracing > > > > > > > are no longer supported. Everything should still work like it > > did in > > > > > > > previous versions, but it's possible 2.0.1 does have issues. > For > > > > 2.1, > > > > > > > we're expecting to use OpenTelemetry, which should make > > tracing much > > > > > > > easier and more reliable to configure. However, we do not > > provide a > > > > > > > built-in OpenTelemetry sink in 2.1 to write tracing > > information to a > > > > > > > table in Accumulo, like we did in previous versions for > HTrace. > > > > Such a > > > > > > > thing could be added by the user or a 3rd party, though. > > > > > > > > > > > > > > On Tue, Jul 12, 2022 at 3:25 PM kma <km...@phemi.com> wrote: > > > > > > > > > > > > > > > > Hi Team, > > > > > > > > > > > > > > > > In one of out latest deployment of Accumulo 2.0.1, the trace > > > > table is > > > > > > > > empty. > > > > > > > > > > > > > > > > Question: is tracing automatically enabled by default ? If > > not, > > > > how to > > > > > > > > enable it ? > > > > > > > > > > > > > > > > -- > > > > > > > > ______________________________ > > > > > > > > Keith Ma > > > > > > > > Software Developer > > > > > > > > *PHEMI Systems* > > > > > > > > Suite 600 – 777 Hornby Street > > > > > > > > Vancouver, BC V6Z 1S4 > > > > > > > > 604-336-1119 > > > > > > > > > > > > > > > > website <http://www.phemi.com/> twitter > > > > > > > > <https://twitter.com/PHEMISystems>linkedin > > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > ______________________________ > > > > > > Keith Ma > > > > > > Software Developer > > > > > > *PHEMI Systems* > > > > > > Suite 600 – 777 Hornby Street > > > > > > Vancouver, BC V6Z 1S4 > > > > > > 604-336-1119 > > > > > > > > > > > > website <http://www.phemi.com/> twitter > > > > > > <https://twitter.com/PHEMISystems>linkedin > > > > > > < > > > > > > > > > > > > > > > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > ______________________________ > > > > Keith Ma > > > > Software Developer > > > > *PHEMI Systems* > > > > Suite 600 – 777 Hornby Street > > > > Vancouver, BC V6Z 1S4 > > > > 604-336-1119 > > > > > > > > website <http://www.phemi.com/> twitter > > > > <https://twitter.com/PHEMISystems>linkedin > > > > < > > > > > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > > > > > > -- > > ______________________________ > > Keith Ma > > Software Developer > > *PHEMI Systems* > > Suite 600 – 777 Hornby Street > > Vancouver, BC V6Z 1S4 > > 604-336-1119 > > > > website <http://www.phemi.com/> twitter > > <https://twitter.com/PHEMISystems>linkedin > > < > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > >
