Do all servers have the same configuration? I would investigate the tablet server debug logs to determine if it's having trouble setting up tracing. It should be able to locate the tracer service by talking to ZooKeeper and observing its service address advertisement, similar to how other servers register themselves in ZooKeeper. I'm not a docker network expert, but whatever service address the tracer service is advertising there should be routable from the tablet servers.
On Thu, Jul 28, 2022 at 7:31 PM kma <k...@phemi.com> wrote: > Thanks again Christopher, > > Our environment is a little different from usual. > > We have accumulo tracer running in it's own docker container and the > other accumulo services (e.g. gc, master, tserver, etc.) are also > running in different docker containers. We also have kerberos enabled. > > The following is our trace related configurations > > default | trace.port.client ................. | 12234 > default | trace.span.receivers .......... | > org.apache.accumulo.tracer.ZooTraceClient > default | trace.table ......................... | trace > default | trace.token.type ................ | > org.apache.accumulo.core.client.security.tokens.PasswordToken > site | @override ...................... | > org.apache.accumulo.core.client.security.tokens.KerberosToken > default | trace.user .......................... | root > site | @override ...................... | > accumulo-tra...@dev.phemi.com > default | trace.zookeeper.path ........ | /tracers > > And we can confirm that `trace on` and `trace off` works well in the > accumulo tracer container where the tracer process is running. > > However, `trace on` and `trace off` does not work in any other > containers. This is probably why we don't see compaction trace messages > in the trace table. And it's likely because these containers don't know > where the tracer service is running ? > > Question: How does the other accumulo services know where tracer service > is running ? > Question: Is there a way to configure the tracer host where the tracer > service is running ? > > Cheers!... > ...Keith > > > On 2022/07/22 14:58:07 Christopher wrote: > > I would double check your trace credentials in your Accumulo > configuration > > file. And double check that the user credentials used to write to the > trace > > table have permission to write to that table, and that you're running > the > > accumulo-tracer service. > > > > On Fri, Jul 22, 2022 at 10:45 AM kma <km...@phemi.com> wrote: > > > > > Thank you Christopher, > > > > > > We can confirm that trace on and trace off are working fine in > accumulo > > > shell. We can see those trace messages in the trace table and tracer > > > service is running. > > > > > > However, we are not seeing trace messages for MajC / MinC > > > > > > We do see these warnings in Monitor > > > > > > "Tracing spans are being dropped because there are already 5000 spans > > > queued for delivery. This does not affect performance, security or > data > > > integrity, but distributed tracing information is being lost." > > > > > > Any suggestions ? > > > > > > Cheers!... > > > ...Keith > > > > > > > > > On 2022/07/12 19:42:07 Christopher wrote: > > > > By default, the tracer service that collects traces and writes > them to > > > > the trace table does not run. Generally, it is also responsible for > > > > creating the trace table. If you have an empty trace table, was it > > > > created by the tracer service, or did you create it manually? > > > > > > > > Assuming tracing is working as intended and there is a tracer > service > > > > running, there's only a few operations that trace automatically, and > > > > even they do so at a small probability by default (like > compactions). > > > > If your system is just started and you're using default settings, it > > > > may not have had a chance to create any traces yet. > > > > > > > > Some operations can be traced by explicitly turning on tracing once > > > > everything is set up. You can test this by trying to turn on tracing > > > > in the shell, using the "trace on" command, and then doing a scan, > > > > then doing "trace off". I believe this still works as expected in > > > > 2.0.1. This should trace client side operations through to the > server, > > > > initiated from the shell. This is also possible to be done in > > > > user-code, but there is no public API for it. > > > > > > > > A lot of the behavior of tracing has had to change in the last few > > > > versions because of the dependencies we previously used for tracing > > > > are no longer supported. Everything should still work like it did in > > > > previous versions, but it's possible 2.0.1 does have issues. For > 2.1, > > > > we're expecting to use OpenTelemetry, which should make tracing much > > > > easier and more reliable to configure. However, we do not provide a > > > > built-in OpenTelemetry sink in 2.1 to write tracing information to a > > > > table in Accumulo, like we did in previous versions for HTrace. > Such a > > > > thing could be added by the user or a 3rd party, though. > > > > > > > > On Tue, Jul 12, 2022 at 3:25 PM kma <km...@phemi.com> wrote: > > > > > > > > > > Hi Team, > > > > > > > > > > In one of out latest deployment of Accumulo 2.0.1, the trace > table is > > > > > empty. > > > > > > > > > > Question: is tracing automatically enabled by default ? If not, > how to > > > > > enable it ? > > > > > > > > > > -- > > > > > ______________________________ > > > > > Keith Ma > > > > > Software Developer > > > > > *PHEMI Systems* > > > > > Suite 600 – 777 Hornby Street > > > > > Vancouver, BC V6Z 1S4 > > > > > 604-336-1119 > > > > > > > > > > website <http://www.phemi.com/> twitter > > > > > <https://twitter.com/PHEMISystems>linkedin > > > > > > > > < > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > > > > -- > > > ______________________________ > > > Keith Ma > > > Software Developer > > > *PHEMI Systems* > > > Suite 600 – 777 Hornby Street > > > Vancouver, BC V6Z 1S4 > > > 604-336-1119 > > > > > > website <http://www.phemi.com/> twitter > > > <https://twitter.com/PHEMISystems>linkedin > > > < > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > -- > ______________________________ > Keith Ma > Software Developer > *PHEMI Systems* > Suite 600 – 777 Hornby Street > Vancouver, BC V6Z 1S4 > 604-336-1119 > > website <http://www.phemi.com/> twitter > <https://twitter.com/PHEMISystems>linkedin > < > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > >
