Ok, I wasn't sure what version was being used here. I saw a reference to 2.0.1 in an earlier email.
On Fri, Jul 29, 2022 at 3:03 PM Christopher <[email protected]> wrote: > That was a bug in the new AbstractServer class in 2.x. I don't think it > ever affected 1.x > I checked the 1.10 code and it wouldn't affect the tracer server. `-a` > should still work fine there. > > On Fri, Jul 29, 2022 at 3:00 PM Dave Marion <[email protected]> wrote: > > > https://github.com/apache/accumulo/pull/2119 fixed a bug in 2.1 where > the > > -a argument was not correctly setting the hostname. It looks like 2.0.1 > is > > affected by this too. > > > > On Fri, Jul 29, 2022 at 12:57 PM Christopher <[email protected]> > wrote: > > > > > The tracer should be advertising its own address in ZK. By default, the > > > server listens on `0.0.0.0`, unless `-a` or `--address` is specified on > > the > > > command-line when it is started. Most server types use a utility class > > that > > > will use `InetAddress.getLocalHost().getCanonicalHostName()` for the > > > advertisement address if it sees that it is listening on `0.0.0.0`. > > > However, it looks like the tracer doesn't do this. > > > > > > I would try to verify that `0.0.0.0` actually appears as the advertised > > > address in ZK for the tracer service, just be sure it's not a > > > ZooTraceClient bug on the tablet server side (but I'm pretty sure it's > > not, > > > after looking at the code). > > > > > > As a workaround, I would start up the tracer service using a script > that > > > sets `--address accumulo_tracer` on the command-line, where > > > "accumulo_tracer" is the hostname or IP address that you want the > tracer > > > listening on and advertising to other servers. This address should be > > > bindable where the tracer is running, and reachable from wherever the > > > tservers are (so make sure it's not a private address only available > > inside > > > the container running the tracer). The reason for this is that the > > hostname > > > will resolve to an IP address inside the tracer process, and then the > > > tracer will advertise the IP address. It won't advertise the original > > > hostname, if you specified a hostname. > > > > > > I hope that helps. > > > > > > > > > On Fri, Jul 29, 2022 at 12:18 PM kma <[email protected]> wrote: > > > > > > > Nice, > > > > > > > > Turn on debug really help. > > > > > > > > 2022-07-29 12:12:14,235 [tracer.ZooTraceClient] DEBUG: Scanning trace > > > > hosts in zookeeper: /tracers > > > > 2022-07-29 12:12:14,240 [tracer.ZooTraceClient] DEBUG: Trace hosts: > > > > [0.0.0.0:12234, 0.0.0.0:12234] > > > > 2022-07-29 12:12:14,240 [tracer.ZooTraceClient] DEBUG: Successfully > > > > initialized tracer hosts from ZooKeeper > > > > > > > > [root@accumulo_gc /]# curl 0.0.0.0:12234 > > > > curl: (7) Failed connect to 0.0.0.0:12234; Connection refused > > > > > > > > [root@accumulo_gc /]# curl accumulo_tracer:12234 > > > > curl: (52) Empty reply from server > > > > > > > > Is there a way to set Trace hosts to accumulo_tracer:12234 > > > > > > > > Cheers!... > > > > ...Keith > > > > > > > > > > > > On 2022/07/28 23:40:18 Christopher wrote: > > > > > Do all servers have the same configuration? > > > > > > > > > > I would investigate the tablet server debug logs to determine if > > it's > > > > > having trouble setting up tracing. It should be able to locate the > > > > tracer > > > > > service by talking to ZooKeeper and observing its service address > > > > > advertisement, similar to how other servers register themselves in > > > > > ZooKeeper. I'm not a docker network expert, but whatever service > > > address > > > > > the tracer service is advertising there should be routable from > the > > > > tablet > > > > > servers. > > > > > > > > > > On Thu, Jul 28, 2022 at 7:31 PM kma <[email protected]> wrote: > > > > > > > > > > > Thanks again Christopher, > > > > > > > > > > > > Our environment is a little different from usual. > > > > > > > > > > > > We have accumulo tracer running in it's own docker container and > > the > > > > > > other accumulo services (e.g. gc, master, tserver, etc.) are > also > > > > > > running in different docker containers. We also have kerberos > > > enabled. > > > > > > > > > > > > The following is our trace related configurations > > > > > > > > > > > > default | trace.port.client ................. | 12234 > > > > > > default | trace.span.receivers .......... | > > > > > > org.apache.accumulo.tracer.ZooTraceClient > > > > > > default | trace.table ......................... | trace > > > > > > default | trace.token.type ................ | > > > > > > org.apache.accumulo.core.client.security.tokens.PasswordToken > > > > > > site | @override ...................... | > > > > > > org.apache.accumulo.core.client.security.tokens.KerberosToken > > > > > > default | trace.user .......................... | root > > > > > > site | @override ...................... | > > > > > > [email protected] > > > > > > default | trace.zookeeper.path ........ | /tracers > > > > > > > > > > > > And we can confirm that `trace on` and `trace off` works well in > > the > > > > > > accumulo tracer container where the tracer process is running. > > > > > > > > > > > > However, `trace on` and `trace off` does not work in any other > > > > > > containers. This is probably why we don't see compaction trace > > > > messages > > > > > > in the trace table. And it's likely because these containers > don't > > > > know > > > > > > where the tracer service is running ? > > > > > > > > > > > > Question: How does the other accumulo services know where tracer > > > > service > > > > > > is running ? > > > > > > Question: Is there a way to configure the tracer host where the > > > tracer > > > > > > service is running ? > > > > > > > > > > > > Cheers!... > > > > > > ...Keith > > > > > > > > > > > > > > > > > > On 2022/07/22 14:58:07 Christopher wrote: > > > > > > > I would double check your trace credentials in your Accumulo > > > > > > configuration > > > > > > > file. And double check that the user credentials used to write > > to > > > > the > > > > > > trace > > > > > > > table have permission to write to that table, and that you're > > > > running > > > > > > the > > > > > > > accumulo-tracer service. > > > > > > > > > > > > > > On Fri, Jul 22, 2022 at 10:45 AM kma <[email protected]> wrote: > > > > > > > > > > > > > > > Thank you Christopher, > > > > > > > > > > > > > > > > We can confirm that trace on and trace off are working fine > in > > > > > > accumulo > > > > > > > > shell. We can see those trace messages in the trace table > and > > > > tracer > > > > > > > > service is running. > > > > > > > > > > > > > > > > However, we are not seeing trace messages for MajC / MinC > > > > > > > > > > > > > > > > We do see these warnings in Monitor > > > > > > > > > > > > > > > > "Tracing spans are being dropped because there are already > > 5000 > > > > spans > > > > > > > > queued for delivery. This does not affect performance, > > security > > > or > > > > > > data > > > > > > > > integrity, but distributed tracing information is being > lost." > > > > > > > > > > > > > > > > Any suggestions ? > > > > > > > > > > > > > > > > Cheers!... > > > > > > > > ...Keith > > > > > > > > > > > > > > > > > > > > > > > > On 2022/07/12 19:42:07 Christopher wrote: > > > > > > > > > By default, the tracer service that collects traces and > > writes > > > > > > them to > > > > > > > > > the trace table does not run. Generally, it is also > > > > responsible for > > > > > > > > > creating the trace table. If you have an empty trace > table, > > > > was it > > > > > > > > > created by the tracer service, or did you create it > > manually? > > > > > > > > > > > > > > > > > > Assuming tracing is working as intended and there is a > > tracer > > > > > > service > > > > > > > > > running, there's only a few operations that trace > > > > automatically, and > > > > > > > > > even they do so at a small probability by default (like > > > > > > compactions). > > > > > > > > > If your system is just started and you're using default > > > > settings, it > > > > > > > > > may not have had a chance to create any traces yet. > > > > > > > > > > > > > > > > > > Some operations can be traced by explicitly turning on > > > > tracing once > > > > > > > > > everything is set up. You can test this by trying to turn > on > > > > tracing > > > > > > > > > in the shell, using the "trace on" command, and then > doing a > > > > scan, > > > > > > > > > then doing "trace off". I believe this still works as > > expected > > > > in > > > > > > > > > 2.0.1. This should trace client side operations through to > > the > > > > > > server, > > > > > > > > > initiated from the shell. This is also possible to be done > > in > > > > > > > > > user-code, but there is no public API for it. > > > > > > > > > > > > > > > > > > A lot of the behavior of tracing has had to change in the > > > > last few > > > > > > > > > versions because of the dependencies we previously used > for > > > > tracing > > > > > > > > > are no longer supported. Everything should still work like > > it > > > > did in > > > > > > > > > previous versions, but it's possible 2.0.1 does have > issues. > > > For > > > > > > 2.1, > > > > > > > > > we're expecting to use OpenTelemetry, which should make > > > > tracing much > > > > > > > > > easier and more reliable to configure. However, we do not > > > > provide a > > > > > > > > > built-in OpenTelemetry sink in 2.1 to write tracing > > > > information to a > > > > > > > > > table in Accumulo, like we did in previous versions for > > > HTrace. > > > > > > Such a > > > > > > > > > thing could be added by the user or a 3rd party, though. > > > > > > > > > > > > > > > > > > On Tue, Jul 12, 2022 at 3:25 PM kma <[email protected]> > > wrote: > > > > > > > > > > > > > > > > > > > > Hi Team, > > > > > > > > > > > > > > > > > > > > In one of out latest deployment of Accumulo 2.0.1, the > > trace > > > > > > table is > > > > > > > > > > empty. > > > > > > > > > > > > > > > > > > > > Question: is tracing automatically enabled by default ? > If > > > > not, > > > > > > how to > > > > > > > > > > enable it ? > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > ______________________________ > > > > > > > > > > Keith Ma > > > > > > > > > > Software Developer > > > > > > > > > > *PHEMI Systems* > > > > > > > > > > Suite 600 – 777 Hornby Street > > > > > > > > > > Vancouver, BC V6Z 1S4 > > > > > > > > > > 604-336-1119 > > > > > > > > > > > > > > > > > > > > website <http://www.phemi.com/> twitter > > > > > > > > > > <https://twitter.com/PHEMISystems>linkedin > > > > > > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > ______________________________ > > > > > > > > Keith Ma > > > > > > > > Software Developer > > > > > > > > *PHEMI Systems* > > > > > > > > Suite 600 – 777 Hornby Street > > > > > > > > Vancouver, BC V6Z 1S4 > > > > > > > > 604-336-1119 > > > > > > > > > > > > > > > > website <http://www.phemi.com/> twitter > > > > > > > > <https://twitter.com/PHEMISystems>linkedin > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > ______________________________ > > > > > > Keith Ma > > > > > > Software Developer > > > > > > *PHEMI Systems* > > > > > > Suite 600 – 777 Hornby Street > > > > > > Vancouver, BC V6Z 1S4 > > > > > > 604-336-1119 > > > > > > > > > > > > website <http://www.phemi.com/> twitter > > > > > > <https://twitter.com/PHEMISystems>linkedin > > > > > > < > > > > > > > > > > > > > > > > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > ______________________________ > > > > Keith Ma > > > > Software Developer > > > > *PHEMI Systems* > > > > Suite 600 – 777 Hornby Street > > > > Vancouver, BC V6Z 1S4 > > > > 604-336-1119 > > > > > > > > website <http://www.phemi.com/> twitter > > > > <https://twitter.com/PHEMISystems>linkedin > > > > < > > > > > > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > > > > > > >
