If the version being used is 2.0.1, then that version is still affected by the bug you mention. However, since TraceServer never used AbstractServer, it was not affected and should still work fine.
On Fri, Jul 29, 2022 at 3:05 PM Dave Marion <dmario...@gmail.com> wrote: > Ok, I wasn't sure what version was being used here. I saw a reference to > 2.0.1 in an earlier email. > > On Fri, Jul 29, 2022 at 3:03 PM Christopher <ctubb...@apache.org> wrote: > > > That was a bug in the new AbstractServer class in 2.x. I don't think it > > ever affected 1.x > > I checked the 1.10 code and it wouldn't affect the tracer server. `-a` > > should still work fine there. > > > > On Fri, Jul 29, 2022 at 3:00 PM Dave Marion <dmario...@gmail.com> wrote: > > > > > https://github.com/apache/accumulo/pull/2119 fixed a bug in 2.1 where > > the > > > -a argument was not correctly setting the hostname. It looks like 2.0.1 > > is > > > affected by this too. > > > > > > On Fri, Jul 29, 2022 at 12:57 PM Christopher <ctubb...@apache.org> > > wrote: > > > > > > > The tracer should be advertising its own address in ZK. By default, > the > > > > server listens on `0.0.0.0`, unless `-a` or `--address` is specified > on > > > the > > > > command-line when it is started. Most server types use a utility > class > > > that > > > > will use `InetAddress.getLocalHost().getCanonicalHostName()` for the > > > > advertisement address if it sees that it is listening on `0.0.0.0`. > > > > However, it looks like the tracer doesn't do this. > > > > > > > > I would try to verify that `0.0.0.0` actually appears as the > advertised > > > > address in ZK for the tracer service, just be sure it's not a > > > > ZooTraceClient bug on the tablet server side (but I'm pretty sure > it's > > > not, > > > > after looking at the code). > > > > > > > > As a workaround, I would start up the tracer service using a script > > that > > > > sets `--address accumulo_tracer` on the command-line, where > > > > "accumulo_tracer" is the hostname or IP address that you want the > > tracer > > > > listening on and advertising to other servers. This address should be > > > > bindable where the tracer is running, and reachable from wherever the > > > > tservers are (so make sure it's not a private address only available > > > inside > > > > the container running the tracer). The reason for this is that the > > > hostname > > > > will resolve to an IP address inside the tracer process, and then the > > > > tracer will advertise the IP address. It won't advertise the original > > > > hostname, if you specified a hostname. > > > > > > > > I hope that helps. > > > > > > > > > > > > On Fri, Jul 29, 2022 at 12:18 PM kma <k...@phemi.com> wrote: > > > > > > > > > Nice, > > > > > > > > > > Turn on debug really help. > > > > > > > > > > 2022-07-29 12:12:14,235 [tracer.ZooTraceClient] DEBUG: Scanning > trace > > > > > hosts in zookeeper: /tracers > > > > > 2022-07-29 12:12:14,240 [tracer.ZooTraceClient] DEBUG: Trace hosts: > > > > > [0.0.0.0:12234, 0.0.0.0:12234] > > > > > 2022-07-29 12:12:14,240 [tracer.ZooTraceClient] DEBUG: Successfully > > > > > initialized tracer hosts from ZooKeeper > > > > > > > > > > [root@accumulo_gc /]# curl 0.0.0.0:12234 > > > > > curl: (7) Failed connect to 0.0.0.0:12234; Connection refused > > > > > > > > > > [root@accumulo_gc /]# curl accumulo_tracer:12234 > > > > > curl: (52) Empty reply from server > > > > > > > > > > Is there a way to set Trace hosts to accumulo_tracer:12234 > > > > > > > > > > Cheers!... > > > > > ...Keith > > > > > > > > > > > > > > > On 2022/07/28 23:40:18 Christopher wrote: > > > > > > Do all servers have the same configuration? > > > > > > > > > > > > I would investigate the tablet server debug logs to determine if > > > it's > > > > > > having trouble setting up tracing. It should be able to locate > the > > > > > tracer > > > > > > service by talking to ZooKeeper and observing its service > address > > > > > > advertisement, similar to how other servers register themselves > in > > > > > > ZooKeeper. I'm not a docker network expert, but whatever service > > > > address > > > > > > the tracer service is advertising there should be routable from > > the > > > > > tablet > > > > > > servers. > > > > > > > > > > > > On Thu, Jul 28, 2022 at 7:31 PM kma <km...@phemi.com> wrote: > > > > > > > > > > > > > Thanks again Christopher, > > > > > > > > > > > > > > Our environment is a little different from usual. > > > > > > > > > > > > > > We have accumulo tracer running in it's own docker container > and > > > the > > > > > > > other accumulo services (e.g. gc, master, tserver, etc.) are > > also > > > > > > > running in different docker containers. We also have kerberos > > > > enabled. > > > > > > > > > > > > > > The following is our trace related configurations > > > > > > > > > > > > > > default | trace.port.client ................. | 12234 > > > > > > > default | trace.span.receivers .......... | > > > > > > > org.apache.accumulo.tracer.ZooTraceClient > > > > > > > default | trace.table ......................... | trace > > > > > > > default | trace.token.type ................ | > > > > > > > org.apache.accumulo.core.client.security.tokens.PasswordToken > > > > > > > site | @override ...................... | > > > > > > > org.apache.accumulo.core.client.security.tokens.KerberosToken > > > > > > > default | trace.user .......................... | root > > > > > > > site | @override ...................... | > > > > > > > accumulo-tra...@dev.phemi.com > > > > > > > default | trace.zookeeper.path ........ | /tracers > > > > > > > > > > > > > > And we can confirm that `trace on` and `trace off` works well > in > > > the > > > > > > > accumulo tracer container where the tracer process is running. > > > > > > > > > > > > > > However, `trace on` and `trace off` does not work in any other > > > > > > > containers. This is probably why we don't see compaction trace > > > > > messages > > > > > > > in the trace table. And it's likely because these containers > > don't > > > > > know > > > > > > > where the tracer service is running ? > > > > > > > > > > > > > > Question: How does the other accumulo services know where > tracer > > > > > service > > > > > > > is running ? > > > > > > > Question: Is there a way to configure the tracer host where > the > > > > tracer > > > > > > > service is running ? > > > > > > > > > > > > > > Cheers!... > > > > > > > ...Keith > > > > > > > > > > > > > > > > > > > > > On 2022/07/22 14:58:07 Christopher wrote: > > > > > > > > I would double check your trace credentials in your Accumulo > > > > > > > configuration > > > > > > > > file. And double check that the user credentials used to > write > > > to > > > > > the > > > > > > > trace > > > > > > > > table have permission to write to that table, and that > you're > > > > > running > > > > > > > the > > > > > > > > accumulo-tracer service. > > > > > > > > > > > > > > > > On Fri, Jul 22, 2022 at 10:45 AM kma <km...@phemi.com> > wrote: > > > > > > > > > > > > > > > > > Thank you Christopher, > > > > > > > > > > > > > > > > > > We can confirm that trace on and trace off are working > fine > > in > > > > > > > accumulo > > > > > > > > > shell. We can see those trace messages in the trace table > > and > > > > > tracer > > > > > > > > > service is running. > > > > > > > > > > > > > > > > > > However, we are not seeing trace messages for MajC / MinC > > > > > > > > > > > > > > > > > > We do see these warnings in Monitor > > > > > > > > > > > > > > > > > > "Tracing spans are being dropped because there are already > > > 5000 > > > > > spans > > > > > > > > > queued for delivery. This does not affect performance, > > > security > > > > or > > > > > > > data > > > > > > > > > integrity, but distributed tracing information is being > > lost." > > > > > > > > > > > > > > > > > > Any suggestions ? > > > > > > > > > > > > > > > > > > Cheers!... > > > > > > > > > ...Keith > > > > > > > > > > > > > > > > > > > > > > > > > > > On 2022/07/12 19:42:07 Christopher wrote: > > > > > > > > > > By default, the tracer service that collects traces and > > > writes > > > > > > > them to > > > > > > > > > > the trace table does not run. Generally, it is also > > > > > responsible for > > > > > > > > > > creating the trace table. If you have an empty trace > > table, > > > > > was it > > > > > > > > > > created by the tracer service, or did you create it > > > manually? > > > > > > > > > > > > > > > > > > > > Assuming tracing is working as intended and there is a > > > tracer > > > > > > > service > > > > > > > > > > running, there's only a few operations that trace > > > > > automatically, and > > > > > > > > > > even they do so at a small probability by default (like > > > > > > > compactions). > > > > > > > > > > If your system is just started and you're using default > > > > > settings, it > > > > > > > > > > may not have had a chance to create any traces yet. > > > > > > > > > > > > > > > > > > > > Some operations can be traced by explicitly turning on > > > > > tracing once > > > > > > > > > > everything is set up. You can test this by trying to > turn > > on > > > > > tracing > > > > > > > > > > in the shell, using the "trace on" command, and then > > doing a > > > > > scan, > > > > > > > > > > then doing "trace off". I believe this still works as > > > expected > > > > > in > > > > > > > > > > 2.0.1. This should trace client side operations through > to > > > the > > > > > > > server, > > > > > > > > > > initiated from the shell. This is also possible to be > done > > > in > > > > > > > > > > user-code, but there is no public API for it. > > > > > > > > > > > > > > > > > > > > A lot of the behavior of tracing has had to change in > the > > > > > last few > > > > > > > > > > versions because of the dependencies we previously used > > for > > > > > tracing > > > > > > > > > > are no longer supported. Everything should still work > like > > > it > > > > > did in > > > > > > > > > > previous versions, but it's possible 2.0.1 does have > > issues. > > > > For > > > > > > > 2.1, > > > > > > > > > > we're expecting to use OpenTelemetry, which should make > > > > > tracing much > > > > > > > > > > easier and more reliable to configure. However, we do > not > > > > > provide a > > > > > > > > > > built-in OpenTelemetry sink in 2.1 to write tracing > > > > > information to a > > > > > > > > > > table in Accumulo, like we did in previous versions for > > > > HTrace. > > > > > > > Such a > > > > > > > > > > thing could be added by the user or a 3rd party, though. > > > > > > > > > > > > > > > > > > > > On Tue, Jul 12, 2022 at 3:25 PM kma <km...@phemi.com> > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hi Team, > > > > > > > > > > > > > > > > > > > > > > In one of out latest deployment of Accumulo 2.0.1, the > > > trace > > > > > > > table is > > > > > > > > > > > empty. > > > > > > > > > > > > > > > > > > > > > > Question: is tracing automatically enabled by default > ? > > If > > > > > not, > > > > > > > how to > > > > > > > > > > > enable it ? > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > ______________________________ > > > > > > > > > > > Keith Ma > > > > > > > > > > > Software Developer > > > > > > > > > > > *PHEMI Systems* > > > > > > > > > > > Suite 600 – 777 Hornby Street > > > > > > > > > > > Vancouver, BC V6Z 1S4 > > > > > > > > > > > 604-336-1119 > > > > > > > > > > > > > > > > > > > > > > website <http://www.phemi.com/> twitter > > > > > > > > > > > <https://twitter.com/PHEMISystems>linkedin > > > > > > > > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > ______________________________ > > > > > > > > > Keith Ma > > > > > > > > > Software Developer > > > > > > > > > *PHEMI Systems* > > > > > > > > > Suite 600 – 777 Hornby Street > > > > > > > > > Vancouver, BC V6Z 1S4 > > > > > > > > > 604-336-1119 > > > > > > > > > > > > > > > > > > website <http://www.phemi.com/> twitter > > > > > > > > > <https://twitter.com/PHEMISystems>linkedin > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > ______________________________ > > > > > > > Keith Ma > > > > > > > Software Developer > > > > > > > *PHEMI Systems* > > > > > > > Suite 600 – 777 Hornby Street > > > > > > > Vancouver, BC V6Z 1S4 > > > > > > > 604-336-1119 > > > > > > > > > > > > > > website <http://www.phemi.com/> twitter > > > > > > > <https://twitter.com/PHEMISystems>linkedin > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > ______________________________ > > > > > Keith Ma > > > > > Software Developer > > > > > *PHEMI Systems* > > > > > Suite 600 – 777 Hornby Street > > > > > Vancouver, BC V6Z 1S4 > > > > > 604-336-1119 > > > > > > > > > > website <http://www.phemi.com/> twitter > > > > > <https://twitter.com/PHEMISystems>linkedin > > > > > < > > > > > > > > > > > > > > > http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1 > > > > > > > > > > > > > > > > > > > > > > > > > >
