Thanks again Christopher,
Our environment is a little different from usual.
We have accumulo tracer running in it's own docker container and the
other accumulo services (e.g. gc, master, tserver, etc.) are also
running in different docker containers. We also have kerberos enabled.
The following is our trace related configurations
default | trace.port.client ................. | 12234
default | trace.span.receivers .......... |
org.apache.accumulo.tracer.ZooTraceClient
default | trace.table ......................... | trace
default | trace.token.type ................ |
org.apache.accumulo.core.client.security.tokens.PasswordToken
site | @override ...................... |
org.apache.accumulo.core.client.security.tokens.KerberosToken
default | trace.user .......................... | root
site | @override ...................... |
accumulo-tra...@dev.phemi.com
default | trace.zookeeper.path ........ | /tracers
And we can confirm that `trace on` and `trace off` works well in the
accumulo tracer container where the tracer process is running.
However, `trace on` and `trace off` does not work in any other
containers. This is probably why we don't see compaction trace messages
in the trace table. And it's likely because these containers don't know
where the tracer service is running ?
Question: How does the other accumulo services know where tracer service
is running ?
Question: Is there a way to configure the tracer host where the tracer
service is running ?
Cheers!...
...Keith
On 2022/07/22 14:58:07 Christopher wrote:
> I would double check your trace credentials in your Accumulo
configuration
> file. And double check that the user credentials used to write to the
trace
> table have permission to write to that table, and that you're running the
> accumulo-tracer service.
>
> On Fri, Jul 22, 2022 at 10:45 AM kma <km...@phemi.com> wrote:
>
> > Thank you Christopher,
> >
> > We can confirm that trace on and trace off are working fine in accumulo
> > shell. We can see those trace messages in the trace table and tracer
> > service is running.
> >
> > However, we are not seeing trace messages for MajC / MinC
> >
> > We do see these warnings in Monitor
> >
> > "Tracing spans are being dropped because there are already 5000 spans
> > queued for delivery. This does not affect performance, security or data
> > integrity, but distributed tracing information is being lost."
> >
> > Any suggestions ?
> >
> > Cheers!...
> > ...Keith
> >
> >
> > On 2022/07/12 19:42:07 Christopher wrote:
> > > By default, the tracer service that collects traces and writes
them to
> > > the trace table does not run. Generally, it is also responsible for
> > > creating the trace table. If you have an empty trace table, was it
> > > created by the tracer service, or did you create it manually?
> > >
> > > Assuming tracing is working as intended and there is a tracer service
> > > running, there's only a few operations that trace automatically, and
> > > even they do so at a small probability by default (like compactions).
> > > If your system is just started and you're using default settings, it
> > > may not have had a chance to create any traces yet.
> > >
> > > Some operations can be traced by explicitly turning on tracing once
> > > everything is set up. You can test this by trying to turn on tracing
> > > in the shell, using the "trace on" command, and then doing a scan,
> > > then doing "trace off". I believe this still works as expected in
> > > 2.0.1. This should trace client side operations through to the
server,
> > > initiated from the shell. This is also possible to be done in
> > > user-code, but there is no public API for it.
> > >
> > > A lot of the behavior of tracing has had to change in the last few
> > > versions because of the dependencies we previously used for tracing
> > > are no longer supported. Everything should still work like it did in
> > > previous versions, but it's possible 2.0.1 does have issues. For 2.1,
> > > we're expecting to use OpenTelemetry, which should make tracing much
> > > easier and more reliable to configure. However, we do not provide a
> > > built-in OpenTelemetry sink in 2.1 to write tracing information to a
> > > table in Accumulo, like we did in previous versions for HTrace.
Such a
> > > thing could be added by the user or a 3rd party, though.
> > >
> > > On Tue, Jul 12, 2022 at 3:25 PM kma <km...@phemi.com> wrote:
> > > >
> > > > Hi Team,
> > > >
> > > > In one of out latest deployment of Accumulo 2.0.1, the trace
table is
> > > > empty.
> > > >
> > > > Question: is tracing automatically enabled by default ? If not,
how to
> > > > enable it ?
> > > >
> > > > --
> > > > ______________________________
> > > > Keith Ma
> > > > Software Developer
> > > > *PHEMI Systems*
> > > > Suite 600 – 777 Hornby Street
> > > > Vancouver, BC V6Z 1S4
> > > > 604-336-1119
> > > >
> > > > website <http://www.phemi.com/> twitter
> > > > <https://twitter.com/PHEMISystems>linkedin
> > > >
> > <
> >
http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1
> > >
> > > >
> > >
> > --
> > ______________________________
> > Keith Ma
> > Software Developer
> > *PHEMI Systems*
> > Suite 600 – 777 Hornby Street
> > Vancouver, BC V6Z 1S4
> > 604-336-1119
> >
> > website <http://www.phemi.com/> twitter
> > <https://twitter.com/PHEMISystems>linkedin
> > <
> >
http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1
> > >
> >
> >
>
--
______________________________
Keith Ma
Software Developer
*PHEMI Systems*
Suite 600 – 777 Hornby Street
Vancouver, BC V6Z 1S4
604-336-1119
website <http://www.phemi.com/> twitter
<https://twitter.com/PHEMISystems>linkedin
<http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1>