[
https://issues.apache.org/jira/browse/SOLR-18682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amrit Sarkar updated SOLR-18682:
--------------------------------
Description:
When the Collection API runs on the Overseer, a traced request like {{RELOAD}}
or {{CREATE}} shows up as two unrelated traces: one for the HTTP request, and
one for the command span (e.g. {{ReloadCollectionCmd}}) with all its
{{/admin/cores}} sub-requests. You can't follow the request from the client
down to the cores.
With the distributed Collection API it already works, because the command runs
in an executor that carries the caller's context. The Overseer path loses it:
the message goes through the ZK queue, and {{CollApiCmds.TraceAwareCommand}}
starts its span with no parent. {{TestDistributedTracing}} (added in
SOLR-17011) currently asserts the two separate traces.
*Proposed fix*
* {{CollectionsHandler.submitCollectionApiCommand}} adds the caller's trace
context to the Overseer message, next to {{async}} and the calling lock id.
* {{TraceAwareCommand}} uses it as the parent of the command span. An unsampled
caller stays unsampled, as in distributed mode.
* MODIFYCOLLECTION is left out: {{ModifyCollectionCmd}} waits for every message
key to appear in the collection state, so an older Overseer would time out
during a rolling upgrade. Its traces stay separate, as today.
* Messages without the keys (from an older node, or with tracing disabled)
start a new trace, as today.
Config Set API messages take the same path and can get the same fix in a
follow-up.
was:
When the Collection API runs on the Overseer, a traced request like {{RELOAD}}
or {{CREATE}} shows up as two unrelated traces: one for the HTTP request, and
one for the command span (e.g. {{ReloadCollectionCmd}}) with all its
{{/admin/cores}} sub-requests. You can't follow the request from the client
down to the cores.
With the distributed Collection API it already works, because the command runs
in an executor that carries the caller's context. The Overseer path loses it:
the message goes through the ZK queue, and {{CollApiCmds.TraceAwareCommand}}
starts its span with no parent. {{TestDistributedTracing}} (added in
SOLR-17011) currently asserts the two separate traces.
*Proposed fix*
* {{CollectionsHandler.submitCollectionApiCommand}} adds the sampled caller's
trace context to the Overseer message, next to {{async}} and the calling lock
id.
* {{TraceAwareCommand}} uses it as the parent of the command span, and drops
those keys before the command reads the message (MODIFYCOLLECTION validates
every message key against the collection state).
* Messages without the keys (from an older node during a rolling upgrade, or an
unsampled request) start a new trace, as today.
Config Set API messages take the same path and can get the same fix in a
follow-up.
> Collection API commands run by the Overseer start a new trace instead of
> continuing the caller's
> ------------------------------------------------------------------------------------------------
>
> Key: SOLR-18682
> URL: https://issues.apache.org/jira/browse/SOLR-18682
> Project: Solr
> Issue Type: Bug
> Components: tracing
> Reporter: Amrit Sarkar
> Priority: Major
>
> When the Collection API runs on the Overseer, a traced request like
> {{RELOAD}} or {{CREATE}} shows up as two unrelated traces: one for the HTTP
> request, and one for the command span (e.g. {{ReloadCollectionCmd}}) with all
> its {{/admin/cores}} sub-requests. You can't follow the request from the
> client down to the cores.
> With the distributed Collection API it already works, because the command
> runs in an executor that carries the caller's context. The Overseer path
> loses it: the message goes through the ZK queue, and
> {{CollApiCmds.TraceAwareCommand}} starts its span with no parent.
> {{TestDistributedTracing}} (added in SOLR-17011) currently asserts the two
> separate traces.
> *Proposed fix*
> * {{CollectionsHandler.submitCollectionApiCommand}} adds the caller's trace
> context to the Overseer message, next to {{async}} and the calling lock id.
> * {{TraceAwareCommand}} uses it as the parent of the command span. An
> unsampled caller stays unsampled, as in distributed mode.
> * MODIFYCOLLECTION is left out: {{ModifyCollectionCmd}} waits for every
> message key to appear in the collection state, so an older Overseer would
> time out during a rolling upgrade. Its traces stay separate, as today.
> * Messages without the keys (from an older node, or with tracing disabled)
> start a new trace, as today.
> Config Set API messages take the same path and can get the same fix in a
> follow-up.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]