ErnestasVaiciukeviciusAdform opened a new issue, #9174:
URL: https://github.com/apache/storm/issues/9174
## Summary
When `topology.upstream.feedback.enable=true`, workers emit large volumes of
```
WARN o.a.s.utils.TransferDrainer - No remote destination available for task
<id>
```
at WARN level. The warning is not merely cosmetic: the upstream-feedback
tuple that triggered it is **dropped**. As a result, upstream feedback is only
ever delivered to tasks that live on the *same* worker; feedback destined for
upstream tasks on *other* workers never arrives.
Whether a given worker logs the warning (and drops feedback) depends
entirely on **task placement**, which makes the symptom intermittent and
misleading (see "Misleading symptom" below).
## Affected versions
- Reproduced on `v3.0.0`.
- Present on current `master` (the relevant code in
`WorkerState.workerOutboundTasks()` is unchanged).
## Environment
- Any topology running with `topology.upstream.feedback.enable=true` across
**more than one worker** where at least one worker does **not** host a
downstream consumer of a given upstream component.
## Steps to reproduce
1. Submit any multi-component topology (e.g. `spout -> bolt1 -> bolt2`) with
`topology.upstream.feedback.enable=true` (and
`topology.stats.ewma.enable=true`, which feedback requires).
2. Request enough workers that the components are spread out, so some worker
hosts upstream tasks of a component but none of that component's downstream
consumers.
3. Observe the worker logs.
## Expected behaviour
- Upstream feedback tuples are delivered to their target upstream tasks,
including tasks on remote workers.
- No WARN spam during normal operation.
## Actual behaviour
- `TransferDrainer.send` cannot resolve the destination task and logs `No
remote destination available for task {}` for every remote upstream-feedback
recipient, on every feedback tick.
- The corresponding feedback tuples are dropped, so remote upstream tasks
never receive feedback.
## Root cause
Upstream feedback is sent directly to **upstream (source) component task
ids** via `Task.sendUnanchoredFeedback`, bypassing stream grouping
(`Executor.sendUpstreamFeedback` → recipients from
`Executor.computeUpstreamTaskIds`, `Executor.java:603,642`).
A worker's routable remote destinations (`WorkerState.cachedTaskToNodePort`)
and its remote connections are built **only** from `outboundTasks`, which
`WorkerState.workerOutboundTasks()` computes as *the tasks of the
downstream-target components of the locally-hosted tasks* (`WorkerState.java`
`workerOutboundTasks`, consumed at `WorkerState.java:429`).
`addUpstreamFeedback` (`StormCommon.java:363`) declares `FEEDBACK_STREAM_ID`
only as an **output** stream on each component — it creates **no input/grouping
edge**. Consequently, upstream feedback recipients are never part of
`outboundTasks`. When such a recipient is remote, `taskToNode.get(taskId)` is
`null` in `TransferDrainer.send` (`TransferDrainer.java:72`), the tuple is
dropped, and the warning is logged.
## Misleading symptom (important)
`outboundTasks` membership is decided **per component**: if a component is a
downstream target of *any* local task, **all** of that component's tasks
(topology-wide) become routable.
Because of this:
- A worker produces **no warning** for feedback to an upstream component `X`
whenever `X` also happens to be a **downstream target** of some task co-located
on that same worker — the feedback rides the already-established connection and
looks healthy.
- The *same* feedback edge on a *different* worker (one hosting upstream
tasks of `X` but none of `X`'s downstream consumers) logs the warning and drops
the tuple.
Therefore **"no warnings" does not mean feedback is being delivered** — it
only means the scheduler happened to co-locate a downstream consumer of `X`. In
particular, if every worker hosts at least one task of every component, the
warnings disappear entirely even though the underlying routing gap still
exists. This placement dependence can easily mislead operators into believing
the feature works.
## Proposed fix
When upstream feedback is enabled, also include the non-system **source
(upstream) components** of the locally-hosted components when building
`outboundTasks`, mirroring `Executor.computeUpstreamTaskIds()` at worker
granularity. This makes every feedback recipient (bolt or spout) a routable
remote destination with a connection, delivering feedback correctly and
eliminating the spurious warning — independent of task placement.
The fix is limited to `WorkerState.workerOutboundTasks()`; behaviour only
changes when `topology.upstream.feedback.enable=true` (off by default).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
