dwsmith1983 opened a new issue, #5943:
URL: https://github.com/apache/datafusion-comet/issues/5943
### What is the problem the feature request solves?
`common.source` in the Delta contrib's scan builders
(`CometDeltaNativeScan.scala`) is derived from `simpleStringWithNodeId()`,
which reads Spark's explain-local id map and can yield `(unknown)` during
ordinary planning. Two independently converted scans of the same table with
identical common fields but different partition selections can then receive the
same injection key, and if both contribute to one collection scope the map
merge keeps only one payload. Exchange boundaries isolate the common self-join
shapes today, so this is a key-contract concern rather than an observed wrong
answer.
### Describe the potential solution
Use the original physical scan's `SparkPlan.id` in both builders, for
example `s"${scanExec.nodeName} (${scanExec.id})"`, and add a regression that
clears the explain map, converts two scans with disjoint selected files, and
asserts distinct keys, with an equivalent-scan control that keeps `sameResult`
and the semantic hashes unchanged. The key stays excluded from semantic
equality.
### Additional context
Raised in the #5365 review (item 1). Filed as a follow-up so the contrib can
land and the change gets its own review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
