eugenegujing opened a new issue, #7925:
URL: https://github.com/apache/texera/issues/7925
### What happened?
The Projection operator's "Drop Option" (`isDrop = true`) computes "which
attributes to remove" in two places that disagree on whether attribute names
are case-sensitive.
The descriptor derives the output schema through `Schema.remove`, which
lowercases both sides before comparing:
```scala
//
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/projection/ProjectionOpDesc.scala:67-71
} else {
attributes.foldLeft(inputSchema) { (schema, attribute) =>
schema.remove(attribute.getOriginalAttribute)
}
}
```
```scala
//
common/workflow-core/src/main/scala/org/apache/texera/amber/core/tuple/Schema.scala:170-174
def remove(attributeNames: Iterable[String]): Schema = {
val attributesToRemove = attributeNames.map(_.toLowerCase).toSet
```
The executor computes the kept set with `List.diff`, which compares names
exactly:
```scala
//
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/projection/ProjectionOpExec.scala:40-43
if (desc.isDrop) {
val allAttribute = tuple.schema.getAttributeNames
val selectedAttributes = desc.attributes.map(_.getOriginalAttribute)
val keepAttributes = allAttribute.diff(selectedAttributes)
```
So for an input schema containing `field1` and a drop entry spelled
`Field1`: the descriptor removes `field1` from the declared output schema,
while the executor's `diff` does not match it and `project()` keeps it. A
genuinely misspelled attribute fails at plan time (`Schema.remove` throws on
non-existent names), but a merely differently-cased one passes that check and
diverges silently at runtime.
The divergence does not reach downstream operators in the current engine:
`DataProcessor` re-projects the executor's `MapTupleLike` onto the declared
port schema by attribute name, which discards the extra field, so the
end-to-end output happens to match the declared schema. The inconsistency is
internal to the operator, and end-to-end correctness currently depends on that
name-based re-projection rather than on the two halves agreeing. The `Schema`
class is otherwise uniformly case-insensitive (`getIndex`, `containsAttribute`,
`add`, `remove` all lowercase before comparing); the executor's raw `diff` is
the outlier.
The executor-side behavior is pinned by the drop-mode spec `"match drop
names case-sensitively"` in `ProjectionOpExecSpec`, whose comment notes that it
diverges from the descriptor.
### How to reproduce?
Reproducible at the unit level, against a schema with attributes `field1`,
`field2`, `field3`:
1. Configure a `ProjectionOpDesc` with `isDrop = true` and a single drop
entry spelled `FIELD2`.
2. Derive the output schema through the descriptor: `Schema.remove` matches
case-insensitively, so the declared output schema is `field1`, `field3`.
3. Feed one tuple through `ProjectionOpExec.project` with the same
configuration: the executor's `diff` matches exactly, finds no attribute named
`FIELD2`, and the returned tuple still contains all three fields — including
the `field2` the declared schema removed.
The existing spec `"match drop names case-sensitively"` in
`ProjectionOpExecSpec` runs exactly step 3 and pins its result, so running that
spec is the fastest way to see the divergence.
There are no end-to-end steps because the divergence is not observable from
a running workflow: creating the case-mismatched configuration requires
bypassing the UI in the first place (the attribute picker autofills exact
names, so it takes a hand-edited or imported workflow JSON, or an upstream
schema whose casing changed after the operator was configured), and even then
the engine re-projects the executor's output onto the declared schema before it
reaches downstream operators or storage, so the final result looks correct
regardless.
### Version/Branch
1.3.0-incubating-SNAPSHOT (main)
### Commit Hash (Optional)
_No response_
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
```shell
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
