kevinwilfong commented on code in PR #10697:
URL:
https://github.com/apache/incubator-gluten/pull/10697#discussion_r2388654792
##########
cpp/velox/substrait/SubstraitToVeloxPlan.cc:
##########
@@ -1272,26 +1272,30 @@ core::PlanNodePtr
SubstraitToVeloxPlanConverter::toVeloxPlan(const ::substrait::
SubstraitParser::parseColumnTypes(baseSchema, columnTypes);
}
- // Velox requires Filter Pushdown must being enabled.
- bool filterPushdownEnabled = true;
auto names = colNameList;
auto types = veloxTypeList;
- auto dataColumns = ROW(std::move(names), std::move(types));
+ // The columns we project from the file.
+ auto baseSchema = ROW(std::move(names), std::move(types));
+ // The columns present in the table, if not available default to the
baseSchema.
+ auto tableSchema = splitInfo->tableSchema ? splitInfo->tableSchema :
baseSchema;
Review Comment:
Got it, will do.
As an alternative I could make the Validator more pessimistic when either of
these options are set and always validate the entire table schema, then
fallback to Vanilla Spark to avoid the failure. I'm not sure which is
preferrable (falling back to Vanilla Spark more often or failing in what I'm
assuming will be rare cases).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
