felipepessoto commented on code in PR #13136:
URL: https://github.com/apache/gluten/pull/13136#discussion_r4405635711
##########
cpp/velox/tests/Substrait2VeloxPlanConversionTest.cc:
##########
@@ -75,12 +93,89 @@ class Substrait2VeloxPlanConversionTest : public
exec::test::HiveConnectorTestBa
if (splitInfo->columnMappingMode.has_value()) {
splitBuilder.columnMappingMode(*splitInfo->columnMappingMode);
}
+ for (const auto& [name, value] : splitInfo->metadataColumns.at(i)) {
+ splitBuilder.infoColumn(name, value);
+ }
auto split = splitBuilder.build();
+ EXPECT_EQ(split->columnMappingMode, splitInfo->columnMappingMode);
splits.emplace_back(split);
}
return splits;
}
+ ::substrait::ReadRel makeRead(
+ const RowTypePtr& type,
+ const std::vector<::substrait::NamedStruct::ColumnType>& columnTypes =
{}) {
+ google::protobuf::Arena arena;
+ VeloxToSubstraitTypeConvertor typeConverter;
+ ::substrait::ReadRel read;
+ read.mutable_common()->mutable_direct();
+
read.mutable_base_schema()->CopyFrom(typeConverter.toSubstraitNamedStruct(arena,
type));
+ for (auto columnType : columnTypes) {
Review Comment:
`columnTypes` contains protobuf `NamedStruct::ColumnType` enum values, not
class objects. Copying these small scalar values is appropriate, so I am
keeping the by-value loop.
##########
.github/workflows/util/delta-spark-ut/known-failures.txt:
##########
@@ -369,14 +369,10 @@
org.apache.spark.sql.delta.deletionvectors.DeletionVectorsWithPredicatePushdownS
org.apache.spark.sql.delta.deletionvectors.DeletionVectorsWithPredicatePushdownSuite#PredicatePushdown:
Single deletion at the second row group. vectorizedReaderEnabled: true
readColumnarBatchAsRows: true
org.apache.spark.sql.delta.deletionvectors.DeletionVectorsWithPredicatePushdownSuite#huge
table: delete a small number of rows from tables of 2B rows with DVs
org.apache.spark.sql.delta.deletionvectors.DeletionVectorsWithPredicatePushdownSuite#huge
table: read from tables of 2B rows with existing DV of many zeros
-org.apache.spark.sql.delta.generatedsuites.dataskipping.DataSkippingDeltaV1CheckpointV2JsonSuite#Data
skipping handles aliasing for _metadata fields
-org.apache.spark.sql.delta.generatedsuites.dataskipping.DataSkippingDeltaV1CheckpointV2JsonSuite#Data
skipping handles aliasing for _metadata fields - old behavior with DataFrame
schema
org.apache.spark.sql.delta.generatedsuites.dataskipping.DataSkippingDeltaV1CheckpointV2JsonSuite#data
skipping flags
Review Comment:
The deleted entries already use Delta 4.4.1's
`org.apache.spark.sql.delta.generatedsuites.dataskipping` namespace, including
`DataSkippingDeltaV1CheckpointV2JsonSuite` and
`DataSkippingDeltaV1CheckpointV2ParquetSuite`. Their renaming happened on
`main`, so this PR's diff only shows the removals.
The current `delta4.version` is 4.4.1, and the workflow selects that release
from the POM. I agree that the older 4.2.0 result alone is insufficient; the
description labels it historical. The current-version [Delta
run](https://github.com/apache/gluten/actions/runs/40754485094) is in progress.
Leaving this thread open until the current-version result is confirmed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
