dmitry-chirkov-dremio commented on code in PR #54725:
URL: https://github.com/apache/arrow/pull/54725#discussion_r4346012264
##########
cpp/src/arrow/compute/kernels/vector_selection.cc:
##########
@@ -126,15 +167,25 @@ Result<Datum> DropNullRecordBatch(const
std::shared_ptr<RecordBatch>& batch,
bit_util::SetBitsTo(dst->mutable_data(), 0, batch->num_rows(), false);
break;
}
- if (column->null_bitmap_data()) {
+ if (is_union(column->type_id())) {
+ ARROW_ASSIGN_OR_RAISE(auto validity, MakeDropNullFilter(*column, ctx));
+ ::arrow::internal::BitmapAnd(validity->values()->data(),
validity->offset(),
+ dst->data(), 0, column->length(), 0,
+ dst->mutable_data());
+ } else if (column->null_bitmap_data()) {
::arrow::internal::BitmapAnd(column->null_bitmap_data(),
column->offset(),
dst->data(), 0, column->length(), 0,
dst->mutable_data());
}
}
auto drop_null_filter = std::make_shared<BooleanArray>(batch->num_rows(),
dst);
if (drop_null_filter->true_count() == 0) {
- return RecordBatch::MakeEmpty(batch->schema(), ctx->memory_pool());
+ ArrayVector empty_columns;
+ for (const auto& column : batch->columns()) {
+ ARROW_ASSIGN_OR_RAISE(auto empty, MakeDropNullEmptyArray(column->type(),
ctx));
Review Comment:
The extension fix misses a sibling-column case: an all-null union beside a
valid list<bool8> column containing [[], []].
Dropping all rows calls MakeEmptyArray for the list column, which fails with
"MakeBuilder: cannot construct builder for type extension<arrow.bool8>".
Could we handle this case too and add dense/sparse regressions for
RecordBatch and Table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
