github-actions[bot] commented on code in PR #68335:
URL: https://github.com/apache/doris/pull/68335#discussion_r4061175922
##########
be/src/format_v2/lance/lance_reader_helper.cpp:
##########
@@ -373,89 +354,6 @@ Status set_lance_nested_type(std::string_view field_name,
return Status::OK();
}
-// Check whether an Arrow type tree contains a registered extension wrapper.
-bool type_contains_registered_extension(const
std::shared_ptr<arrow::DataType>& type) {
- if (type->id() == arrow::Type::EXTENSION) {
- return true;
- }
- for (const auto& field : type->fields()) {
- if (type_contains_registered_extension(field->type())) {
- return true;
- }
- }
- return false;
-}
-
-// Remove registered ExtensionArray wrappers only along extension-bearing
branches.
-Status unwrap_lance_extension_arrays(const std::shared_ptr<arrow::DataType>&
expected_type,
- const std::shared_ptr<arrow::Array>&
array,
- std::shared_ptr<arrow::Array>* unwrapped)
{
- DORIS_CHECK(expected_type != nullptr);
- DORIS_CHECK(array != nullptr);
- DORIS_CHECK(unwrapped != nullptr);
-
- auto storage_array = array;
- auto expected_storage_type = expected_type;
- if (expected_type->id() == arrow::Type::EXTENSION) {
- const auto extension_type =
std::dynamic_pointer_cast<arrow::ExtensionType>(expected_type);
- if (extension_type == nullptr) {
- return Status::InvalidArgument("invalid expected Arrow extension
type {}",
- expected_type->ToString());
- }
- expected_storage_type = extension_type->storage_type();
- }
- if (array->type_id() == arrow::Type::EXTENSION) {
- const auto extension_array =
std::dynamic_pointer_cast<arrow::ExtensionArray>(array);
- if (extension_array == nullptr) {
- return Status::InvalidArgument("invalid Arrow extension array: {}",
- array->type()->ToString());
- }
- storage_array = extension_array->storage();
- }
-
- const auto& child_data = storage_array->data()->child_data;
- const auto& child_fields = expected_storage_type->fields();
- if (child_data.empty()) {
Review Comment:
Because this compaction now runs before child normalizers, a sliced
LIST/STRUCT whose child is a registered ExtensionType reaches
arrow::MakeBuilder(list<extension>). Arrow 24 returns NotImplemented for
ExtensionType builders, so the existing ReadsRegisteredJsonNestedInSlicedList
path fails instead of unwrapping the child. The old routine unwrapped first.
Please unwrap/rebuild extension children before compacting the parent (or
compact with storage-type builders) and retain this regression coverage.
##########
be/src/format_v2/table/lance_reader.cpp:
##########
@@ -28,57 +28,17 @@
#include <cstring>
#include <limits>
#include <memory>
-#include <unordered_set>
#include "common/config.h"
-#include "common/consts.h"
#include "common/logging.h"
-#include "core/column/column_nullable.h"
Review Comment:
This new null-counter guard makes scanner-less row-ID fetches lose cache
statistics. _init_scanner_profile is only called from _open_scanner, but
read_by_row_ids opens the dataset and calls take_rows without a scanner;
_close_dataset therefore reaches this guard with null counters and skips
lance_dataset_get_data_cache_statistics. Before the refactor the counters were
initialized in init, so phase-two I/O exposed these bytes. Please
initialize/collect the cache counters for take_rows as well, or explicitly wire
those statistics into the row-ID fetch profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
