AntoinePrv commented on code in PR #50629:
URL: https://github.com/apache/arrow/pull/50629#discussion_r4216868930
##########
cpp/src/parquet/column_reader.h:
##########
@@ -289,11 +300,16 @@ class PARQUET_EXPORT RecordReader {
/// values
/// @param arrow_type Which type to read this column as (optional). Currently
/// only used for byte array columns (see
BinaryRecordReader::GetBuilderChunks).
+ /// @param flat_optional_optimization An optimization for flat (non nested)
+ /// optional values where definition levels are decoded directly into the
validity
+ /// bitmap. This delivers significant speedup, but `int16_t` definition
levels are
+ /// no longer available and `def_levels()` will throw.
static std::shared_ptr<RecordReader> Make(
const ColumnDescriptor* descr, LevelInfo leaf_info,
::arrow::MemoryPool* pool = ::arrow::default_memory_pool(),
bool read_dictionary = false, bool read_dense_for_nullable = false,
- const std::shared_ptr<::arrow::DataType>& arrow_type = NULLPTR);
+ const std::shared_ptr<::arrow::DataType>& arrow_type = NULLPTR,
+ bool flat_optional_optimization = false);
Review Comment:
I disagree. We only added this option because the optimization came with a
break to the existing contract (no definition levels available), otherwise we
would just have it always enabled. Having a generic option would means
opting-in to future unknown breakage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
