Re: [PR] [Parquet] perf: Create PrimitiveArrays directly rather than via `ArrayData` [arrow-rs]
alamb-ghbot commented on PR #9122: URL: https://github.com/apache/arrow-rs/pull/9122#issuecomment-3730447632 🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing alamb/less_primitive_allocations (4148a3d53f0f18a760fd573aae65a9b104ef9da1) to 96637fc8b928a94de53bbec3501337c0ecfbf936 [diff](https://github.com/apache/arrow-rs/compare/96637fc8b928a94de53bbec3501337c0ecfbf936..4148a3d53f0f18a760fd573aae65a9b104ef9da1) BENCH_NAME=arrow_reader BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench arrow_reader BENCH_FILTER= BENCH_BRANCH_NAME=alamb_less_primitive_allocations Results will be posted here when complete -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [Parquet] perf: Create PrimitiveArrays directly rather than via `ArrayData` [arrow-rs]
alamb commented on PR #9122: URL: https://github.com/apache/arrow-rs/pull/9122#issuecomment-3729412464 run benchmark arrow_reader arrow_reader_clickbench -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [Parquet] perf: Create PrimitiveArrays directly rather than via `ArrayData` [arrow-rs]
alamb commented on code in PR #9122:
URL: https://github.com/apache/arrow-rs/pull/9122#discussion_r2676635354
##
parquet/src/arrow/array_reader/primitive_array.rs:
##
@@ -212,41 +211,69 @@ where
.consume_record_data()
.into_buffer(target_type);
-let array_data = ArrayDataBuilder::new(arrow_data_type)
-.len(self.record_reader.num_values())
-.add_buffer(record_data)
-.null_bit_buffer(self.record_reader.consume_bitmap_buffer());
+let len = self.record_reader.num_values();
+let nulls = self
+.record_reader
+.consume_bitmap_buffer()
+.map(|b| NullBuffer::new(BooleanBuffer::new(b, 0, len)));
-let array_data = unsafe { array_data.build_unchecked() };
Review Comment:
the point is to avoid this ArrayData as we know here what types of arrays
will be built
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
