Copilot commented on code in PR #50483:
URL: https://github.com/apache/arrow/pull/50483#discussion_r3630527766
##########
cpp/src/arrow/csv/chunker.cc:
##########
@@ -56,6 +56,10 @@ class Lexer {
// Decide whether it's worth using a bulk filter over the given data area
bool ShouldUseBulkFilter(const char* data, const char* data_end) {
+ if (!bulk_filter_.CanUseOnBlock(
+ std::string_view(data, static_cast<size_t>(data_end - data)))) {
+ return false;
+ }
Review Comment:
`ShouldUseBulkFilter()` now calls `bulk_filter_.CanUseOnBlock()` on the full
`[data, data_end)` range before doing the existing 32-word probe. On SSE4.2
builds this performs a full `memchr` scan even in cases where the probe would
quickly decide bulk filtering is not worthwhile, adding avoidable O(n) work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
