vvysotskyi commented on code in PR #2632:
URL: https://github.com/apache/drill/pull/2632#discussion_r954278444
##########
exec/java-exec/src/main/resources/drill-module.conf:
##########
@@ -115,7 +115,8 @@ drill.exec: {
text: {
buffer.size: 262144,
batch.size: 4000
- }
+ },
+ recursive_listing_max_size: 10000
Review Comment:
Yes, the default value should be adjusted. For the big data world, thousands
of files are quite a small amount. For non-parquet files FileStatus is small,
so it shouldn't cause large pressure on memory. For parquet files, it would be
good to provide the functionality to disable reading metadata for planning and
use it only during execution to avoid issues with huge files amount.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
