wesm commented on a change in pull request #8188:
URL: https://github.com/apache/arrow/pull/8188#discussion_r490454944
##########
File path: python/pyarrow/_dataset.pyx
##########
@@ -1013,27 +1013,36 @@ cdef class ParquetReadOptions(_Weakrefable):
dictionary_columns : list of string, default None
Names of columns which should be dictionary encoded as
they are read.
+ enable_parallel_column_conversion : bool, default False
+ Whether single files may be read in parallel (ignored when reading
+ multiple files to avoid thread contention).
Review comment:
Add note that this is experimental. Once we fix our nested parallelism
problem this shouldn't be needed
##########
File path: python/pyarrow/_dataset.pyx
##########
@@ -1899,7 +1912,7 @@ cdef shared_ptr[CScanContext] _build_scan_context(bint
use_threads=True,
cdef void _populate_builder(const shared_ptr[CScannerBuilder]& ptr,
list columns=None, Expression filter=None,
- int batch_size=32*2**10) except *:
+ int batch_size=2**20) except *:
Review comment:
Can this be put in a global variable (`DEFAULT_BATCH_SIZE`) rather than
hard-coded?
##########
File path: cpp/src/arrow/dataset/file_parquet.h
##########
@@ -91,6 +91,10 @@ class ARROW_DS_EXPORT ParquetFileFormat : public FileFormat {
/// @{
std::unordered_set<std::string> dict_columns;
/// @}
+
+ /// Parallelize conversion across columns. This option is ignored if a
scan is already
+ /// parallelized across input files.
Review comment:
Add note that this is experimental. Once we fix our nested parallelism
problem this shouldn't be needed
##########
File path: python/pyarrow/_dataset.pyx
##########
@@ -1992,7 +2005,7 @@ cdef class Scanner(_Weakrefable):
def from_fragment(Fragment fragment not None, Schema schema=None,
bint use_threads=True, MemoryPool memory_pool=None,
list columns=None, Expression filter=None,
- int batch_size=32*2**10):
+ int batch_size=2**20):
Review comment:
use common variable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
