qstommyshu commented on code in PR #15610:
URL: https://github.com/apache/datafusion/pull/15610#discussion_r2040846556
##########
docs/source/user-guide/configs.md:
##########
@@ -84,6 +84,7 @@ Environment variables are read during `SessionConfig`
initialisation so they mus
| datafusion.execution.skip_physical_aggregate_schema_check |
false | When set to true, skips verifying that the schema
produced by planning the input of `LogicalPlan::Aggregate` exactly matches the
schema of the input plan. When set to false, if the schema does not match
exactly (including nullability and metadata), a planning error will be raised.
This is used to workaround bugs in the planner that are now caught by the new
schema verification step.
|
| datafusion.execution.sort_spill_reservation_bytes |
10485760 | Specifies the reserved memory for each spillable
sort operation to facilitate an in-memory merge. When a sort operation spills
to disk, the in-memory data must be sorted and merged before being written to a
file. This setting reserves a specific amount of memory for that in-memory
sort/merge process. Note: This setting is irrelevant if the sort operation
cannot spill (i.e., if there's no `DiskManager` configured).
|
| datafusion.execution.sort_in_place_threshold_bytes |
1048576 | When sorting, below what size should data be
concatenated and sorted in a single RecordBatch rather than sorted in batches
and merged.
|
+| datafusion.execution.sort_max_spill_merge_degree | 16
| When doing external sorting, the maximum number of
spilled files to read back at once. Those read files in the same merge step
will be sort- preserving-merged and re-spilled, and the step will be repeated
to reduce the number of spilled files in multiple passes, until a final sorted
run can be produced.
|
Review Comment:
Great attention to detail, for updating the user guide!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
