[ https://issues.apache.org/jira/browse/DRILL-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854937#comment-15854937 ]
ASF GitHub Bot commented on DRILL-5080: --------------------------------------- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/717#discussion_r99250103 --- Diff: exec/java-exec/src/main/resources/drill-module.conf --- @@ -177,13 +177,47 @@ drill.exec: { sort: { purge.threshold : 1000, external: { - batch.size : 4000, + // Drill uses the managed External Sort Batch by default. + // Set this to true to use the legacy, unmanaged version. + // Disabled in the intial commit, to be enabled after + // tests are committed. + disable_managed: true + // Limit on the number of batches buffered in memory. + // Primarily for testing. + // 0 = unlimited + batch_limit: 0 + // Limit on the amount of memory used for xsort. Overrides the + // value provided by Foreman. Primarily for testing. + // 0 = unlimited, Supports HOCON memory suffixes. + mem_limit: 0 + // Limit on the number of spilled batches that can be merged in + // a single pass. Limits the number of open file handles. + // 0 = unlimited + merge_limit: 0 spill: { - batch.size : 4000, - group.size : 40000, - threshold : 40000, - directories : [ "/tmp/drill/spill" ], - fs : "file:///" + // Deprecated for managed xsort; used only by legacy xsort + group.size: 40000, + // Deprecated for managed xsort; used only by legacy xsort + threshold: 40000, + // Minimum number of in-memory batches to spill per spill file + // Affects only spilling from memory to disk. + // Primarily for testing. + min_batches: 2, + // Maximum number of in-memory batches to spill per spill file + // Affects only spilling from memory to disk. + // Primarily for testing. + // 0 = unlimited + max_batches: 0, + // File system to use. Local file system by default. + fs: "file:///" + // List of directories to use. Directories are created --- End diff -- Here that is implied by the JSON-like syntax. > Create a memory-managed version of the External Sort operator > ------------------------------------------------------------- > > Key: DRILL-5080 > URL: https://issues.apache.org/jira/browse/DRILL-5080 > Project: Apache Drill > Issue Type: Improvement > Affects Versions: 1.8.0 > Reporter: Paul Rogers > Assignee: Paul Rogers > Fix For: 1.10.0 > > Attachments: ManagedExternalSortDesign.pdf > > > We propose to create a "managed" version of the external sort operator that > works to a clearly-defined memory limit. Attached is a design specification > for the work. > The project will include fixing a number of bugs related to the external > sort, include as sub-tasks of this umbrella task. -- This message was sent by Atlassian JIRA (v6.3.15#6346)