[
https://issues.apache.org/jira/browse/DRILL-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546512#comment-14546512
]
Hao Zhu commented on DRILL-2100:
--------------------------------
Tested on Drill 1.0, when query finishes successfully, the spill directories
remain but the files are deleted.
The minimum reproduce is on a single node cluster:
{code}
alter system set `planner.memory.max_query_memory_per_node`=21474836;
select count(*) from
(
select columns[5] from dfs.root.`user/hive/warehouse/passwords_csv_middle`
order by columns[0], columns[1],columns[2]
);
{code}
The table "passwords_csv_middle" is about 400MB.
{code}
[root@h1 spill]# ls -altr 2aa9600f-016a-5283-f98e-ef22942981c2/*/*/*/
2aa9600f-016a-5283-f98e-ef22942981c2/major_fragment_2/minor_fragment_5/operator_2/:
total 8
drwxr-xr-x 3 mapr mapr 4096 May 16 01:40 ..
drwxr-xr-x 2 mapr mapr 4096 May 16 01:41 .
2aa9600f-016a-5283-f98e-ef22942981c2/major_fragment_2/minor_fragment_4/operator_2/:
total 8
drwxr-xr-x 3 mapr mapr 4096 May 16 01:40 ..
drwxr-xr-x 2 mapr mapr 4096 May 16 01:41 .
2aa9600f-016a-5283-f98e-ef22942981c2/major_fragment_2/minor_fragment_3/operator_2/:
total 8
drwxr-xr-x 3 mapr mapr 4096 May 16 01:40 ..
drwxr-xr-x 2 mapr mapr 4096 May 16 01:41 .
2aa9600f-016a-5283-f98e-ef22942981c2/major_fragment_2/minor_fragment_2/operator_2/:
total 8
drwxr-xr-x 3 mapr mapr 4096 May 16 01:40 ..
drwxr-xr-x 2 mapr mapr 4096 May 16 01:41 .
2aa9600f-016a-5283-f98e-ef22942981c2/major_fragment_2/minor_fragment_1/operator_2/:
total 8
drwxr-xr-x 3 mapr mapr 4096 May 16 01:40 ..
drwxr-xr-x 2 mapr mapr 4096 May 16 01:41 .
2aa9600f-016a-5283-f98e-ef22942981c2/major_fragment_2/minor_fragment_0/operator_2/:
total 8
drwxr-xr-x 3 mapr mapr 4096 May 16 01:40 ..
drwxr-xr-x 2 mapr mapr 4096 May 16 01:41 .
[root@h1 spill]# pwd
/tmp/drill/spill
{code}
I would suggest if SQL finishes successfully, the whole directory for SQL
profile Id should be removed.
> Drill not deleting spooling files
> ---------------------------------
>
> Key: DRILL-2100
> URL: https://issues.apache.org/jira/browse/DRILL-2100
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
> Affects Versions: 0.8.0
> Reporter: Abhishek Girish
> Assignee: Steven Phillips
> Fix For: 1.1.0
>
>
> Currently, after forcing queries to use an external sort by switching off
> hash join/agg causes spill-to-disk files accumulating.
> This causes issues with disk space availability when the spill is configured
> to be on the local file system (/tmp/drill). Also not optimal when configured
> to use DFS (custom).
> Drill must clean up all temporary files created after a query completes or
> after a drillbit restart.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
