[
https://issues.apache.org/jira/browse/DRILL-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067605#comment-16067605
]
Paul Rogers edited comment on DRILL-5453 at 6/29/17 2:37 AM:
-------------------------------------------------------------
This bug is a tough one. There is certainly a bug, but it is not in the
external sort. It is, instead, either a user error, or an unfortunate
consequence of how the text reader works.
When the query runs, we get this error:
{code}
Records: 8096, Total size: 678,068,224, Data size: 389425696, Gross row
width: 83754, Net row width: 48101, Density: 1}
{code}
Not sure how this worked on the test machine with the memory assigned.
The memory per node is only 482,344,960. Even if the sort is given the entire
amount (which it is, because the query is single-fragment), one cannot fit 678
MB of data in a 482 MB container. Given that, the very next allocation (the
SV2) fails.
There is nothing the sort can do in this case. The solution is to modify the
scan operator to create much smaller record batches. That solution is coming in
the form of DRILL-5211. So, I've associated this ticket with that project.
was (Author: paul-rogers):
This bug is a tough one. There is certainly a bug, but it is not in the
external sort. It is, instead, either a user error, or an unfortunate
consequence of how the text reader works.
When the query runs, we get this error:
{code}
Records: 8096, Total size: 678,068,224, Data size: 389425696, Gross row
width: 83754, Net row width: 48101, Density: 1}
{code}
The memory per node is only 482,344,960. Even if the sort is given the entire
amount (which it is, because the query is single-fragment), one cannot fit 678
MB of data in a 482 MB container. Given that, the very next allocation (the
SV2) fails.
There is nothing the sort can do in this case. The solution is to modify the
scan operator to create much smaller record batches. That solution is coming in
the form of DRILL-5211. So, I've associated this ticket with that project.
> Managed External Sort : Sorting on a lot of columns is taking unreasonably
> long time
> ------------------------------------------------------------------------------------
>
> Key: DRILL-5453
> URL: https://issues.apache.org/jira/browse/DRILL-5453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
> Affects Versions: 1.10.0
> Reporter: Rahul Challapalli
> Assignee: Paul Rogers
> Attachments: drill5453.sys.drill
>
>
> The below query ran for ~16hrs before I cancelled it.
> {code}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.memory.max_query_memory_per_node` = 482344960;
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.width.max_per_query` = 1;
> select count(*) from (select * from
> dfs.`/drill/testdata/resource-manager/3500cols.tbl` order by
> columns[450],columns[330],columns[230],columns[220],columns[110],columns[90],columns[80],columns[70],columns[40],columns[10],columns[20],columns[30],columns[40],columns[50],
>
> columns[454],columns[413],columns[940],columns[834],columns[73],columns[140],columns[104],columns[2222],columns[30],columns[2420],columns[1520],
> columns[1410],
> columns[1110],columns[1290],columns[2380],columns[705],columns[45],columns[1054],columns[2430],columns[420],columns[404],columns[3350],
>
> columns[3333],columns[153],columns[356],columns[84],columns[745],columns[1450],columns[103],columns[2065],columns[343],columns[3420],columns[530],
> columns[3210] ) d where d.col433 = 'sjka skjf';
> alter session set `planner.memory.max_query_memory_per_node` = 2147483648;
> {code}
> The data set and the logs are too large to attach to a jira. But below is a
> description of the data
> {code}
> No of records : 1,000,000
> No of columns : 3500
> Length of each column : < 50
> {code}
> The profile is attached and I will give my analysis on why I think its an
> un-reasonable amount of time soon.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
