[ https://issues.apache.org/jira/browse/DRILL-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Challapalli updated DRILL-5528: ------------------------------------- Attachment: drillbit.out drillbit.log drill-env.sh 26e2183f-2c5d-4044-0e23-4aeb45c40be5.sys.drill > Sorting 19GB data with 14GB memory in a single fragment takes ~150 minutes > -------------------------------------------------------------------------- > > Key: DRILL-5528 > URL: https://issues.apache.org/jira/browse/DRILL-5528 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators > Affects Versions: 1.10.0 > Reporter: Rahul Challapalli > Assignee: Paul Rogers > Attachments: 26e2183f-2c5d-4044-0e23-4aeb45c40be5.sys.drill, > drillbit.log, drillbit.out, drill-env.sh > > > Configuration : > {code} > git.commit.id.abbrev=1e0a14c > DRILL_MAX_DIRECT_MEMORY="32G" > DRILL_MAX_HEAP="4G" > {code} > Based on the runtime of the below query, I suspect there is a performance > bottleneck somewhere > {code} > [root@qa-node190 external-sort]# /opt/drill/bin/sqlline -u > jdbc:drill:zk=10.10.100.190:5181 > apache drill 1.11.0-SNAPSHOT > "start your sql engine" > 0: jdbc:drill:zk=10.10.100.190:5181> ALTER SESSION SET > `exec.sort.disable_managed` = false; > +-------+-------------------------------------+ > | ok | summary | > +-------+-------------------------------------+ > | true | exec.sort.disable_managed updated. | > +-------+-------------------------------------+ > 1 row selected (0.975 seconds) > 0: jdbc:drill:zk=10.10.100.190:5181> alter session set > `planner.width.max_per_node` = 1; > +-------+--------------------------------------+ > | ok | summary | > +-------+--------------------------------------+ > | true | planner.width.max_per_node updated. | > +-------+--------------------------------------+ > 1 row selected (0.371 seconds) > 0: jdbc:drill:zk=10.10.100.190:5181> alter session set > `planner.disable_exchanges` = true; > +-------+-------------------------------------+ > | ok | summary | > +-------+-------------------------------------+ > | true | planner.disable_exchanges updated. | > +-------+-------------------------------------+ > 1 row selected (0.292 seconds) > 0: jdbc:drill:zk=10.10.100.190:5181> alter session set > `planner.memory.max_query_memory_per_node` = 14106127360; > +-------+----------------------------------------------------+ > | ok | summary | > +-------+----------------------------------------------------+ > | true | planner.memory.max_query_memory_per_node updated. | > +-------+----------------------------------------------------+ > 1 row selected (0.316 seconds) > 0: jdbc:drill:zk=10.10.100.190:5181> select count(*) from (select * from > dfs.`/drill/testdata/resource-manager/250wide.tbl` order by columns[0])d > where d.columns[0] = 'ljdfhwuehnoiueyf'; > +---------+ > | EXPR$0 | > +---------+ > | 0 | > +---------+ > 1 row selected (8530.719 seconds) > {code} > I attached the logs and profile files. The data is too large to attach to a > jira. Reach out to me if you need any more information -- This message was sent by Atlassian JIRA (v6.3.15#6346)