[jira] [Commented] (DRILL-4274) ExternalSort doesn't always handle low memory condition well, failing execution instead of spilling in some cases
[ https://issues.apache.org/jira/browse/DRILL-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940865#comment-15940865 ] Zelaine Fong commented on DRILL-4274: - [~Paul.Rogers], [~rkins] - do you guys know if this still is an issue with the new external sort? > ExternalSort doesn't always handle low memory condition well, failing > execution instead of spilling in some cases > - > > Key: DRILL-4274 > URL: https://issues.apache.org/jira/browse/DRILL-4274 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Reporter: Jacques Nadeau > Fix For: Future > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-4274) ExternalSort doesn't always handle low memory condition well, failing execution instead of spilling in some cases
[ https://issues.apache.org/jira/browse/DRILL-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940932#comment-15940932 ] Paul Rogers commented on DRILL-4274: [~zfong], this is a very nice summary of what the external sort work is supposed to fix. Thanks for finding this one. I'll review it to see if it can be resolved as a result of our recent work. > ExternalSort doesn't always handle low memory condition well, failing > execution instead of spilling in some cases > - > > Key: DRILL-4274 > URL: https://issues.apache.org/jira/browse/DRILL-4274 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Reporter: Jacques Nadeau > Fix For: Future > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-4274) ExternalSort doesn't always handle low memory condition well, failing execution instead of spilling in some cases
[ https://issues.apache.org/jira/browse/DRILL-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15101246#comment-15101246 ] Jacques Nadeau commented on DRILL-4274: --- [~vicky], can you share any stack traces that you are seeing related to this? > ExternalSort doesn't always handle low memory condition well, failing > execution instead of spilling in some cases > - > > Key: DRILL-4274 > URL: https://issues.apache.org/jira/browse/DRILL-4274 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Reporter: Jacques Nadeau >Assignee: Jacques Nadeau > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4274) ExternalSort doesn't always handle low memory condition well, failing execution instead of spilling in some cases
[
https://issues.apache.org/jira/browse/DRILL-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102123#comment-15102123
]
Deneche A. Hakim commented on DRILL-4274:
-
Many functional tests that use a "wide table" fail with the following error
when run on 32 core machines (or using planner.width.max_per_node=23):
{noformat}
/root/drillAutomation/framework-master/framework/resources/Functional/data-shapes/wide-columns/5000/1000rows/parquet/q163_DRILL-2046.q
Query:
select ws1.* from widestrings ws1 INNER JOIN widestrings ws2 on
ws1.str_var=ws2.str_var_null_empty
Failed with exception
java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory
while executing the query.
Unable to allocate sv2 for 1000 records, and not enough batchGroups to spill.
batchGroups.size 0
spilledBatchGroups.size 0
allocated memory 52520576
allocator limit 46684427
Fragment 0:0
{noformat}
In this case the sort exceeds it's limit on the first data batch, as you can
see from the message context, the batch wasn't added to batchGroups yet.
> ExternalSort doesn't always handle low memory condition well, failing
> execution instead of spilling in some cases
> -
>
> Key: DRILL-4274
> URL: https://issues.apache.org/jira/browse/DRILL-4274
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Jacques Nadeau
>Assignee: Jacques Nadeau
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4274) ExternalSort doesn't always handle low memory condition well, failing execution instead of spilling in some cases
[
https://issues.apache.org/jira/browse/DRILL-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102133#comment-15102133
]
Deneche A. Hakim commented on DRILL-4274:
-
Another error we saw in the same "wide table" tests:
{noformat}
/root/drillAutomation/framework-master/framework/resources/Functional/data-shapes/wide-columns/5000/1000rows/parquet/q168_DRILL-2046.q
Query:
select ws1.* from widestrings ws1 INNER JOIN (select str_var_null_empty from
widestrings where str_var_null_empty is not null and length(str_var_null_empty)
<> 0 )ws2 on ws1.str_null=ws2.str_var_null_empty where ws1.str_null is not null
and length(ws1.str_null) <> 0
Failed with exception
java.sql.SQLException: SYSTEM ERROR: DrillRuntimeException: Failed to
pre-allocate memory for SV. Existing recordCount*4 = 0, incoming batch
recordCount*4 = 676
{noformat}
I debugged this one, and if I remember correctly, the sort exceeds it's limit
on the first batch, tries to spill but doesn't (because mergeAndSpill()
requires at least 2 batches to spill). Later, it finds out it didn't spill and
assumes it's because it can merge the batches in memory but fails with this
exception.
> ExternalSort doesn't always handle low memory condition well, failing
> execution instead of spilling in some cases
> -
>
> Key: DRILL-4274
> URL: https://issues.apache.org/jira/browse/DRILL-4274
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Jacques Nadeau
>Assignee: Jacques Nadeau
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4274) ExternalSort doesn't always handle low memory condition well, failing execution instead of spilling in some cases
[
https://issues.apache.org/jira/browse/DRILL-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111993#comment-15111993
]
Khurram Faraaz commented on DRILL-4274:
---
This is also seen on Drill 1.5.0 (commit ID : 2f0e3f27) with JDK8
{noformat}
framework/framework/resources/Functional/tpcds/original/json/query56.sql
(connection: 1117749405)
[#2158] Query failed:
oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
DrillRuntimeException: Failed to pre-allocate memory for SV. Existing
recordCount*4 = 0, incoming batch recordCount*4 = 3348
Fragment 0:0
[Error Id: f50dcc75-0f83-401b-8249-599c6a1d0d8c on drill-democ1:31010]
at
oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:119)
at
oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:113)
at
oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
at
oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:67)
at
oadd.org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:374)
at
oadd.org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:89)
at
oadd.org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:252)
at
oadd.org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:123)
at
oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:285)
at
oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:257)
at
oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at
oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at
oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at
oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at
oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at
oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
at
oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at
oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at
oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at
oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at
oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> ExternalSort doesn't always handle low memory condition well, failing
> execution instead of spilling in some cases
> -
>
> Key: DRILL-4274
> URL: https://issues.apache.org/jira/browse/DRILL-4274
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Jacques Nadeau
>Assignee: Jacques Nadeau
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4274) ExternalSort doesn't always handle low memory condition well, failing execution instead of spilling in some cases
[ https://issues.apache.org/jira/browse/DRILL-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112943#comment-15112943 ] Victoria Markman commented on DRILL-4274: - Can we please look at the cause of this sooner than later and if we can't fix it maybe propose a workaround for 1.5.0 ? > ExternalSort doesn't always handle low memory condition well, failing > execution instead of spilling in some cases > - > > Key: DRILL-4274 > URL: https://issues.apache.org/jira/browse/DRILL-4274 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Reporter: Jacques Nadeau >Assignee: Jacques Nadeau >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4274) ExternalSort doesn't always handle low memory condition well, failing execution instead of spilling in some cases
[ https://issues.apache.org/jira/browse/DRILL-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112953#comment-15112953 ] Jacques Nadeau commented on DRILL-4274: --- This will be a substantial undertaking. I don't see it as a blocker since it has existed in all the previous Drill releases. It definitely should be addressed but I want to make sure that I'm managing expectations. > ExternalSort doesn't always handle low memory condition well, failing > execution instead of spilling in some cases > - > > Key: DRILL-4274 > URL: https://issues.apache.org/jira/browse/DRILL-4274 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Reporter: Jacques Nadeau >Assignee: Jacques Nadeau >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4274) ExternalSort doesn't always handle low memory condition well, failing execution instead of spilling in some cases
[
https://issues.apache.org/jira/browse/DRILL-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129066#comment-15129066
]
Victoria Markman commented on DRILL-4274:
-
These 4 tests have been disabled temporarily (commitId:
01a4a303770d7ffa564e0c1f0497049a4721bd9c), need to move them back when bug is
fixed.
{code}
/root/drillAutomation/framework/framework/resources/Advanced/data-shapes/wide-columns/5000/10rows/parquet/q174.q
/root/drillAutomation/framework/framework/resources/Advanced/data-shapes/wide-columns/5000/10rows/parquet/q209.q
/root/drillAutomation/framework/framework/resources/Advanced/data-shapes/wide-columns/5000/10rows/parquet/q207.q
/root/drillAutomation/framework/framework/resources/Advanced/data-shapes/wide-columns/5000/10rows/parquet/q213.q
{code}
> ExternalSort doesn't always handle low memory condition well, failing
> execution instead of spilling in some cases
> -
>
> Key: DRILL-4274
> URL: https://issues.apache.org/jira/browse/DRILL-4274
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Jacques Nadeau
>Assignee: Jacques Nadeau
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
