[jira] [Created] (DRILL-6828) Hit UnrecognizedPropertyException when run tpch queries
Dechang Gu created DRILL-6828:
-
Summary: Hit UnrecognizedPropertyException when run tpch queries
Key: DRILL-6828
URL: https://issues.apache.org/jira/browse/DRILL-6828
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 1.15.0
Environment: RHEL 7, Apache Drill commit id:
18e09a1b1c801f2691a05ae7db543bf71874cfea
Reporter: Dechang Gu
Assignee: Boaz Ben-Zvi
Fix For: 1.15.0
Installed Apache Drill 1.15.0 commit id:
18e09a1b1c801f2691a05ae7db543bf71874cfea DRILL-6763: Codegen optimization of
SQL functions with constant values(\#1481)
Hit the following errors:
{code}
java.sql.SQLException: SYSTEM ERROR: UnrecognizedPropertyException:
Unrecognized field "outgoingBatchSize" (class
org.apache.drill.exec.physical.config.HashPartitionSender), not marked as
ignorable (9 known properties: "receiver-major-fragment", "initialAllocation",
"expr", "userName", "@id", "child", "cost", "destinations", "maxAllocation"])
at [Source: (StringReader); line: 1000, column: 29] (through reference chain:
org.apache.drill.exec.physical.config.HashPartitionSender["outgoingBatchSize"])
Fragment 3:175
Please, refer to logs for more information.
[Error Id: cc023cdb-9a46-4edd-ad0b-6da1e9085291 on ucs-node6.perf.lab:31010]
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:528)
at
org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:600)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1288)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
at
org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
at
org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1109)
at
org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1120)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:196)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
at
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
at PipSQueak.executeQuery(PipSQueak.java:289)
at PipSQueak.runTest(PipSQueak.java:104)
at PipSQueak.main(PipSQueak.java:477)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
ERROR: UnrecognizedPropertyException: Unrecognized field "outgoingBatchSize"
(class org.apache.drill.exec.physical.config.HashPartitionSender), not marked
as ignorable (9 known properties: "receiver-major-fragment",
"initialAllocation", "expr", "userName", "@id", "child", "cost",
"destinations", "maxAllocation"])
at [Source: (StringReader); line: 1000, column: 29] (through reference chain:
org.apache.drill.exec.physical.config.HashPartitionSender["outgoingBatchSize"])
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DRILL-3926) TPCH Concurrency Scale tests hit ChannelClosedException
[ https://issues.apache.org/jira/browse/DRILL-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu closed DRILL-3926. - It's no longer reproducible. So close it. > TPCH Concurrency Scale tests hit ChannelClosedException > --- > > Key: DRILL-3926 > URL: https://issues.apache.org/jira/browse/DRILL-3926 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill > Environment: ucs-node 1 - node 11 (10+1 node cluster), RHEL 6.4 Linux > 2.6.32-358.el6.x86_64, MapR 4.0.2.29870.GA, MapR Drill 1.2 gitID eafe0a2 >Reporter: Dechang Gu >Assignee: Dechang Gu >Priority: Major > Fix For: 1.15.0 > > > In TPCH Concurrency tests, we try to see how drill scales up with number of > threads with each threads running a simple query (tpch query #6). With 96 > threads, many threads terminated due to ChannelClosedException and/or > FormanException: > 2015-10-07 18:01:26 [pip87] ERROR PipSQuawkling executeQuery - [ 0 / > 06_par100 ] SYSTEM ERROR: ChannelClosedException > [Error Id: cbae3879-8067-47cd-8c42-91a38896b81a on ucs-node9.perf.lab:31010] > java.sql.SQLException: SYSTEM ERROR: ChannelClosedException > [Error Id: cbae3879-8067-47cd-8c42-91a38896b81a on ucs-node9.perf.lab:31010] > at > org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247) > at > org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:290) > at > org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1359) > at > org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:74) > at > net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404) > at > net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351) > at > net.hydromatic.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:78) > at > org.apache.drill.jdbc.impl.DrillStatementImpl.executeQuery(DrillStatementImpl.java:97) > at PipSQuawkling.executeQuery(PipSQuawkling.java:295) > at PipSQuawkling.executeTest(PipSQuawkling.java:148) > at PipSQuawkling.run(PipSQuawkling.java:76) > Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM > ERROR: ChannelClosedException > [Error Id: cbae3879-8067-47cd-8c42-91a38896b81a on ucs-node9.perf.lab:31010] > at > org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:118) > at > org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:110) > at > org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:47) > at > org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:32) > at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:61) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:233) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:205) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at >
[jira] [Resolved] (DRILL-3926) TPCH Concurrency Scale tests hit ChannelClosedException
[ https://issues.apache.org/jira/browse/DRILL-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu resolved DRILL-3926. --- Resolution: Fixed > TPCH Concurrency Scale tests hit ChannelClosedException > --- > > Key: DRILL-3926 > URL: https://issues.apache.org/jira/browse/DRILL-3926 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill > Environment: ucs-node 1 - node 11 (10+1 node cluster), RHEL 6.4 Linux > 2.6.32-358.el6.x86_64, MapR 4.0.2.29870.GA, MapR Drill 1.2 gitID eafe0a2 >Reporter: Dechang Gu >Assignee: Dechang Gu >Priority: Major > Fix For: 1.15.0 > > > In TPCH Concurrency tests, we try to see how drill scales up with number of > threads with each threads running a simple query (tpch query #6). With 96 > threads, many threads terminated due to ChannelClosedException and/or > FormanException: > 2015-10-07 18:01:26 [pip87] ERROR PipSQuawkling executeQuery - [ 0 / > 06_par100 ] SYSTEM ERROR: ChannelClosedException > [Error Id: cbae3879-8067-47cd-8c42-91a38896b81a on ucs-node9.perf.lab:31010] > java.sql.SQLException: SYSTEM ERROR: ChannelClosedException > [Error Id: cbae3879-8067-47cd-8c42-91a38896b81a on ucs-node9.perf.lab:31010] > at > org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247) > at > org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:290) > at > org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1359) > at > org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:74) > at > net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404) > at > net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351) > at > net.hydromatic.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:78) > at > org.apache.drill.jdbc.impl.DrillStatementImpl.executeQuery(DrillStatementImpl.java:97) > at PipSQuawkling.executeQuery(PipSQuawkling.java:295) > at PipSQuawkling.executeTest(PipSQuawkling.java:148) > at PipSQuawkling.run(PipSQuawkling.java:76) > Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM > ERROR: ChannelClosedException > [Error Id: cbae3879-8067-47cd-8c42-91a38896b81a on ucs-node9.perf.lab:31010] > at > org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:118) > at > org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:110) > at > org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:47) > at > org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:32) > at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:61) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:233) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:205) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at
[jira] [Commented] (DRILL-3926) TPCH Concurrency Scale tests hit ChannelClosedException
[ https://issues.apache.org/jira/browse/DRILL-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618114#comment-16618114 ] Dechang Gu commented on DRILL-3926: --- It's no longer repro'ed in my stability tests. Hence close it. > TPCH Concurrency Scale tests hit ChannelClosedException > --- > > Key: DRILL-3926 > URL: https://issues.apache.org/jira/browse/DRILL-3926 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill > Environment: ucs-node 1 - node 11 (10+1 node cluster), RHEL 6.4 Linux > 2.6.32-358.el6.x86_64, MapR 4.0.2.29870.GA, MapR Drill 1.2 gitID eafe0a2 >Reporter: Dechang Gu >Assignee: Dechang Gu >Priority: Major > Fix For: 1.15.0 > > > In TPCH Concurrency tests, we try to see how drill scales up with number of > threads with each threads running a simple query (tpch query #6). With 96 > threads, many threads terminated due to ChannelClosedException and/or > FormanException: > 2015-10-07 18:01:26 [pip87] ERROR PipSQuawkling executeQuery - [ 0 / > 06_par100 ] SYSTEM ERROR: ChannelClosedException > [Error Id: cbae3879-8067-47cd-8c42-91a38896b81a on ucs-node9.perf.lab:31010] > java.sql.SQLException: SYSTEM ERROR: ChannelClosedException > [Error Id: cbae3879-8067-47cd-8c42-91a38896b81a on ucs-node9.perf.lab:31010] > at > org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247) > at > org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:290) > at > org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1359) > at > org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:74) > at > net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404) > at > net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351) > at > net.hydromatic.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:78) > at > org.apache.drill.jdbc.impl.DrillStatementImpl.executeQuery(DrillStatementImpl.java:97) > at PipSQuawkling.executeQuery(PipSQuawkling.java:295) > at PipSQuawkling.executeTest(PipSQuawkling.java:148) > at PipSQuawkling.run(PipSQuawkling.java:76) > Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM > ERROR: ChannelClosedException > [Error Id: cbae3879-8067-47cd-8c42-91a38896b81a on ucs-node9.perf.lab:31010] > at > org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:118) > at > org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:110) > at > org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:47) > at > org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:32) > at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:61) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:233) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:205) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at >
[jira] [Commented] (DRILL-6718) TPCH Queries hit IOB when running with the JPPD feature
[
https://issues.apache.org/jira/browse/DRILL-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617786#comment-16617786
]
Dechang Gu commented on DRILL-6718:
---
[~weijie] Here is the stack:
{code}
2018-08-27 17:27:14,916 [247b6925-b3f6-381f-12d2-7971abf317b7:frag:5:35] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IndexOutOfBoundsException:
DrillBuf[1], udle: [1 0..0], index: 0, length: 1 (expected: range(0, 0))
Fragment 5:35
[Error Id: a3de08b2-6453-4354-8d58-8a79d4c62469 on perf109-68.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IndexOutOfBoundsException: DrillBuf[1], udle: [1 0..0], index: 0, length: 1
(expected: range(0, 0))
Fragment 5:35
[Error Id: a3de08b2-6453-4354-8d58-8a79d4c62469 on perf109-68.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:360)
[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:215)
[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:326)
[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_171]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: java.lang.IndexOutOfBoundsException: DrillBuf[1], udle: [1 0..0],
index: 0, length: 1 (expected: range(0, 0))
at
org.apache.drill.exec.memory.BoundsChecking.checkIndex(BoundsChecking.java:80)
~[drill-memory-base-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.memory.BoundsChecking.lengthCheck(BoundsChecking.java:86)
~[drill-memory-base-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at io.netty.buffer.DrillBuf.chk(DrillBuf.java:119)
~[drill-memory-base-1.15.0-SNAPSHOT.jar:4.0.48.Final]
at io.netty.buffer.DrillBuf.getByte(DrillBuf.java:757)
~[drill-memory-base-1.15.0-SNAPSHOT.jar:4.0.48.Final]
at
org.apache.drill.exec.vector.UInt1Vector$Accessor.get(UInt1Vector.java:424)
~[vector-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.vector.NullableBigIntVector$Accessor.isSet(NullableBigIntVector.java:446)
~[vector-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.Hash64Gen2282971.hash64Code(Hash64Gen2282971.java:62)
~[na:na]
at
org.apache.drill.exec.physical.impl.filter.RuntimeFilterRecordBatch.computeBitSet(RuntimeFilterRecordBatch.java:241)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.filter.RuntimeFilterRecordBatch.applyRuntimeFilter(RuntimeFilterRecordBatch.java:219)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.filter.RuntimeFilterRecordBatch.doWork(RuntimeFilterRecordBatch.java:92)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:117)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
[jira] [Commented] (DRILL-6718) TPCH Queries hit IOB when running with the JPPD feature
[
https://issues.apache.org/jira/browse/DRILL-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596528#comment-16596528
]
Dechang Gu commented on DRILL-6718:
---
[~weijie] Here is a query that was failed, complete profile is also uploaded
(TPCH_03_1_id_247b6925-b3f6-381f-12d2-7971abf317b7.jsonQuery):
{code}
select l.l_orderkey, sum(l.l_extendedprice * (1 - l.l_discount)) as revenue,
o.o_orderdate, o.o_shippriority from customer c, orders o, lineitem l where
c.c_mktsegment = 'HOUSEHOLD' and c.c_custkey = o.o_custkey and l.l_orderkey =
o.o_orderkey and o.o_orderdate < date '1995-03-25' and l.l_shipdate > date
'1995-03-25' group by l.l_orderkey, o.o_orderdate, o.o_shippriority order by
revenue desc, o.o_orderdate limit 10 {code}
and plan
{code}
00-00Screen : rowType = RecordType(ANY l_orderkey, ANY revenue, ANY
o_orderdate, ANY o_shippriority): rowcount = 10.0, cumulative cost =
{4.266969862594E9 rows, 3.744022097539064E10 cpu, 0.0 io,
7.926132591820801E12 network, 1.19202670752E10 memory}, id = 18236
00-01 Project(l_orderkey=[$0], revenue=[$1], o_orderdate=[$2],
o_shippriority=[$3]) : rowType = RecordType(ANY l_orderkey, ANY revenue, ANY
o_orderdate, ANY o_shippriority): rowcount = 10.0, cumulative cost =
{4.266969861594E9 rows, 3.744022097439064E10 cpu, 0.0 io,
7.926132591820801E12 network, 1.19202670752E10 memory}, id = 18235
00-02SelectionVectorRemover : rowType = RecordType(ANY l_orderkey, ANY
revenue, ANY o_orderdate, ANY o_shippriority): rowcount = 10.0, cumulative cost
= {4.266969851594E9 rows, 3.744022093439064E10 cpu, 0.0 io,
7.926132591820801E12 network, 1.19202670752E10 memory}, id = 18234
00-03 Limit(fetch=[10]) : rowType = RecordType(ANY l_orderkey, ANY
revenue, ANY o_orderdate, ANY o_shippriority): rowcount = 10.0, cumulative cost
= {4.266969841594E9 rows, 3.744022092439064E10 cpu, 0.0 io,
7.926132591820801E12 network, 1.19202670752E10 memory}, id = 18233
00-04SingleMergeExchange(sort0=[1 DESC], sort1=[2]) : rowType =
RecordType(ANY l_orderkey, ANY revenue, ANY o_orderdate, ANY o_shippriority):
rowcount = 3.00018951E7, cumulative cost = {4.266969831594E9 rows,
3.744022088439064E10 cpu, 0.0 io, 7.926132591820801E12 network,
1.19202670752E10 memory}, id = 18232
01-01 OrderedMuxExchange(sort0=[1 DESC], sort1=[2]) : rowType =
RecordType(ANY l_orderkey, ANY revenue, ANY o_orderdate, ANY o_shippriority):
rowcount = 3.00018951E7, cumulative cost = {4.236967936495E9 rows,
3.680154917066042E10 cpu, 0.0 io, 7.4345815425024E12 network, 1.19202670752E10
memory}, id = 18231
02-01SelectionVectorRemover : rowType = RecordType(ANY
l_orderkey, ANY revenue, ANY o_orderdate, ANY o_shippriority): rowcount =
3.00018951E7, cumulative cost = {4.206966041396E9 rows,
3.6771547275560425E10 cpu, 0.0 io, 7.4345815425024E12 network, 1.19202670752E10
memory}, id = 18230
02-02 TopN(limit=[10]) : rowType = RecordType(ANY l_orderkey,
ANY revenue, ANY o_orderdate, ANY o_shippriority): rowcount = 3.00018951E7,
cumulative cost = {4.176964146297E9 rows, 3.674154538046043E10 cpu, 0.0 io,
7.4345815425024E12 network, 1.19202670752E10 memory}, id = 18229
02-03HashToRandomExchange(dist0=[[$1]], dist1=[[$2]]) :
rowType = RecordType(ANY l_orderkey, ANY revenue, ANY o_orderdate, ANY
o_shippriority): rowcount = 3.00018951E7, cumulative cost = {4.1469622512E9
rows, 3.59442322746E10 cpu, 0.0 io, 7.4345815425024E12 network,
1.19202670752E10 memory}, id = 18228
03-01 Project(l_orderkey=[$0], revenue=[$3],
o_orderdate=[$1], o_shippriority=[$2]) : rowType = RecordType(ANY l_orderkey,
ANY revenue, ANY o_orderdate, ANY o_shippriority): rowcount = 3.00018951E7,
cumulative cost = {4.1169603561E9 rows, 3.55842095334E10 cpu, 0.0 io,
6.943030493184E12 network, 1.19202670752E10 memory}, id = 18227
03-02HashAgg(group=[{0, 1, 2}], revenue=[SUM($3)]) :
rowType = RecordType(ANY l_orderkey, ANY o_orderdate, ANY o_shippriority, ANY
revenue): rowcount = 3.00018951E7, cumulative cost = {4.086958461E9 rows,
3.5464201953E10 cpu, 0.0 io, 6.943030493184E12 network, 1.19202670752E10
memory}, id = 18226
03-03 Project(l_orderkey=[$6], o_orderdate=[$4],
o_shippriority=[$5], $f3=[*($8, -(1, $9))]) : rowType = RecordType(ANY
l_orderkey, ANY o_orderdate, ANY o_shippriority, ANY $f3): rowcount =
3.00018951E8, cumulative cost = {3.78693951E9 rows, 2.4663519717E10 cpu, 0.0
io, 6.943030493184E12 network, 1.3596E9 memory}, id = 18225
03-04Project(c_mktsegment=[$8], c_custkey=[$9],
o_custkey=[$4], o_orderkey=[$5], o_orderdate=[$6], o_shippriority=[$7],
l_orderkey=[$0], l_shipdate=[$1], l_extendedprice=[$2], l_discount=[$3]) :
rowType = RecordType(ANY c_mktsegment, ANY c_custkey, ANY o_custkey, ANY
o_orderkey, ANY
[jira] [Updated] (DRILL-6718) TPCH Queries hit IOB when running with the JPPD feature
[
https://issues.apache.org/jira/browse/DRILL-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu updated DRILL-6718:
--
Attachment: TPCH_03_1_id_247b6925-b3f6-381f-12d2-7971abf317b7.json
> TPCH Queries hit IOB when running with the JPPD feature
> ---
>
> Key: DRILL-6718
> URL: https://issues.apache.org/jira/browse/DRILL-6718
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.15.0
> Environment: RHEL 7
>Reporter: Dechang Gu
>Assignee: weijie.tong
>Priority: Major
> Attachments: TPCH_03_1_id_247b6925-b3f6-381f-12d2-7971abf317b7.json
>
>
> Running TPCH SF100 queries on drill master commit
> b895b28182a981e5948ffa292da827cb8b2e571e (DRILL-6385: Support JPPD feature).
> 9 queries hit IOB error, the stacks are similar, as follows
> {code}
> 2018-08-27 17:27:14,916 [247b6925-b3f6-381f-12d2-7971abf317b7:frag:5:35]
> ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
> IndexOutOfBoundsException: DrillBuf[1], udle: [1 0..0], index: 0, length: 1
> (expected: range(0, 0))
> Fragment 5:35
> [Error Id: a3de08b2-6453-4354-8d58-8a79d4c62469 on perf109-68.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IndexOutOfBoundsException: DrillBuf[1], udle: [1 0..0], index: 0, length: 1
> (expected: range(0, 0))
> Fragment 5:35
> [Error Id: a3de08b2-6453-4354-8d58-8a79d4c62469 on perf109-68.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:360)
> [drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:215)
> [drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:326)
> [drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_171]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_171]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
> Caused by: java.lang.IndexOutOfBoundsException: DrillBuf[1], udle: [1 0..0],
> index: 0, length: 1 (expected: range(0, 0))
> at
> org.apache.drill.exec.memory.BoundsChecking.checkIndex(BoundsChecking.java:80)
> ~[drill-memory-base-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.memory.BoundsChecking.lengthCheck(BoundsChecking.java:86)
> ~[drill-memory-base-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at io.netty.buffer.DrillBuf.chk(DrillBuf.java:119)
> ~[drill-memory-base-1.15.0-SNAPSHOT.jar:4.0.48.Final]
> at io.netty.buffer.DrillBuf.getByte(DrillBuf.java:757)
> ~[drill-memory-base-1.15.0-SNAPSHOT.jar:4.0.48.Final]
> at
> org.apache.drill.exec.vector.UInt1Vector$Accessor.get(UInt1Vector.java:424)
> ~[vector-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableBigIntVector$Accessor.isSet(NullableBigIntVector.java:446)
> ~[vector-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.Hash64Gen2282971.hash64Code(Hash64Gen2282971.java:62)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.filter.RuntimeFilterRecordBatch.computeBitSet(RuntimeFilterRecordBatch.java:241)
> ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.filter.RuntimeFilterRecordBatch.applyRuntimeFilter(RuntimeFilterRecordBatch.java:219)
> ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.filter.RuntimeFilterRecordBatch.doWork(RuntimeFilterRecordBatch.java:92)
> ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:117)
> ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
> ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
>
[jira] [Created] (DRILL-6718) TPCH Queries hit IOB when running with the JPPD feature
Dechang Gu created DRILL-6718:
-
Summary: TPCH Queries hit IOB when running with the JPPD feature
Key: DRILL-6718
URL: https://issues.apache.org/jira/browse/DRILL-6718
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 1.15.0
Environment: RHEL 7
Reporter: Dechang Gu
Running TPCH SF100 queries on drill master commit
b895b28182a981e5948ffa292da827cb8b2e571e (DRILL-6385: Support JPPD feature).
9 queries hit IOB error, the stacks are similar, as follows
{code}
2018-08-27 17:27:14,916 [247b6925-b3f6-381f-12d2-7971abf317b7:frag:5:35] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IndexOutOfBoundsException:
DrillBuf[1], udle: [1 0..0], index: 0, length: 1 (expected: range(0, 0))
Fragment 5:35
[Error Id: a3de08b2-6453-4354-8d58-8a79d4c62469 on perf109-68.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IndexOutOfBoundsException: DrillBuf[1], udle: [1 0..0], index: 0, length: 1
(expected: range(0, 0))
Fragment 5:35
[Error Id: a3de08b2-6453-4354-8d58-8a79d4c62469 on perf109-68.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:360)
[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:215)
[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:326)
[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_171]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: java.lang.IndexOutOfBoundsException: DrillBuf[1], udle: [1 0..0],
index: 0, length: 1 (expected: range(0, 0))
at
org.apache.drill.exec.memory.BoundsChecking.checkIndex(BoundsChecking.java:80)
~[drill-memory-base-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.memory.BoundsChecking.lengthCheck(BoundsChecking.java:86)
~[drill-memory-base-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at io.netty.buffer.DrillBuf.chk(DrillBuf.java:119)
~[drill-memory-base-1.15.0-SNAPSHOT.jar:4.0.48.Final]
at io.netty.buffer.DrillBuf.getByte(DrillBuf.java:757)
~[drill-memory-base-1.15.0-SNAPSHOT.jar:4.0.48.Final]
at
org.apache.drill.exec.vector.UInt1Vector$Accessor.get(UInt1Vector.java:424)
~[vector-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.vector.NullableBigIntVector$Accessor.isSet(NullableBigIntVector.java:446)
~[vector-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.Hash64Gen2282971.hash64Code(Hash64Gen2282971.java:62)
~[na:na]
at
org.apache.drill.exec.physical.impl.filter.RuntimeFilterRecordBatch.computeBitSet(RuntimeFilterRecordBatch.java:241)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.filter.RuntimeFilterRecordBatch.applyRuntimeFilter(RuntimeFilterRecordBatch.java:219)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.filter.RuntimeFilterRecordBatch.doWork(RuntimeFilterRecordBatch.java:92)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:117)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
[jira] [Closed] (DRILL-6622) UNION on tpcds sf100 tables hit SYSTEM ERROR: NullPointerException
[
https://issues.apache.org/jira/browse/DRILL-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu closed DRILL-6622.
-
Verified that the issue is fixed in commit c58735a.
> UNION on tpcds sf100 tables hit SYSTEM ERROR: NullPointerException
> ---
>
> Key: DRILL-6622
> URL: https://issues.apache.org/jira/browse/DRILL-6622
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: salim achouche
>Priority: Blocker
> Labels: ready-to-commit
> Fix For: 1.14.0
>
> Attachments:
> MD4208_id_05_1_id_24b2a6f9-ed66-b97e-594d-f116cd3fdd23.json,
> MD4208_id_05_3_id_24b2ad9c-4568-a476-bbf6-2e17441078b1.json
>
>
> {code}
> SELECT c_customer_id FROM customer
> UNION
> SELECT ca_address_id FROM customer_address
> UNION
> SELECT cd_credit_rating FROM customer_demographics
> UNION
> SELECT hd_buy_potential FROM household_demographics
> UNION
> SELECT i_item_id FROM item
> UNION
> SELECT p_promo_id FROM promotion
> UNION
> SELECT t_time_id FROM time_dim
> UNION
> SELECT d_date_id FROM date_dim
> UNION
> SELECT s_store_id FROM store
> UNION
> SELECT w_warehouse_id FROM warehouse
> UNION
> SELECT sm_ship_mode_id FROM ship_mode
> UNION
> SELECT r_reason_id FROM reason
> UNION
> SELECT cc_call_center_id FROM call_center
> UNION
> SELECT web_site_id FROM web_site
> UNION
> SELECT wp_web_page_id FROM web_page
> UNION
> SELECT cp_catalog_page_id FROM catalog_page;
> {code}
> hit the following error:
> {code}
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:96)
> ~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.HashTableGen3$BatchHolder.isKeyMatchInternalBuild(BatchHolder.java:171)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.isKeyMatch(HashTableTemplate.java:218)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.access$1000(HashTableTemplate.java:120)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.common.HashTableTemplate.put(HashTableTemplate.java:650)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.HashAggregatorGen0.checkGroupAndAggrValues(HashAggTemplate.java:1372)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.HashAggregatorGen0.doWork(HashAggTemplate.java:599)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:268)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.union.UnionAllRecordBatch$UnionInputIterator.next(UnionAllRecordBatch.java:381)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> {code}
> [~dechanggu] found that the issue is absent in Drill 1.13.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6622) UNION on tpcds sf100 tables hit SYSTEM ERROR: NullPointerException
[
https://issues.apache.org/jira/browse/DRILL-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549920#comment-16549920
]
Dechang Gu commented on DRILL-6622:
---
profiles are uploaded:
failed one for 1.14.0:
https://issues.apache.org/jira/secure/attachment/12932320/MD4208_id_05_1_id_24b2a6f9-ed66-b97e-594d-f116cd3fdd23.json
good one for 1.13.0:
https://issues.apache.org/jira/secure/attachment/12932321/MD4208_id_05_3_id_24b2ad9c-4568-a476-bbf6-2e17441078b1.json
> UNION on tpcds sf100 tables hit SYSTEM ERROR: NullPointerException
> ---
>
> Key: DRILL-6622
> URL: https://issues.apache.org/jira/browse/DRILL-6622
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.14.0
>
> Attachments:
> MD4208_id_05_1_id_24b2a6f9-ed66-b97e-594d-f116cd3fdd23.json,
> MD4208_id_05_3_id_24b2ad9c-4568-a476-bbf6-2e17441078b1.json
>
>
> {code}
> SELECT c_customer_id FROM customer
> UNION
> SELECT ca_address_id FROM customer_address
> UNION
> SELECT cd_credit_rating FROM customer_demographics
> UNION
> SELECT hd_buy_potential FROM household_demographics
> UNION
> SELECT i_item_id FROM item
> UNION
> SELECT p_promo_id FROM promotion
> UNION
> SELECT t_time_id FROM time_dim
> UNION
> SELECT d_date_id FROM date_dim
> UNION
> SELECT s_store_id FROM store
> UNION
> SELECT w_warehouse_id FROM warehouse
> UNION
> SELECT sm_ship_mode_id FROM ship_mode
> UNION
> SELECT r_reason_id FROM reason
> UNION
> SELECT cc_call_center_id FROM call_center
> UNION
> SELECT web_site_id FROM web_site
> UNION
> SELECT wp_web_page_id FROM web_page
> UNION
> SELECT cp_catalog_page_id FROM catalog_page;
> {code}
> hit the following error:
> {code}
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:96)
> ~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.HashTableGen3$BatchHolder.isKeyMatchInternalBuild(BatchHolder.java:171)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.isKeyMatch(HashTableTemplate.java:218)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.access$1000(HashTableTemplate.java:120)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.common.HashTableTemplate.put(HashTableTemplate.java:650)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.HashAggregatorGen0.checkGroupAndAggrValues(HashAggTemplate.java:1372)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.HashAggregatorGen0.doWork(HashAggTemplate.java:599)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:268)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.union.UnionAllRecordBatch$UnionInputIterator.next(UnionAllRecordBatch.java:381)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> {code}
> [~dechanggu] found that the issue is absent in Drill 1.13.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6622) UNION on tpcds sf100 tables hit SYSTEM ERROR: NullPointerException
[
https://issues.apache.org/jira/browse/DRILL-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu updated DRILL-6622:
--
Attachment: MD4208_id_05_3_id_24b2ad9c-4568-a476-bbf6-2e17441078b1.json
> UNION on tpcds sf100 tables hit SYSTEM ERROR: NullPointerException
> ---
>
> Key: DRILL-6622
> URL: https://issues.apache.org/jira/browse/DRILL-6622
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.14.0
>
> Attachments:
> MD4208_id_05_1_id_24b2a6f9-ed66-b97e-594d-f116cd3fdd23.json,
> MD4208_id_05_3_id_24b2ad9c-4568-a476-bbf6-2e17441078b1.json
>
>
> {code}
> SELECT c_customer_id FROM customer
> UNION
> SELECT ca_address_id FROM customer_address
> UNION
> SELECT cd_credit_rating FROM customer_demographics
> UNION
> SELECT hd_buy_potential FROM household_demographics
> UNION
> SELECT i_item_id FROM item
> UNION
> SELECT p_promo_id FROM promotion
> UNION
> SELECT t_time_id FROM time_dim
> UNION
> SELECT d_date_id FROM date_dim
> UNION
> SELECT s_store_id FROM store
> UNION
> SELECT w_warehouse_id FROM warehouse
> UNION
> SELECT sm_ship_mode_id FROM ship_mode
> UNION
> SELECT r_reason_id FROM reason
> UNION
> SELECT cc_call_center_id FROM call_center
> UNION
> SELECT web_site_id FROM web_site
> UNION
> SELECT wp_web_page_id FROM web_page
> UNION
> SELECT cp_catalog_page_id FROM catalog_page;
> {code}
> hit the following error:
> {code}
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:96)
> ~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.HashTableGen3$BatchHolder.isKeyMatchInternalBuild(BatchHolder.java:171)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.isKeyMatch(HashTableTemplate.java:218)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.access$1000(HashTableTemplate.java:120)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.common.HashTableTemplate.put(HashTableTemplate.java:650)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.HashAggregatorGen0.checkGroupAndAggrValues(HashAggTemplate.java:1372)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.HashAggregatorGen0.doWork(HashAggTemplate.java:599)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:268)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.union.UnionAllRecordBatch$UnionInputIterator.next(UnionAllRecordBatch.java:381)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> {code}
> [~dechanggu] found that the issue is absent in Drill 1.13.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6622) UNION on tpcds sf100 tables hit SYSTEM ERROR: NullPointerException
[
https://issues.apache.org/jira/browse/DRILL-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu updated DRILL-6622:
--
Attachment: MD4208_id_05_1_id_24b2a6f9-ed66-b97e-594d-f116cd3fdd23.json
> UNION on tpcds sf100 tables hit SYSTEM ERROR: NullPointerException
> ---
>
> Key: DRILL-6622
> URL: https://issues.apache.org/jira/browse/DRILL-6622
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.14.0
>
> Attachments:
> MD4208_id_05_1_id_24b2a6f9-ed66-b97e-594d-f116cd3fdd23.json
>
>
> {code}
> SELECT c_customer_id FROM customer
> UNION
> SELECT ca_address_id FROM customer_address
> UNION
> SELECT cd_credit_rating FROM customer_demographics
> UNION
> SELECT hd_buy_potential FROM household_demographics
> UNION
> SELECT i_item_id FROM item
> UNION
> SELECT p_promo_id FROM promotion
> UNION
> SELECT t_time_id FROM time_dim
> UNION
> SELECT d_date_id FROM date_dim
> UNION
> SELECT s_store_id FROM store
> UNION
> SELECT w_warehouse_id FROM warehouse
> UNION
> SELECT sm_ship_mode_id FROM ship_mode
> UNION
> SELECT r_reason_id FROM reason
> UNION
> SELECT cc_call_center_id FROM call_center
> UNION
> SELECT web_site_id FROM web_site
> UNION
> SELECT wp_web_page_id FROM web_page
> UNION
> SELECT cp_catalog_page_id FROM catalog_page;
> {code}
> hit the following error:
> {code}
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:96)
> ~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.HashTableGen3$BatchHolder.isKeyMatchInternalBuild(BatchHolder.java:171)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.isKeyMatch(HashTableTemplate.java:218)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.access$1000(HashTableTemplate.java:120)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.common.HashTableTemplate.put(HashTableTemplate.java:650)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.HashAggregatorGen0.checkGroupAndAggrValues(HashAggTemplate.java:1372)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.HashAggregatorGen0.doWork(HashAggTemplate.java:599)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:268)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.union.UnionAllRecordBatch$UnionInputIterator.next(UnionAllRecordBatch.java:381)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> {code}
> [~dechanggu] found that the issue is absent in Drill 1.13.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6593) unordered receivers for broadcast senders dont rerpot memmory consumption
[
https://issues.apache.org/jira/browse/DRILL-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu updated DRILL-6593:
--
Attachment: TPCDS_78_3_id_24c43954-65b4-07b6-7e53-a37ad47fa963.json
> unordered receivers for broadcast senders dont rerpot memmory consumption
> -
>
> Key: DRILL-6593
> URL: https://issues.apache.org/jira/browse/DRILL-6593
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.14.0
> Environment: RHEL 7
>Reporter: Dechang Gu
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: TPCDS_78_3_id_24c43954-65b4-07b6-7e53-a37ad47fa963.json
>
>
> In my regression test on TPCDS SF100 dataset, query 78 profile shows the
> following:
> {code}
> 05-xx-02 PROJECT 0.000s 0.001s 0.003s 0.022s 0.000s 0.000s 0.000s
> 0.02% 0.00% 64,787,488 3MB 3MB
> 05-xx-03 HASH_JOIN 0.000s 0.000s 0.774s 1.002s 0.000s 0.000s
> 0.000s 6.87% 0.32% 69,186,507 8MB 10MB
> 05-xx-04 UNORDERED_RECEIVER 0.000s 0.000s 0.000s 0.000s 0.000s
> 0.000s 0.000s 0.00% 0.00% 4,382,940 - -
> 05-xx-05 PROJECT 0.000s 0.001s 0.002s 0.015s 0.000s 0.000s 0.000s
> 0.02% 0.00% 64,803,567 3MB 3MB
> 05-xx-06 SELECTION_VECTOR_REMOVER0.000s 0.000s 0.333s 0.566s
> 0.000s 0.000s 0.000s 2.95% 0.14% 64,803,567 5MB 5MB
> {code}
> Note 05-xx-04 did not show memory usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6593) unordered receivers for broadcast senders dont rerpot memmory consumption
[
https://issues.apache.org/jira/browse/DRILL-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu updated DRILL-6593:
--
Description:
In my regression test on TPCDS SF100 dataset, query 78 profile shows the
following:
{code}
05-xx-02PROJECT 0.000s 0.001s 0.003s 0.022s 0.000s 0.000s 0.000s
0.02% 0.00% 64,787,488 3MB 3MB
05-xx-03HASH_JOIN 0.000s 0.000s 0.774s 1.002s 0.000s 0.000s
0.000s 6.87% 0.32% 69,186,507 8MB 10MB
05-xx-04UNORDERED_RECEIVER 0.000s 0.000s 0.000s 0.000s 0.000s
0.000s 0.000s 0.00% 0.00% 4,382,940 - -
05-xx-05PROJECT 0.000s 0.001s 0.002s 0.015s 0.000s 0.000s 0.000s
0.02% 0.00% 64,803,567 3MB 3MB
05-xx-06SELECTION_VECTOR_REMOVER0.000s 0.000s 0.333s 0.566s
0.000s 0.000s 0.000s 2.95% 0.14% 64,803,567 5MB 5MB
{code}
Note 05-xx-04 did not show memory usage.
was:
In my regression test on TPCDS SF100 dataset, query 72 profile shows the
following:
{code}
05-xx-02PROJECT 0.000s 0.001s 0.003s 0.022s 0.000s 0.000s 0.000s
0.02% 0.00% 64,787,488 3MB 3MB
05-xx-03HASH_JOIN 0.000s 0.000s 0.774s 1.002s 0.000s 0.000s
0.000s 6.87% 0.32% 69,186,507 8MB 10MB
05-xx-04UNORDERED_RECEIVER 0.000s 0.000s 0.000s 0.000s 0.000s
0.000s 0.000s 0.00% 0.00% 4,382,940 - -
05-xx-05PROJECT 0.000s 0.001s 0.002s 0.015s 0.000s 0.000s 0.000s
0.02% 0.00% 64,803,567 3MB 3MB
05-xx-06SELECTION_VECTOR_REMOVER0.000s 0.000s 0.333s 0.566s
0.000s 0.000s 0.000s 2.95% 0.14% 64,803,567 5MB 5MB
{code}
Note 05-xx-04 did not show memory usage.
> unordered receivers for broadcast senders dont rerpot memmory consumption
> -
>
> Key: DRILL-6593
> URL: https://issues.apache.org/jira/browse/DRILL-6593
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.14.0
> Environment: RHEL 7
>Reporter: Dechang Gu
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> In my regression test on TPCDS SF100 dataset, query 78 profile shows the
> following:
> {code}
> 05-xx-02 PROJECT 0.000s 0.001s 0.003s 0.022s 0.000s 0.000s 0.000s
> 0.02% 0.00% 64,787,488 3MB 3MB
> 05-xx-03 HASH_JOIN 0.000s 0.000s 0.774s 1.002s 0.000s 0.000s
> 0.000s 6.87% 0.32% 69,186,507 8MB 10MB
> 05-xx-04 UNORDERED_RECEIVER 0.000s 0.000s 0.000s 0.000s 0.000s
> 0.000s 0.000s 0.00% 0.00% 4,382,940 - -
> 05-xx-05 PROJECT 0.000s 0.001s 0.002s 0.015s 0.000s 0.000s 0.000s
> 0.02% 0.00% 64,803,567 3MB 3MB
> 05-xx-06 SELECTION_VECTOR_REMOVER0.000s 0.000s 0.333s 0.566s
> 0.000s 0.000s 0.000s 2.95% 0.14% 64,803,567 5MB 5MB
> {code}
> Note 05-xx-04 did not show memory usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6593) unordered receivers for broadcast senders dont rerpot memmory consumption
Dechang Gu created DRILL-6593:
-
Summary: unordered receivers for broadcast senders dont rerpot
memmory consumption
Key: DRILL-6593
URL: https://issues.apache.org/jira/browse/DRILL-6593
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 1.14.0
Environment: RHEL 7
Reporter: Dechang Gu
Assignee: salim achouche
Fix For: 1.14.0
In my regression test on TPCDS SF100 dataset, query 72 profile shows the
following:
{code}
05-xx-02PROJECT 0.000s 0.001s 0.003s 0.022s 0.000s 0.000s 0.000s
0.02% 0.00% 64,787,488 3MB 3MB
05-xx-03HASH_JOIN 0.000s 0.000s 0.774s 1.002s 0.000s 0.000s
0.000s 6.87% 0.32% 69,186,507 8MB 10MB
05-xx-04UNORDERED_RECEIVER 0.000s 0.000s 0.000s 0.000s 0.000s
0.000s 0.000s 0.00% 0.00% 4,382,940 - -
05-xx-05PROJECT 0.000s 0.001s 0.002s 0.015s 0.000s 0.000s 0.000s
0.02% 0.00% 64,803,567 3MB 3MB
05-xx-06SELECTION_VECTOR_REMOVER0.000s 0.000s 0.333s 0.566s
0.000s 0.000s 0.000s 2.95% 0.14% 64,803,567 5MB 5MB
{code}
Note 05-xx-04 did not show memory usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6448) query on nested data returns bogus results
Dechang Gu created DRILL-6448:
-
Summary: query on nested data returns bogus results
Key: DRILL-6448
URL: https://issues.apache.org/jira/browse/DRILL-6448
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 1.13.0
Reporter: Dechang Gu
I created a nested parquet table nested_c_o_l with the following schema:
{code}
message root {
optional binary c_custkey (UTF8);
optional binary c_name (UTF8);
optional binary c_address (UTF8);
optional int64 c_nationkey;
optional binary c_phone (UTF8);
optional double c_acctbal;
optional binary c_mktsegment (UTF8);
optional binary c_comment (UTF8);
repeated group c_orders {
optional int64 o_orderkey;
optional binary o_orderstatus (UTF8);
optional double o_totalprice;
optional binary o_orderdate (UTF8);
optional binary o_orderpriority (UTF8);
optional binary o_clerk (UTF8);
optional int64 o_shippriority;
optional binary o_comment (UTF8);
repeated group o_lineitems {
optional int64 l_partkey;
optional int64 l_suppkey;
optional int64 l_linenumber;
optional int64 l_quantity;
optional double l_extendedprice;
optional double l_discount;
optional double l_tax;
optional binary l_returnflag (UTF8);
optional binary l_linestatus (UTF8);
optional binary l_shipdate (UTF8);
optional binary l_commitdate (UTF8);
optional binary l_receiptdate (UTF8);
optional binary l_shipinstruct (UTF8);
optional binary l_shipmode (UTF8);
optional binary l_comment (UTF8);
}
}
}
{code}
and run the following query:
{code}
select * from nested_c_o_l c where c.c_orders.o_orderdate='1997-06-23';
{code}
It returns two row, with the 2nd row does not satisfy the filter condition:
{code}0: jdbc:drill:schema=dfs.tpcdsView> select * from nested_c_o_l c where
c.c_orders.o_orderdate='1997-06-23';
+---++---+-+-+---+--+---+--+
| c_custkey | c_name | c_address | c_nationkey | c_phone | c_acctbal |
c_mktsegment | c_comment | c_orders |
+---++---+-+-+---+--+---+--+
| 1 | Customer#1 | IVhzIApeRb ot,c,E | 15 | 25-989-741-2988 | 711.56 |
BUILDING | to the even, regular platelets. regular, ironic epitaphs nag e |
[{"o_orderkey":9154,"o_orderstatus":"O","o_totalprice":357345.46,"o_orderdate":"1997-06-23","o_orderpriority":"4-NOT
SPECIFIED","o_clerk":"Clerk#00328","o_shippriority":0,"o_comment":"y
ironic packages cajole. blithely final
depende","o_lineitems":[{"l_partkey":866,"l_suppkey":100,"l_linenumber":1,"l_quantity":45,"l_extendedprice":79508.7,"l_discount":0.06,"l_tax":0.06,"l_returnflag":"N","l_linestatus":"O","l_shipdate":"1997-09-24","l_commitdate":"1997-08-11","l_receiptdate":"1997-10-14","l_shipinstruct":"NONE","l_shipmode":"FOB","l_comment":"nal,
careful instructions wake carefully.
b"},{"l_partkey":1735,"l_suppkey":36,"l_linenumber":7,"l_quantity":47,"l_extendedprice":76926.31,"l_discount":0.04,"l_tax":0.05,"l_returnflag":"N","l_linestatus":"O","l_shipdate":"1997-07-07","l_commitdate":"1997-09-07","l_receiptdate":"1997-07-25","l_shipinstruct":"DELIVER
IN PERSON","l_shipmode":"SHIP","l_comment":"wake boldly above the
furiousl"},{"l_partkey":1403,"l_suppkey":43,"l_linenumber":6,"l_quantity":40,"l_extendedprice":52176.0,"l_discount":0.03,"l_tax":0.02,"l_returnflag":"N","l_linestatus":"O","l_shipdate":"1997-06-24","l_commitdate":"1997-09-03","l_receiptdate":"1997-07-24","l_shipinstruct":"COLLECT
COD","l_shipmode":"TRUCK","l_comment":"t haggle
bli"},{"l_partkey":1770,"l_suppkey":13,"l_linenumber":5,"l_quantity":12,"l_extendedprice":20061.24,"l_discount":0.0,"l_tax":0.0,"l_returnflag":"N","l_linestatus":"O","l_shipdate":"1997-08-20","l_commitdate":"1997-07-26","l_receiptdate":"1997-09-17","l_shipinstruct":"NONE","l_shipmode":"RAIL","l_comment":"es.
requests print furiously instead of
th"},{"l_partkey":1967,"l_suppkey":100,"l_linenumber":4,"l_quantity":31,"l_extendedprice":57937.76,"l_discount":0.0,"l_tax":0.0,"l_returnflag":"N","l_linestatus":"O","l_shipdate":"1997-08-28","l_commitdate":"1997-07-29","l_receiptdate":"1997-09-07","l_shipinstruct":"NONE","l_shipmode":"AIR","l_comment":"final
warthogs. slyly pending
request"},{"l_partkey":534,"l_suppkey":65,"l_linenumber":3,"l_quantity":46,"l_extendedprice":65988.38,"l_discount":0.04,"l_tax":0.01,"l_returnflag":"N","l_linestatus":"O","l_shipdate":"1997-07-18","l_commitdate":"1997-08-22","l_receiptdate":"1997-07-31","l_shipinstruct":"NONE","l_shipmode":"MAIL","l_comment":"inal
depths. blithely quick deposits
[jira] [Updated] (DRILL-6374) TPCH Queries regressed and OOM when run concurrency test
[ https://issues.apache.org/jira/browse/DRILL-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu updated DRILL-6374: -- Description: Run TPCH regression test on Apache Drill 1.14.0 master commit 6fcaf4268eddcb09010b5d9c5dfb3b3be5c3f903 (DRILL-6173), most of the queries regressed. In particular, TPC-H Query 9 takes about 4x time (36 sec vs 8.6 sec), comparing to that when run against the parent commit (9173308710c3decf8ff745493ad3e85ccdaf7c37). Further in the concurrency test for the commit, with 48 clients each running 16 TPCH queries (so total 768 queries are executed) with planner.width.max_per_node=5, some queries hit OOM and caused 273 queries failed, while for the parent commit all the 768 queries completed successfully.  Profiles for TPCH_09 in the regression tests are uploaded: * The failing commit file name: [^TPCH_09_2_id_2517381b-1a61-3db5-40c3-4463bd421365.json], * The parent commit file name: [^TPCH_09_2_id_2517497b-d4da-dab6-6124-abde5804a25f.json] ). was: Run TPCH regression test on Apache Drill 1.14.0 master commit 6fcaf4268eddcb09010b5d9c5dfb3b3be5c3f903 (DRILL-6173), most of the queries regressed. In particular, TPC-H Query 9 takes about 4x time (36 sec vs 8.6 sec), comparing to that when run against the parent commit (9173308710c3decf8ff745493ad3e85ccdaf7c37). Further in the concurrency test for the commit, with 48 clients each running 16 TPCH queries (so total 768 queries are executed) with planner.width.max_per_node=5, some queries hit OOM and caused 266 queries failed, while for the parent commit all the 768 queries completed successfully.  Profiles for TPCH_09 in the regression tests are uploaded: * The failing commit file name: [^TPCH_09_2_id_2517381b-1a61-3db5-40c3-4463bd421365.json], * The parent commit file name: [^TPCH_09_2_id_2517497b-d4da-dab6-6124-abde5804a25f.json] ). > TPCH Queries regressed and OOM when run concurrency test > > > Key: DRILL-6374 > URL: https://issues.apache.org/jira/browse/DRILL-6374 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.14.0 > Environment: RHEL 7 >Reporter: Dechang Gu >Assignee: Vitalii Diravka >Priority: Critical > Fix For: 1.14.0 > > Attachments: TPCH_09_2_id_2517381b-1a61-3db5-40c3-4463bd421365.json, > TPCH_09_2_id_2517497b-d4da-dab6-6124-abde5804a25f.json > > > Run TPCH regression test on Apache Drill 1.14.0 master commit > 6fcaf4268eddcb09010b5d9c5dfb3b3be5c3f903 (DRILL-6173), most of the queries > regressed. > In particular, TPC-H Query 9 takes about 4x time (36 sec vs 8.6 sec), > comparing to that when run against the parent commit > (9173308710c3decf8ff745493ad3e85ccdaf7c37). > Further in the concurrency test for the commit, with 48 clients each running > 16 TPCH queries (so total 768 queries are executed) with > planner.width.max_per_node=5, some queries hit OOM and caused 273 queries > failed, while for the parent commit all the 768 queries completed > successfully. >  > Profiles for TPCH_09 in the regression tests are uploaded: > * The failing commit file name: > [^TPCH_09_2_id_2517381b-1a61-3db5-40c3-4463bd421365.json], > * The parent commit file name: > [^TPCH_09_2_id_2517497b-d4da-dab6-6124-abde5804a25f.json] ). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6374) TPCH Queries regressed and OOM when run concurrency test
[
https://issues.apache.org/jira/browse/DRILL-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460238#comment-16460238
]
Dechang Gu commented on DRILL-6374:
---
For the OOM, here is the stack:
{code}
2018-05-01 13:40:42,457 [25172fdc-6af9-e530-dbcd-6ac47cf16b00:frag:4:41] INFO
o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes
ran out of memory while executing the query. (AGGR OOM at First Phase.
Partitions: 16. Estimated batch size: 26673152. values size: 1048576. Output
alloc size: 1048576. Planned batches: 2 Memory limit: 856896307 so far
allocated: 377618432. )
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more
nodes ran out of memory while executing the query.
AGGR OOM at First Phase. Partitions: 16. Estimated batch size: 26673152. values
size: 1048576. Output alloc size: 1048576. Planned batches: 2 Memory limit:
856896307 so far allocated: 377618432.
[Error Id: 8eb64127-e22f-4aea-aff9-db5be8ff3574 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:304)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: AGGR OOM at
First Phase. Partitions: 16. Estimated batch size: 26673152. values size:
1048576. Output alloc size: 1048576. Planned batches: 2 Memory limit: 856896307
so far allocated: 377618432.
at
org.apache.drill.exec.test.generated.HashAggregatorGen3940.spillIfNeeded(HashAggTemplate.java:1419)
~[na:na]
at
org.apache.drill.exec.test.generated.HashAggregatorGen3940.doSpill(HashAggTemplate.java:1381)
~[na:na]
at
org.apache.drill.exec.test.generated.HashAggregatorGen3940.checkGroupAndAggrValues(HashAggTemplate.java:1304)
~[na:na]
at
org.apache.drill.exec.test.generated.HashAggregatorGen3940.doWork(HashAggTemplate.java:592)
~[na:na]
at
org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:176)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext(PartitionSenderRootExec.java:152)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:292)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:279)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method)
~[na:1.8.0_112]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_112]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
~[hadoop-common-2.7.0-mapr-1707.jar:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:279)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
... 4 common frames omitted{code}
> TPCH Queries regressed and OOM when run concurrency test
>
>
> Key: DRILL-6374
> URL: https://issues.apache.org/jira/browse/DRILL-6374
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.14.0
> Environment: RHEL 7
>Reporter: Dechang Gu
>Assignee: Vitalii Diravka
>Priority: Critical
> Fix For: 1.14.0
>
> Attachments: TPCH_09_2_id_2517381b-1a61-3db5-40c3-4463bd421365.json,
> TPCH_09_2_id_2517497b-d4da-dab6-6124-abde5804a25f.json
>
>
> Run TPCH regression test on Apache Drill 1.14.0 master commit
> 6fcaf4268eddcb09010b5d9c5dfb3b3be5c3f903 (DRILL-6173), most of the queries
> regressed.
> In particular, TPC-H Query 9 takes about 4x time
[jira] [Commented] (DRILL-6374) TPCH Queries regressed and OOM when run concurrency test
[ https://issues.apache.org/jira/browse/DRILL-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460231#comment-16460231 ] Dechang Gu commented on DRILL-6374: --- Query 08 also takes 4x time: 30 sec vs 7.5 sec in the parent commit. > TPCH Queries regressed and OOM when run concurrency test > > > Key: DRILL-6374 > URL: https://issues.apache.org/jira/browse/DRILL-6374 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.14.0 > Environment: RHEL 7 >Reporter: Dechang Gu >Assignee: Vitalii Diravka >Priority: Critical > Fix For: 1.14.0 > > Attachments: TPCH_09_2_id_2517381b-1a61-3db5-40c3-4463bd421365.json, > TPCH_09_2_id_2517497b-d4da-dab6-6124-abde5804a25f.json > > > Run TPCH regression test on Apache Drill 1.14.0 master commit > 6fcaf4268eddcb09010b5d9c5dfb3b3be5c3f903 (DRILL-6173), most of the queries > regressed. > In particular, TPC-H Query 9 takes about 4x time (36 sec vs 8.6 sec), > comparing to that when run against the parent commit > (9173308710c3decf8ff745493ad3e85ccdaf7c37). > Further in the concurrency test for the commit, with 48 clients each running > 16 TPCH queries (so total 768 queries are executed) with > planner.width.max_per_node=5, some queries hit OOM and caused 266 queries > failed, while for the parent commit all the 768 queries completed > successfully. >  > Profiles for TPCH_09 in the regression tests are uploaded: > * The failing commit file name: > [^TPCH_09_2_id_2517381b-1a61-3db5-40c3-4463bd421365.json], > * The parent commit file name: > [^TPCH_09_2_id_2517497b-d4da-dab6-6124-abde5804a25f.json] ). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6374) TPCH Queries regressed and OOM when run concurrency test
Dechang Gu created DRILL-6374: - Summary: TPCH Queries regressed and OOM when run concurrency test Key: DRILL-6374 URL: https://issues.apache.org/jira/browse/DRILL-6374 Project: Apache Drill Issue Type: Bug Components: Functions - Drill Affects Versions: 1.14.0 Environment: RHEL 7 Reporter: Dechang Gu Assignee: Vitalii Diravka Fix For: 1.14.0 Attachments: TPCH_09_2_id_2517381b-1a61-3db5-40c3-4463bd421365.json, TPCH_09_2_id_2517497b-d4da-dab6-6124-abde5804a25f.json Run TPCH regression test on Apache Drill 1.14.0 master commit 6fcaf4268eddcb09010b5d9c5dfb3b3be5c3f903 (DRILL-6173), most of the queries regressed. in particular, Queries 9 takes about 4x time (36 sec vs 8.6 sec), comparing to that when run against the parent commit (9173308710c3decf8ff745493ad3e85ccdaf7c37). Further in the concurrency test for the commit, with 48 clients each running 16 TPCH queries (so total 768 queries are executed) with planner.width.max_per_node=5, some queries hit OOM and caused 266 queries failed, while for the parent commit all the 768 queries completed successfully. Profiles for TPCH_09 in the regression tests are uploaded (the failing commit file name: TPCH_09_2_id_2517381b-1a61-3db5-40c3-4463bd421365.json, and the parent commit file name: TPCH_09_2_id_2517497b-d4da-dab6-6124-abde5804a25f.json ). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6329) TPC-DS Query 66 failed due to OOM
[
https://issues.apache.org/jira/browse/DRILL-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441163#comment-16441163
]
Dechang Gu commented on DRILL-6329:
---
I also hit this error on a 10 node cluster with similar setting:
{code}
java.sql.SQLException: RESOURCE ERROR: Not enough memory for internal
partitioning and fallback mechanism for HashAgg to use unbounded memory is
disabled. Either enable fallback config drill.exec.hashagg.fallback.enabled
using Alter session/system command or increase memory limit for Drillbit
Fragment 4:131
[Error Id: db098d7a-109d-49bf-9af3-a1159e083dc7 on ucs-node11.perf.lab:31010]
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:530)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:634)
at
org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:239)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:159)
at PipSQueak.fetchRows(PipSQueak.java:346)
at PipSQueak.runTest(PipSQueak.java:113)
at PipSQueak.main(PipSQueak.java:477)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: RESOURCE
ERROR: Not enough memory for internal partitioning and fallback mechanism for
HashAgg to use unbounded memory is disabled. Either enable fallback config
drill.exec.hashagg.fallback.enabled using Alter session/system command or
increase memory limit for Drillbit
Fragment 4:131
[Error Id: db098d7a-109d-49bf-9af3-a1159e083dc7 on ucs-node11.perf.lab:31010]
at
org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:127)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:429)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:98)
at
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)
at
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at
[jira] [Commented] (DRILL-5967) Memory leak by HashPartitionSender
[
https://issues.apache.org/jira/browse/DRILL-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284483#comment-16284483
]
Dechang Gu commented on DRILL-5967:
---
Is it this one: c8ca958eeb5e3b314aca9bb451e1631cbde18d83
@ilooner
- Possible fix for memory leak in HashPartitionSender
ilooner committed 17 days ago
Will try it.
> Memory leak by HashPartitionSender
> --
>
> Key: DRILL-5967
> URL: https://issues.apache.org/jira/browse/DRILL-5967
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> The error found by [~cch...@maprtech.com] and [~dechanggu]
> {code}
> 2017-10-25 15:43:28,658 [260eec84-7de3-03ec-300f-7fdbc111fb7c:frag:2:9] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
> Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
> ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
> [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
> [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
> [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query.
> Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100
> (res/actual/peak/limit)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5967) Memory leak by HashPartitionSender
[
https://issues.apache.org/jira/browse/DRILL-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284170#comment-16284170
]
Dechang Gu commented on DRILL-5967:
---
The query did not complete, it was cancelled after this error. detailed log can
be found here (on selfhosting, it's too big to upload tto the jira):
MAPRTECH/home/dgu/perf_tests/private-drill-perf-test-framework/concurrency/log/231_5.1.0.37549.GA_f599401_TPCH_QueueOFF_24_20171207_002530:$
ls -ltr
-rw-r--r-- 1 root wheel 127527377 Dec 7 02:26
drillbit-231_20171207_002530_ucs-node7.log
-rw-r--r-- 1 root wheel 262923105 Dec 7 02:26
drillbit-231_20171207_002530_ucs-node6.log
> Memory leak by HashPartitionSender
> --
>
> Key: DRILL-5967
> URL: https://issues.apache.org/jira/browse/DRILL-5967
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> The error found by [~cch...@maprtech.com] and [~dechanggu]
> {code}
> 2017-10-25 15:43:28,658 [260eec84-7de3-03ec-300f-7fdbc111fb7c:frag:2:9] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
> Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
> ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
> [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
> [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
> [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query.
> Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100
> (res/actual/peak/limit)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (DRILL-5967) Memory leak by HashPartitionSender
[
https://issues.apache.org/jira/browse/DRILL-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284121#comment-16284121
]
Dechang Gu edited comment on DRILL-5967 at 12/8/17 7:54 PM:
This is from concurrency test: 16 clients, each executes 16 TPCH queries on
SF500 parquet files, no complex types.
was (Author: dechanggu):
This is from concurrency test: 16 clients, each executes 16 TPCH queries on
SF100 parquet files, no complex types.
> Memory leak by HashPartitionSender
> --
>
> Key: DRILL-5967
> URL: https://issues.apache.org/jira/browse/DRILL-5967
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> The error found by [~cch...@maprtech.com] and [~dechanggu]
> {code}
> 2017-10-25 15:43:28,658 [260eec84-7de3-03ec-300f-7fdbc111fb7c:frag:2:9] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
> Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
> ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
> [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
> [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
> [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query.
> Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100
> (res/actual/peak/limit)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5967) Memory leak by HashPartitionSender
[
https://issues.apache.org/jira/browse/DRILL-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284121#comment-16284121
]
Dechang Gu commented on DRILL-5967:
---
This is from concurrency test: 16 clients, each executes 16 TPCH queries on
SF100 parquet files, no complex types.
> Memory leak by HashPartitionSender
> --
>
> Key: DRILL-5967
> URL: https://issues.apache.org/jira/browse/DRILL-5967
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> The error found by [~cch...@maprtech.com] and [~dechanggu]
> {code}
> 2017-10-25 15:43:28,658 [260eec84-7de3-03ec-300f-7fdbc111fb7c:frag:2:9] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
> Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
> ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
> [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
> [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
> [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query.
> Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100
> (res/actual/peak/limit)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5967) Memory leak by HashPartitionSender
[
https://issues.apache.org/jira/browse/DRILL-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16282032#comment-16282032
]
Dechang Gu commented on DRILL-5967:
---
Tried the fix (gitid f599401) but still hit memory leak:
{code}
2017-12-07 00:30:45,370 ucs-node7.perf.lab
[25d704d2-562a-3cb6-6aa8-0a1e4cd5101f:frag:8:7] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
Memory was leaked by query. Memory leaked: (50176)
Allocator(op:8:7:0:HashPartitionSender) 100/0/13447168/100
(res/actual/peak/limit)
Fragment 8:7
[Error Id: ed187677-6ef6-460f-84aa-e40148021487 on ucs-node7.perf.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (50176)
Allocator(op:8:7:0:HashPartitionSender) 100/0/13447168/100
(res/actual/peak/limit)
Fragment 8:7
[Error Id: ed187677-6ef6-460f-84aa-e40148021487 on ucs-node7.perf.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_144]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory
leaked: (50176)
Allocator(op:8:7:0:HashPartitionSender) 100/0/13447168/100
(res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:520)
~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.ops.BaseOperatorContext.close(BaseOperatorContext.java:157)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:79)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:414)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:403)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:324)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
... 5 common frames omitted
{code}
Besides HashPartitionSender memory leak, some other operators also have memory
leak, e.g.:
{code}
2017-12-07 02:06:20,694 ucs-node7.perf.lab
[25d6ed7c-6119-00d8-1626-cffae53c4b05:frag:11:103] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
Memory was leaked by query. Memory leaked: (524288)
Allocator(op:11:103:6:ParquetRowGroupScan) 100/524288/10043392/100
(res/actual/peak/limit)
Fragment 11:103
[Error Id: b0b43e46-6867-4d47-b296-0f51b7e982cb on ucs-node7.perf.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (524288)
Allocator(op:11:103:6:ParquetRowGroupScan) 100/524288/10043392/100
(res/actual/peak/limit)
Fragment 11:103
[Error Id: b0b43e46-6867-4d47-b296-0f51b7e982cb on ucs-node7.perf.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[jira] [Commented] (DRILL-5468) TPCH Q18 regressed ~3x due to execution plan changes
[ https://issues.apache.org/jira/browse/DRILL-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16227570#comment-16227570 ] Dechang Gu commented on DRILL-5468: --- I tried this setting on SF100, Q18 did not show the 3x regression, performance is on par with previous normal runs. > TPCH Q18 regressed ~3x due to execution plan changes > > > Key: DRILL-5468 > URL: https://issues.apache.org/jira/browse/DRILL-5468 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.11.0 > Environment: 10+1 node ucs-micro cluster RHEL6.4 >Reporter: Dechang Gu >Assignee: Jinfeng Ni > Attachments: Q18_profile_gitid_841ead4, Q18_profile_gitid_adbf363 > > > In a regular regression test on Drill master (commit id 841ead4) TPCH Q18 on > SF100 parquet dataset took ~81 secs, while the same query on 1.10.0 took only > ~27 secs. The query time on the commit adbf363 which is right before 841ead4 > is ~32 secs. > Profiles shows the plans for the query changed quite a bit (profiles will be > uploaded) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5817) deprecated sys options should be deleted from old zookeeper setting
[
https://issues.apache.org/jira/browse/DRILL-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu updated DRILL-5817:
--
Environment: drill 1.11.0 (was: drill 1.12.0 )
> deprecated sys options should be deleted from old zookeeper setting
> ---
>
> Key: DRILL-5817
> URL: https://issues.apache.org/jira/browse/DRILL-5817
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.11.0
> Environment: drill 1.11.0
>Reporter: Dechang Gu
>Assignee: Timothy Farkas
> Fix For: 1.11.0
>
>
> In the following example, exec.query.rowkeyjoin.batchsize is a deprecated
> sys options, which is replaced with exec.query.rowkeyjoin_batchsize. Further
> both of them are just for MapR drill distribution, hence should not be listed
> at all.
> {code}
> 0: jdbc:drill:schema=default> select * from sys.options where name like
> '%batch%';
> +---+---+-+--+--+-+---++
> | name | kind | type | status
> | num_val | string_val | bool_val | float_val |
> +---+---+-+--+--+-+---++
> | drill.exec.hashagg.min_batches_per_partition | LONG | SYSTEM | DEFAULT
> | 3| null| null | null |
> | exec.query.rowkeyjoin.batchsize | LONG | SYSTEM | DEFAULT
> | 128 | null| null | null |
> | exec.query.rowkeyjoin_batchsize | LONG | SYSTEM | DEFAULT
> | 128 | null| null | null |
> +---+---+-+--+--+-+---++
> 3 rows selected (0.118 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (DRILL-5817) deprecated sys options should be deleted from old zookeeper setting
[
https://issues.apache.org/jira/browse/DRILL-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu updated DRILL-5817:
--
Fix Version/s: (was: 1.12.0)
1.11.0
> deprecated sys options should be deleted from old zookeeper setting
> ---
>
> Key: DRILL-5817
> URL: https://issues.apache.org/jira/browse/DRILL-5817
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.11.0
> Environment: drill 1.12.0
>Reporter: Dechang Gu
>Assignee: Timothy Farkas
> Fix For: 1.11.0
>
>
> In the following example, exec.query.rowkeyjoin.batchsize is a deprecated
> sys options, which is replaced with exec.query.rowkeyjoin_batchsize. Further
> both of them are just for MapR drill distribution, hence should not be listed
> at all.
> {code}
> 0: jdbc:drill:schema=default> select * from sys.options where name like
> '%batch%';
> +---+---+-+--+--+-+---++
> | name | kind | type | status
> | num_val | string_val | bool_val | float_val |
> +---+---+-+--+--+-+---++
> | drill.exec.hashagg.min_batches_per_partition | LONG | SYSTEM | DEFAULT
> | 3| null| null | null |
> | exec.query.rowkeyjoin.batchsize | LONG | SYSTEM | DEFAULT
> | 128 | null| null | null |
> | exec.query.rowkeyjoin_batchsize | LONG | SYSTEM | DEFAULT
> | 128 | null| null | null |
> +---+---+-+--+--+-+---++
> 3 rows selected (0.118 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (DRILL-5817) deprecated sys options should be deleted from old zookeeper setting
[
https://issues.apache.org/jira/browse/DRILL-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu updated DRILL-5817:
--
Affects Version/s: (was: 1.12.0)
1.11.0
> deprecated sys options should be deleted from old zookeeper setting
> ---
>
> Key: DRILL-5817
> URL: https://issues.apache.org/jira/browse/DRILL-5817
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.11.0
> Environment: drill 1.12.0
>Reporter: Dechang Gu
>Assignee: Timothy Farkas
> Fix For: 1.11.0
>
>
> In the following example, exec.query.rowkeyjoin.batchsize is a deprecated
> sys options, which is replaced with exec.query.rowkeyjoin_batchsize. Further
> both of them are just for MapR drill distribution, hence should not be listed
> at all.
> {code}
> 0: jdbc:drill:schema=default> select * from sys.options where name like
> '%batch%';
> +---+---+-+--+--+-+---++
> | name | kind | type | status
> | num_val | string_val | bool_val | float_val |
> +---+---+-+--+--+-+---++
> | drill.exec.hashagg.min_batches_per_partition | LONG | SYSTEM | DEFAULT
> | 3| null| null | null |
> | exec.query.rowkeyjoin.batchsize | LONG | SYSTEM | DEFAULT
> | 128 | null| null | null |
> | exec.query.rowkeyjoin_batchsize | LONG | SYSTEM | DEFAULT
> | 128 | null| null | null |
> +---+---+-+--+--+-+---++
> 3 rows selected (0.118 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (DRILL-5817) deprecated sys options should be deleted from old zookeeper setting
[
https://issues.apache.org/jira/browse/DRILL-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu updated DRILL-5817:
--
Fix Version/s: (was: 1.11.0)
1.12.0
> deprecated sys options should be deleted from old zookeeper setting
> ---
>
> Key: DRILL-5817
> URL: https://issues.apache.org/jira/browse/DRILL-5817
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.12.0
> Environment: drill 1.11.0
>Reporter: Dechang Gu
>Assignee: Timothy Farkas
> Fix For: 1.12.0
>
>
> In the following example, exec.query.rowkeyjoin.batchsize is a deprecated
> sys options, which is replaced with exec.query.rowkeyjoin_batchsize. Further
> both of them are just for MapR drill distribution, hence should not be listed
> at all.
> {code}
> 0: jdbc:drill:schema=default> select * from sys.options where name like
> '%batch%';
> +---+---+-+--+--+-+---++
> | name | kind | type | status
> | num_val | string_val | bool_val | float_val |
> +---+---+-+--+--+-+---++
> | drill.exec.hashagg.min_batches_per_partition | LONG | SYSTEM | DEFAULT
> | 3| null| null | null |
> | exec.query.rowkeyjoin.batchsize | LONG | SYSTEM | DEFAULT
> | 128 | null| null | null |
> | exec.query.rowkeyjoin_batchsize | LONG | SYSTEM | DEFAULT
> | 128 | null| null | null |
> +---+---+-+--+--+-+---++
> 3 rows selected (0.118 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (DRILL-5817) deprecated sys options should be deleted from old zookeeper setting
[
https://issues.apache.org/jira/browse/DRILL-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu updated DRILL-5817:
--
Affects Version/s: (was: 1.11.0)
1.12.0
> deprecated sys options should be deleted from old zookeeper setting
> ---
>
> Key: DRILL-5817
> URL: https://issues.apache.org/jira/browse/DRILL-5817
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.12.0
> Environment: drill 1.12.0
>Reporter: Dechang Gu
>Assignee: Timothy Farkas
> Fix For: 1.12.0
>
>
> In the following example, exec.query.rowkeyjoin.batchsize is a deprecated
> sys options, which is replaced with exec.query.rowkeyjoin_batchsize. Further
> both of them are just for MapR drill distribution, hence should not be listed
> at all.
> {code}
> 0: jdbc:drill:schema=default> select * from sys.options where name like
> '%batch%';
> +---+---+-+--+--+-+---++
> | name | kind | type | status
> | num_val | string_val | bool_val | float_val |
> +---+---+-+--+--+-+---++
> | drill.exec.hashagg.min_batches_per_partition | LONG | SYSTEM | DEFAULT
> | 3| null| null | null |
> | exec.query.rowkeyjoin.batchsize | LONG | SYSTEM | DEFAULT
> | 128 | null| null | null |
> | exec.query.rowkeyjoin_batchsize | LONG | SYSTEM | DEFAULT
> | 128 | null| null | null |
> +---+---+-+--+--+-+---++
> 3 rows selected (0.118 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (DRILL-5817) deprecated sys options should be deleted from old zookeeper setting
[
https://issues.apache.org/jira/browse/DRILL-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu updated DRILL-5817:
--
Environment: drill 1.12.0 (was: drill 1.11.0 )
> deprecated sys options should be deleted from old zookeeper setting
> ---
>
> Key: DRILL-5817
> URL: https://issues.apache.org/jira/browse/DRILL-5817
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.12.0
> Environment: drill 1.12.0
>Reporter: Dechang Gu
>Assignee: Timothy Farkas
> Fix For: 1.12.0
>
>
> In the following example, exec.query.rowkeyjoin.batchsize is a deprecated
> sys options, which is replaced with exec.query.rowkeyjoin_batchsize. Further
> both of them are just for MapR drill distribution, hence should not be listed
> at all.
> {code}
> 0: jdbc:drill:schema=default> select * from sys.options where name like
> '%batch%';
> +---+---+-+--+--+-+---++
> | name | kind | type | status
> | num_val | string_val | bool_val | float_val |
> +---+---+-+--+--+-+---++
> | drill.exec.hashagg.min_batches_per_partition | LONG | SYSTEM | DEFAULT
> | 3| null| null | null |
> | exec.query.rowkeyjoin.batchsize | LONG | SYSTEM | DEFAULT
> | 128 | null| null | null |
> | exec.query.rowkeyjoin_batchsize | LONG | SYSTEM | DEFAULT
> | 128 | null| null | null |
> +---+---+-+--+--+-+---++
> 3 rows selected (0.118 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5817) deprecated sys options should be deleted from old zookeeper setting
Dechang Gu created DRILL-5817:
-
Summary: deprecated sys options should be deleted from old
zookeeper setting
Key: DRILL-5817
URL: https://issues.apache.org/jira/browse/DRILL-5817
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 1.11.0
Environment: drill 1.11.0
Reporter: Dechang Gu
Fix For: 1.11.0
In the following example, exec.query.rowkeyjoin.batchsize is a deprecated sys
options, which is replaced with exec.query.rowkeyjoin_batchsize. Further both
of them are just for MapR drill distribution, hence should not be listed at
all.
{code}
0: jdbc:drill:schema=default> select * from sys.options where name like
'%batch%';
+---+---+-+--+--+-+---++
| name | kind | type | status |
num_val | string_val | bool_val | float_val |
+---+---+-+--+--+-+---++
| drill.exec.hashagg.min_batches_per_partition | LONG | SYSTEM | DEFAULT |
3| null| null | null |
| exec.query.rowkeyjoin.batchsize | LONG | SYSTEM | DEFAULT |
128 | null| null | null |
| exec.query.rowkeyjoin_batchsize | LONG | SYSTEM | DEFAULT |
128 | null| null | null |
+---+---+-+--+--+-+---++
3 rows selected (0.118 seconds)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5809) Queries failed with UnrecognizedPropertyException
[
https://issues.apache.org/jira/browse/DRILL-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179303#comment-16179303
]
Dechang Gu commented on DRILL-5809:
---
[~timothyfarkas]Please merge your patch to the master, thanks.
> Queries failed with UnrecognizedPropertyException
> -
>
> Key: DRILL-5809
> URL: https://issues.apache.org/jira/browse/DRILL-5809
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.12.0
> Environment: RHEL 6, 2.6.32-358.el6.x86_64, Apache Drill gitid
> 4c99f0cdd98b1a0e789c5d11b353f88a2f6d54aa
>Reporter: Dechang Gu
>Assignee: Timothy Farkas
>Priority: Blocker
> Fix For: 1.12.0
>
>
> Run tpch queries, all failed due to the following error:
> {code}
> java.sql.SQLException: SYSTEM ERROR: UnrecognizedPropertyException:
> Unrecognized field "type" (class
> org.apache.drill.exec.server.options.OptionValue), not marked as ignorable (8
> known properties: "string_val", "kind", "accessibleScopes", "num_val",
> "name", "bool_val", "float_val", "scope"])
> at [Source: [B@4ec364a3; line: 6, column: 2] (through reference chain:
> org.apache.drill.exec.server.options.OptionValue["type"])
> [Error Id: 9bad34d9-fe21-47fd-ade1-fbee3d5111e5 on ucs-node7.perf.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
> at
> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:561)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1895)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:473)
> at
> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1100)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:477)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:181)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:109)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:130)
> at
> org.apache.drill.jdbc.impl.DrillStatementImpl.executeQuery(DrillStatementImpl.java:112)
> at PipSQueak.executeQuery(PipSQueak.java:289)
> at PipSQueak.runTest(PipSQueak.java:104)
> at PipSQueak.main(PipSQueak.java:477)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: UnrecognizedPropertyException: Unrecognized field "type" (class
> org.apache.drill.exec.server.options.OptionValue), not marked as ignorable (8
> known properties: "string_val", "kind", "accessibleScopes", "num_val",
> "name", "bool_val", "float_val", "scope"])
> at [Source: [B@4ec364a3; line: 6, column: 2] (through reference chain:
> org.apache.drill.exec.server.options.OptionValue["type"])
> {code}
> Looks like the issue was introduced in commit:
> 6adeb986016a769755fd5e8fc66244ee1e8d18e1
> DRILL-5723: Added System Internal Options That can be Modified at Run…
> The commit prior to this one did not show the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (DRILL-5809) Queries failed with UnrecognizedPropertyException
[
https://issues.apache.org/jira/browse/DRILL-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu closed DRILL-5809.
-
Resolution: Fixed
[~timothyfarkas] Tried your new patch, LGTM. Thanks.
> Queries failed with UnrecognizedPropertyException
> -
>
> Key: DRILL-5809
> URL: https://issues.apache.org/jira/browse/DRILL-5809
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.12.0
> Environment: RHEL 6, 2.6.32-358.el6.x86_64, Apache Drill gitid
> 4c99f0cdd98b1a0e789c5d11b353f88a2f6d54aa
>Reporter: Dechang Gu
>Assignee: Timothy Farkas
>Priority: Blocker
> Fix For: 1.12.0
>
>
> Run tpch queries, all failed due to the following error:
> {code}
> java.sql.SQLException: SYSTEM ERROR: UnrecognizedPropertyException:
> Unrecognized field "type" (class
> org.apache.drill.exec.server.options.OptionValue), not marked as ignorable (8
> known properties: "string_val", "kind", "accessibleScopes", "num_val",
> "name", "bool_val", "float_val", "scope"])
> at [Source: [B@4ec364a3; line: 6, column: 2] (through reference chain:
> org.apache.drill.exec.server.options.OptionValue["type"])
> [Error Id: 9bad34d9-fe21-47fd-ade1-fbee3d5111e5 on ucs-node7.perf.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
> at
> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:561)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1895)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:473)
> at
> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1100)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:477)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:181)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:109)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:130)
> at
> org.apache.drill.jdbc.impl.DrillStatementImpl.executeQuery(DrillStatementImpl.java:112)
> at PipSQueak.executeQuery(PipSQueak.java:289)
> at PipSQueak.runTest(PipSQueak.java:104)
> at PipSQueak.main(PipSQueak.java:477)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: UnrecognizedPropertyException: Unrecognized field "type" (class
> org.apache.drill.exec.server.options.OptionValue), not marked as ignorable (8
> known properties: "string_val", "kind", "accessibleScopes", "num_val",
> "name", "bool_val", "float_val", "scope"])
> at [Source: [B@4ec364a3; line: 6, column: 2] (through reference chain:
> org.apache.drill.exec.server.options.OptionValue["type"])
> {code}
> Looks like the issue was introduced in commit:
> 6adeb986016a769755fd5e8fc66244ee1e8d18e1
> DRILL-5723: Added System Internal Options That can be Modified at Run…
> The commit prior to this one did not show the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5809) Queries failed with UnrecognizedPropertyException
[
https://issues.apache.org/jira/browse/DRILL-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175469#comment-16175469
]
Dechang Gu commented on DRILL-5809:
---
[~timothyfarkas] You don't need to run tpch query, even the following repro
the issue:
{code}
0: jdbc:drill:schema=dfs.default> show databases;
Error: SYSTEM ERROR: UnrecognizedPropertyException: Unrecognized field "type"
(class org.apache.drill.exec.server.options.OptionValue), not marked as
ignorable (8 known properties: "string_val", "kind", "accessibleScopes",
"num_val", "name", "bool_val", "float_val", "scope"])
at [Source: [B@28ea6afd; line: 6, column: 2] (through reference chain:
org.apache.drill.exec.server.options.OptionValue["type"])
[Error Id: a816531b-8685-4589-aff2-4f04534300b3 on ucs-node7.perf.lab:31010]
(state=,code=0)
{code}
> Queries failed with UnrecognizedPropertyException
> -
>
> Key: DRILL-5809
> URL: https://issues.apache.org/jira/browse/DRILL-5809
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.12.0
> Environment: RHEL 6, 2.6.32-358.el6.x86_64, Apache Drill gitid
> 4c99f0cdd98b1a0e789c5d11b353f88a2f6d54aa
>Reporter: Dechang Gu
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.12.0
>
>
> Run tpch queries, all failed due to the following error:
> {code}
> java.sql.SQLException: SYSTEM ERROR: UnrecognizedPropertyException:
> Unrecognized field "type" (class
> org.apache.drill.exec.server.options.OptionValue), not marked as ignorable (8
> known properties: "string_val", "kind", "accessibleScopes", "num_val",
> "name", "bool_val", "float_val", "scope"])
> at [Source: [B@4ec364a3; line: 6, column: 2] (through reference chain:
> org.apache.drill.exec.server.options.OptionValue["type"])
> [Error Id: 9bad34d9-fe21-47fd-ade1-fbee3d5111e5 on ucs-node7.perf.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
> at
> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:561)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1895)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:473)
> at
> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1100)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:477)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:181)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:109)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:130)
> at
> org.apache.drill.jdbc.impl.DrillStatementImpl.executeQuery(DrillStatementImpl.java:112)
> at PipSQueak.executeQuery(PipSQueak.java:289)
> at PipSQueak.runTest(PipSQueak.java:104)
> at PipSQueak.main(PipSQueak.java:477)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: UnrecognizedPropertyException: Unrecognized field "type" (class
> org.apache.drill.exec.server.options.OptionValue), not marked as ignorable (8
> known properties: "string_val", "kind", "accessibleScopes", "num_val",
> "name", "bool_val", "float_val", "scope"])
> at [Source: [B@4ec364a3; line: 6, column: 2] (through reference chain:
> org.apache.drill.exec.server.options.OptionValue["type"])
> {code}
> Looks like the issue was introduced in commit:
> 6adeb986016a769755fd5e8fc66244ee1e8d18e1
> DRILL-5723: Added System Internal Options That can be Modified at Run…
> The commit prior to this one did not show the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (DRILL-5809) Queries failed with UnrecognizedPropertyException
[
https://issues.apache.org/jira/browse/DRILL-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu updated DRILL-5809:
--
Priority: Blocker (was: Major)
> Queries failed with UnrecognizedPropertyException
> -
>
> Key: DRILL-5809
> URL: https://issues.apache.org/jira/browse/DRILL-5809
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.12.0
> Environment: RHEL 6, 2.6.32-358.el6.x86_64, Apache Drill gitid
> 4c99f0cdd98b1a0e789c5d11b353f88a2f6d54aa
>Reporter: Dechang Gu
>Priority: Blocker
> Fix For: 1.12.0
>
>
> Run tpch queries, all failed due to the following error:
> {code}
> java.sql.SQLException: SYSTEM ERROR: UnrecognizedPropertyException:
> Unrecognized field "type" (class
> org.apache.drill.exec.server.options.OptionValue), not marked as ignorable (8
> known properties: "string_val", "kind", "accessibleScopes", "num_val",
> "name", "bool_val", "float_val", "scope"])
> at [Source: [B@4ec364a3; line: 6, column: 2] (through reference chain:
> org.apache.drill.exec.server.options.OptionValue["type"])
> [Error Id: 9bad34d9-fe21-47fd-ade1-fbee3d5111e5 on ucs-node7.perf.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
> at
> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:561)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1895)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:473)
> at
> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1100)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:477)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:181)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:109)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:130)
> at
> org.apache.drill.jdbc.impl.DrillStatementImpl.executeQuery(DrillStatementImpl.java:112)
> at PipSQueak.executeQuery(PipSQueak.java:289)
> at PipSQueak.runTest(PipSQueak.java:104)
> at PipSQueak.main(PipSQueak.java:477)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: UnrecognizedPropertyException: Unrecognized field "type" (class
> org.apache.drill.exec.server.options.OptionValue), not marked as ignorable (8
> known properties: "string_val", "kind", "accessibleScopes", "num_val",
> "name", "bool_val", "float_val", "scope"])
> at [Source: [B@4ec364a3; line: 6, column: 2] (through reference chain:
> org.apache.drill.exec.server.options.OptionValue["type"])
> {code}
> Looks like the issue was introduced in commit:
> 6adeb986016a769755fd5e8fc66244ee1e8d18e1
> DRILL-5723: Added System Internal Options That can be Modified at Run…
> The commit prior to this one did not show the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (DRILL-5809) Queries failed with UnrecognizedPropertyException
[
https://issues.apache.org/jira/browse/DRILL-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu reassigned DRILL-5809:
-
Assignee: Boaz Ben-Zvi
> Queries failed with UnrecognizedPropertyException
> -
>
> Key: DRILL-5809
> URL: https://issues.apache.org/jira/browse/DRILL-5809
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.12.0
> Environment: RHEL 6, 2.6.32-358.el6.x86_64, Apache Drill gitid
> 4c99f0cdd98b1a0e789c5d11b353f88a2f6d54aa
>Reporter: Dechang Gu
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.12.0
>
>
> Run tpch queries, all failed due to the following error:
> {code}
> java.sql.SQLException: SYSTEM ERROR: UnrecognizedPropertyException:
> Unrecognized field "type" (class
> org.apache.drill.exec.server.options.OptionValue), not marked as ignorable (8
> known properties: "string_val", "kind", "accessibleScopes", "num_val",
> "name", "bool_val", "float_val", "scope"])
> at [Source: [B@4ec364a3; line: 6, column: 2] (through reference chain:
> org.apache.drill.exec.server.options.OptionValue["type"])
> [Error Id: 9bad34d9-fe21-47fd-ade1-fbee3d5111e5 on ucs-node7.perf.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
> at
> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:561)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1895)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:473)
> at
> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1100)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:477)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:181)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:109)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:130)
> at
> org.apache.drill.jdbc.impl.DrillStatementImpl.executeQuery(DrillStatementImpl.java:112)
> at PipSQueak.executeQuery(PipSQueak.java:289)
> at PipSQueak.runTest(PipSQueak.java:104)
> at PipSQueak.main(PipSQueak.java:477)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: UnrecognizedPropertyException: Unrecognized field "type" (class
> org.apache.drill.exec.server.options.OptionValue), not marked as ignorable (8
> known properties: "string_val", "kind", "accessibleScopes", "num_val",
> "name", "bool_val", "float_val", "scope"])
> at [Source: [B@4ec364a3; line: 6, column: 2] (through reference chain:
> org.apache.drill.exec.server.options.OptionValue["type"])
> {code}
> Looks like the issue was introduced in commit:
> 6adeb986016a769755fd5e8fc66244ee1e8d18e1
> DRILL-5723: Added System Internal Options That can be Modified at Run…
> The commit prior to this one did not show the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5809) Queries failed with UnrecognizedPropertyException
Dechang Gu created DRILL-5809:
-
Summary: Queries failed with UnrecognizedPropertyException
Key: DRILL-5809
URL: https://issues.apache.org/jira/browse/DRILL-5809
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 1.12.0
Environment: RHEL 6, 2.6.32-358.el6.x86_64, Apache Drill gitid
4c99f0cdd98b1a0e789c5d11b353f88a2f6d54aa
Reporter: Dechang Gu
Fix For: 1.12.0
Run tpch queries, all failed due to the following error:
{code}
java.sql.SQLException: SYSTEM ERROR: UnrecognizedPropertyException:
Unrecognized field "type" (class
org.apache.drill.exec.server.options.OptionValue), not marked as ignorable (8
known properties: "string_val", "kind", "accessibleScopes", "num_val", "name",
"bool_val", "float_val", "scope"])
at [Source: [B@4ec364a3; line: 6, column: 2] (through reference chain:
org.apache.drill.exec.server.options.OptionValue["type"])
[Error Id: 9bad34d9-fe21-47fd-ade1-fbee3d5111e5 on ucs-node7.perf.lab:31010]
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
at
org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:561)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1895)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
at
org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:473)
at
org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1100)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:477)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:181)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:109)
at
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:130)
at
org.apache.drill.jdbc.impl.DrillStatementImpl.executeQuery(DrillStatementImpl.java:112)
at PipSQueak.executeQuery(PipSQueak.java:289)
at PipSQueak.runTest(PipSQueak.java:104)
at PipSQueak.main(PipSQueak.java:477)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
ERROR: UnrecognizedPropertyException: Unrecognized field "type" (class
org.apache.drill.exec.server.options.OptionValue), not marked as ignorable (8
known properties: "string_val", "kind", "accessibleScopes", "num_val", "name",
"bool_val", "float_val", "scope"])
at [Source: [B@4ec364a3; line: 6, column: 2] (through reference chain:
org.apache.drill.exec.server.options.OptionValue["type"])
{code}
Looks like the issue was introduced in commit:
6adeb986016a769755fd5e8fc66244ee1e8d18e1
DRILL-5723: Added System Internal Options That can be Modified at Run…
The commit prior to this one did not show the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (DRILL-5669) Multiple TPCH queries failed due to OOM
[
https://issues.apache.org/jira/browse/DRILL-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu updated DRILL-5669:
--
Attachment: 26999476-174e-98fd-e21e-fd53f79284c7.sys.drill
> Multiple TPCH queries failed due to OOM
> ---
>
> Key: DRILL-5669
> URL: https://issues.apache.org/jira/browse/DRILL-5669
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
> Environment: RHEL 6.4 2.6.32-358.el6.x86_64, 10+1 nodes cluster
>Reporter: Dechang Gu
>Assignee: Boaz Ben-Zvi
> Fix For: 1.11.0
>
> Attachments: 26999476-174e-98fd-e21e-fd53f79284c7.sys.drill
>
>
> Running TPCH SF100 Parquet (and CSV) tests, multiple queries failed due to
> OOM. For example, Q16 hit the following error:
> {code}
> java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory
> while executing the query.
> Unable to allocate sv2 for 65536 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 23500416
> allocator limit 2000
> Fragment 1:11
> [Error Id: e58161a6-2383-48b1-a350-50db1b5408c6 on ucs-node10.perf.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
> at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:593)
> at
> org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:215)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:140)
> at PipSQueak.fetchRows(PipSQueak.java:420)
> at PipSQueak.runTest(PipSQueak.java:116)
> at PipSQueak.main(PipSQueak.java:556)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: RESOURCE
> ERROR: One or more nodes ran out of memory while executing the query.
> Unable to allocate sv2 for 65536 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 23500416
> allocator limit 2000
> Fragment 1:11
> {code}
> And in drillbit.log:
> {code}
> 2017-07-12 11:34:11,670 ucs-node10.perf.lab
> [26999476-174e-98fd-e21e-fd53f79284c7:frag:1:11] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - User Error Occurred: One or more nodes
> ran out of memory while executing the query.
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more
> nodes ran out of memory while executing the query.
> Unable to allocate sv2 for 65536 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 23500416
> allocator limit 2000
> [Error Id: e58161a6-2383-48b1-a350-50db1b5408c6 ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.newSV2(ExternalSortBatch.java:639)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:381)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext(StreamingAggBatch.java:140)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext(PartitionSenderRootExec.java:144)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
[jira] [Commented] (DRILL-5669) Multiple TPCH queries failed due to OOM
[
https://issues.apache.org/jira/browse/DRILL-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084881#comment-16084881
]
Dechang Gu commented on DRILL-5669:
---
And the profile is uploaded:
12876973/26999476-174e-98fd-e21e-fd53f79284c7.sys.drill
> Multiple TPCH queries failed due to OOM
> ---
>
> Key: DRILL-5669
> URL: https://issues.apache.org/jira/browse/DRILL-5669
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
> Environment: RHEL 6.4 2.6.32-358.el6.x86_64, 10+1 nodes cluster
>Reporter: Dechang Gu
>Assignee: Boaz Ben-Zvi
> Fix For: 1.11.0
>
> Attachments: 26999476-174e-98fd-e21e-fd53f79284c7.sys.drill
>
>
> Running TPCH SF100 Parquet (and CSV) tests, multiple queries failed due to
> OOM. For example, Q16 hit the following error:
> {code}
> java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory
> while executing the query.
> Unable to allocate sv2 for 65536 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 23500416
> allocator limit 2000
> Fragment 1:11
> [Error Id: e58161a6-2383-48b1-a350-50db1b5408c6 on ucs-node10.perf.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
> at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:593)
> at
> org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:215)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:140)
> at PipSQueak.fetchRows(PipSQueak.java:420)
> at PipSQueak.runTest(PipSQueak.java:116)
> at PipSQueak.main(PipSQueak.java:556)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: RESOURCE
> ERROR: One or more nodes ran out of memory while executing the query.
> Unable to allocate sv2 for 65536 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 23500416
> allocator limit 2000
> Fragment 1:11
> {code}
> And in drillbit.log:
> {code}
> 2017-07-12 11:34:11,670 ucs-node10.perf.lab
> [26999476-174e-98fd-e21e-fd53f79284c7:frag:1:11] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - User Error Occurred: One or more nodes
> ran out of memory while executing the query.
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more
> nodes ran out of memory while executing the query.
> Unable to allocate sv2 for 65536 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 23500416
> allocator limit 2000
> [Error Id: e58161a6-2383-48b1-a350-50db1b5408c6 ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.newSV2(ExternalSortBatch.java:639)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:381)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext(StreamingAggBatch.java:140)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext(PartitionSenderRootExec.java:144)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227)
>
[jira] [Commented] (DRILL-5669) Multiple TPCH queries failed due to OOM
[
https://issues.apache.org/jira/browse/DRILL-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084877#comment-16084877
]
Dechang Gu commented on DRILL-5669:
---
Here is the plan for the failed query:
{code}
00-00Screen : rowType = RecordType(ANY p_brand, ANY p_type, ANY p_size,
BIGINT supplier_cnt): rowcount = 2000.0, cumulative cost = {4.16474202E8 rows,
3.783967028483349E9 cpu, 0.0 io, 1.9509256192E11 network, 9.9034801E8
memory}, id = 41734
00-01 Project(p_brand=[$0], p_type=[$1], p_size=[$2], supplier_cnt=[$3]) :
rowType = RecordType(ANY p_brand, ANY p_type, ANY p_size, BIGINT supplier_cnt):
rowcount = 2000.0, cumulative cost = {4.16474002E8 rows, 3.783966828483349E9
cpu, 0.0 io, 1.9509256192E11 network, 9.9034801E8 memory}, id = 41733
00-02SelectionVectorRemover : rowType = RecordType(ANY p_brand, ANY
p_type, ANY p_size, BIGINT supplier_cnt): rowcount = 2000.0, cumulative cost =
{4.16474002E8 rows, 3.783966828483349E9 cpu, 0.0 io, 1.9509256192E11 network,
9.9034801E8 memory}, id = 41732
00-03 Sort(sort0=[$3], sort1=[$0], sort2=[$1], sort3=[$2],
dir0=[DESC], dir1=[ASC], dir2=[ASC], dir3=[ASC]) : rowType = RecordType(ANY
p_brand, ANY p_type, ANY p_size, BIGINT supplier_cnt): rowcount = 2000.0,
cumulative cost = {4.16472002E8 rows, 3.783964828483349E9 cpu, 0.0 io,
1.9509256192E11 network, 9.9034801E8 memory}, id = 41731
00-04StreamAgg(group=[{0, 1, 2}], supplier_cnt=[$SUM0($3)]) :
rowType = RecordType(ANY p_brand, ANY p_type, ANY p_size, BIGINT supplier_cnt):
rowcount = 2000.0, cumulative cost = {4.16470002E8 rows, 3.7836139233862395E9
cpu, 0.0 io, 1.9509256192E11 network, 9.9028401E8 memory}, id = 41730
00-05 HashToMergeExchange : rowType = RecordType(ANY p_brand, ANY
p_type, ANY p_size, BIGINT supplier_cnt): rowcount = 2.0, cumulative cost =
{4.16450002E8 rows, 3.7831339233862395E9 cpu, 0.0 io, 1.9509256192E11 network,
9.9028401E8 memory}, id = 41729
01-01StreamAgg(group=[{0, 1, 2}], supplier_cnt=[COUNT($3)]) :
rowType = RecordType(ANY p_brand, ANY p_type, ANY p_size, BIGINT supplier_cnt):
rowcount = 2.0, cumulative cost = {4.16430002E8 rows, 3.7822281691386485E9
cpu, 0.0 io, 1.9476488192E11 network, 9.9028401E8 memory}, id = 41728
01-02 Sort(sort0=[$0], sort1=[$1], sort2=[$2], dir0=[ASC],
dir1=[ASC], dir2=[ASC]) : rowType = RecordType(ANY p_brand, ANY p_type, ANY
p_size, ANY ps_suppkey): rowcount = 20.0, cumulative cost = {4.16230002E8
rows, 3.7774281691386485E9 cpu, 0.0 io, 1.9476488192E11 network,
9.9028401E8 memory}, id = 41727
01-03HashAgg(group=[{0, 1, 2, 3}]) : rowType =
RecordType(ANY p_brand, ANY p_type, ANY p_size, ANY ps_suppkey): rowcount =
20.0, cumulative cost = {4.16030002E8 rows, 3.735165032E9 cpu, 0.0 io,
1.9476488192E11 network, 9.8388401E8 memory}, id = 41726
01-04 Project(p_brand=[$0], p_type=[$1], p_size=[$2],
ps_suppkey=[$3]) : rowType = RecordType(ANY p_brand, ANY p_type, ANY p_size,
ANY ps_suppkey): rowcount = 200.0, cumulative cost = {4.14030002E8 rows,
3.671165032E9 cpu, 0.0 io, 1.9476488192E11 network, 8.9588401E8
memory}, id = 41725
01-05HashToRandomExchange(dist0=[[$0]], dist1=[[$1]],
dist2=[[$2]], dist3=[[$3]]) : rowType = RecordType(ANY p_brand, ANY p_type, ANY
p_size, ANY ps_suppkey, ANY E_X_P_R_H_A_S_H_F_I_E_L_D): rowcount = 200.0,
cumulative cost = {4.14030002E8 rows, 3.671165032E9 cpu, 0.0 io,
1.9476488192E11 network, 8.9588401E8 memory}, id = 41724
02-01 UnorderedMuxExchange : rowType = RecordType(ANY
p_brand, ANY p_type, ANY p_size, ANY ps_suppkey, ANY
E_X_P_R_H_A_S_H_F_I_E_L_D): rowcount = 200.0, cumulative cost =
{4.12030002E8 rows, 3.651165032E9 cpu, 0.0 io, 1.5380488192E11 network,
8.9588401E8 memory}, id = 41723
03-01Project(p_brand=[$0], p_type=[$1],
p_size=[$2], ps_suppkey=[$3], E_X_P_R_H_A_S_H_F_I_E_L_D=[hash32AsDouble($3,
hash32AsDouble($2, hash32AsDouble($1, hash32AsDouble($0, 1301011]) :
rowType = RecordType(ANY p_brand, ANY p_type, ANY p_size, ANY ps_suppkey, ANY
E_X_P_R_H_A_S_H_F_I_E_L_D): rowcount = 200.0, cumulative cost =
{4.10030002E8 rows, 3.649165032E9 cpu, 0.0 io, 1.5380488192E11 network,
8.9588401E8 memory}, id = 41722
03-02 HashAgg(group=[{0, 1, 2, 3}]) : rowType =
RecordType(ANY p_brand, ANY p_type, ANY p_size, ANY ps_suppkey): rowcount =
200.0, cumulative cost = {4.08030002E8 rows, 3.641165032E9 cpu, 0.0 io,
1.5380488192E11 network, 8.9588401E8 memory}, id = 41721
03-03Project(p_brand=[$2], p_type=[$3],
p_size=[$4], ps_suppkey=[$1]) : rowType = RecordType(ANY p_brand, ANY p_type,
ANY p_size, ANY ps_suppkey):
[jira] [Commented] (DRILL-5669) Multiple TPCH queries failed due to OOM
[
https://issues.apache.org/jira/browse/DRILL-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084506#comment-16084506
]
Dechang Gu commented on DRILL-5669:
---
I have narrowed down to the commit c16e5f8072f3e5d18157767143f9ccc7669c4380 for
DRILL-5457: Spill implementation for Hash Aggregate. The commit
be43a9edd148ef3af6f92c5ce7cda235c5ac1ad6 doe not show the issues.
> Multiple TPCH queries failed due to OOM
> ---
>
> Key: DRILL-5669
> URL: https://issues.apache.org/jira/browse/DRILL-5669
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
> Environment: RHEL 6.4 2.6.32-358.el6.x86_64, 10+1 nodes cluster
>Reporter: Dechang Gu
>Assignee: Boaz Ben-Zvi
> Fix For: 1.11.0
>
>
> Running TPCH SF100 Parquet (and CSV) tests, multiple queries failed due to
> OOM. For example, Q16 hit the following error:
> {code}
> java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory
> while executing the query.
> Unable to allocate sv2 for 65536 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 23500416
> allocator limit 2000
> Fragment 1:11
> [Error Id: e58161a6-2383-48b1-a350-50db1b5408c6 on ucs-node10.perf.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
> at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:593)
> at
> org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:215)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:140)
> at PipSQueak.fetchRows(PipSQueak.java:420)
> at PipSQueak.runTest(PipSQueak.java:116)
> at PipSQueak.main(PipSQueak.java:556)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: RESOURCE
> ERROR: One or more nodes ran out of memory while executing the query.
> Unable to allocate sv2 for 65536 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 23500416
> allocator limit 2000
> Fragment 1:11
> {code}
> And in drillbit.log:
> {code}
> 2017-07-12 11:34:11,670 ucs-node10.perf.lab
> [26999476-174e-98fd-e21e-fd53f79284c7:frag:1:11] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - User Error Occurred: One or more nodes
> ran out of memory while executing the query.
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more
> nodes ran out of memory while executing the query.
> Unable to allocate sv2 for 65536 records, and not enough batchGroups to spill.
> batchGroups.size 1
> spilledBatchGroups.size 0
> allocated memory 23500416
> allocator limit 2000
> [Error Id: e58161a6-2383-48b1-a350-50db1b5408c6 ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.newSV2(ExternalSortBatch.java:639)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:381)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext(StreamingAggBatch.java:140)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext(PartitionSenderRootExec.java:144)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234)
> [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
>
[jira] [Created] (DRILL-5669) Multiple TPCH queries failed due to OOM
Dechang Gu created DRILL-5669:
-
Summary: Multiple TPCH queries failed due to OOM
Key: DRILL-5669
URL: https://issues.apache.org/jira/browse/DRILL-5669
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Environment: RHEL 6.4 2.6.32-358.el6.x86_64, 10+1 nodes cluster
Reporter: Dechang Gu
Assignee: Boaz Ben-Zvi
Fix For: 1.11.0
Running TPCH SF100 Parquet (and CSV) tests, multiple queries failed due to OOM.
For example, Q16 hit the following error:
{code}
java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory
while executing the query.
Unable to allocate sv2 for 65536 records, and not enough batchGroups to spill.
batchGroups.size 1
spilledBatchGroups.size 0
allocated memory 23500416
allocator limit 2000
Fragment 1:11
[Error Id: e58161a6-2383-48b1-a350-50db1b5408c6 on ucs-node10.perf.lab:31010]
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:593)
at
org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:215)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:140)
at PipSQueak.fetchRows(PipSQueak.java:420)
at PipSQueak.runTest(PipSQueak.java:116)
at PipSQueak.main(PipSQueak.java:556)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: RESOURCE
ERROR: One or more nodes ran out of memory while executing the query.
Unable to allocate sv2 for 65536 records, and not enough batchGroups to spill.
batchGroups.size 1
spilledBatchGroups.size 0
allocated memory 23500416
allocator limit 2000
Fragment 1:11
{code}
And in drillbit.log:
{code}
2017-07-12 11:34:11,670 ucs-node10.perf.lab
[26999476-174e-98fd-e21e-fd53f79284c7:frag:1:11] INFO
o.a.d.e.p.i.xsort.ExternalSortBatch - User Error Occurred: One or more nodes
ran out of memory while executing the query.
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more
nodes ran out of memory while executing the query.
Unable to allocate sv2 for 65536 records, and not enough batchGroups to spill.
batchGroups.size 1
spilledBatchGroups.size 0
allocated memory 23500416
allocator limit 2000
[Error Id: e58161a6-2383-48b1-a350-50db1b5408c6 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.newSV2(ExternalSortBatch.java:639)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:381)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext(StreamingAggBatch.java:140)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext(PartitionSenderRootExec.java:144)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method)
[na:1.7.0_65]
at javax.security.auth.Subject.doAs(Subject.java:415) [na:1.7.0_65]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
[hadoop-common-2.7.0-mapr-1607.jar:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:227)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
[jira] [Updated] (DRILL-5532) TPCH Query 16 failed when planner.width.max_per_node <10
[
https://issues.apache.org/jira/browse/DRILL-5532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu updated DRILL-5532:
--
Attachment:
profile_tpch16_gitid_adbf363.26dce6f3-3d75-e906-bfcb-2c6541ee2ff0.sys.drill
pfofile_tpch16_gitid_8412ad4.26dce567-5ec0-8bbf-c099-f2d6307bdeef.sys.drill
> TPCH Query 16 failed when planner.width.max_per_node <10
>
>
> Key: DRILL-5532
> URL: https://issues.apache.org/jira/browse/DRILL-5532
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.11.0
> Environment: 10 node cluster, rhel6.4
>Reporter: Dechang Gu
>Assignee: Jinfeng Ni
> Attachments:
> pfofile_tpch16_gitid_8412ad4.26dce567-5ec0-8bbf-c099-f2d6307bdeef.sys.drill,
> profile_tpch16_gitid_adbf363.26dce6f3-3d75-e906-bfcb-2c6541ee2ff0.sys.drill
>
>
> When set planner.width.max_per_node < 10 (tried 6, 8, and 9): TPCH query 16
> failed with the following error:
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 1:1
> [Error Id: ba4661b4-2ff6-4eff-889c-fc4604a57418 on ucs-node8.perf.lab:31010]
> (java.lang.NullPointerException) null
> org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe():607
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe():560
>
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputRecordKeysPrev():525
>
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputToBatchPrev():322
>
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.doWork():182
>
> org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():170
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
>
> org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():144
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
> at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:593)
> at
> org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:215)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:140)
> at PipSQueak.fetchRows(PipSQueak.java:420)
> at PipSQueak.runTest(PipSQueak.java:116)
> at PipSQueak.main(PipSQueak.java:556)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: NullPointerException
> Fragment 1:1
> [Error Id: ba4661b4-2ff6-4eff-889c-fc4604a57418 on ucs-node8.perf.lab:31010]
> (java.lang.NullPointerException) null
> org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe():607
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe():560
>
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputRecordKeysPrev():525
>
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputToBatchPrev():322
>
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.doWork():182
>
> org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():170
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
>
> org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():144
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> at
>
[jira] [Commented] (DRILL-5532) TPCH Query 16 failed when planner.width.max_per_node <10
[
https://issues.apache.org/jira/browse/DRILL-5532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019987#comment-16019987
]
Dechang Gu commented on DRILL-5532:
---
set planner.width.max_per_node >=10, the query run correctly.
Looks like the issue was introduced in Apache Drill 1.11.0 master branch commit
id: 841ead40109ff8364bee640a77881a8fea94d152
https://github.com/apache/drill/commit/841ead40109ff8364bee640a77881a8fea94d152
I did bisect on the commits, and singled out this one.
> TPCH Query 16 failed when planner.width.max_per_node <10
>
>
> Key: DRILL-5532
> URL: https://issues.apache.org/jira/browse/DRILL-5532
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.11.0
> Environment: 10 node cluster, rhel6.4
>Reporter: Dechang Gu
>Assignee: Jinfeng Ni
>
> When set planner.width.max_per_node < 10 (tried 6, 8, and 9): TPCH query 16
> failed with the following error:
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 1:1
> [Error Id: ba4661b4-2ff6-4eff-889c-fc4604a57418 on ucs-node8.perf.lab:31010]
> (java.lang.NullPointerException) null
> org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe():607
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe():560
>
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputRecordKeysPrev():525
>
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputToBatchPrev():322
>
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.doWork():182
>
> org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():170
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
>
> org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():144
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
> at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:593)
> at
> org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:215)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:140)
> at PipSQueak.fetchRows(PipSQueak.java:420)
> at PipSQueak.runTest(PipSQueak.java:116)
> at PipSQueak.main(PipSQueak.java:556)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: NullPointerException
> Fragment 1:1
> [Error Id: ba4661b4-2ff6-4eff-889c-fc4604a57418 on ucs-node8.perf.lab:31010]
> (java.lang.NullPointerException) null
> org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe():607
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe():560
>
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputRecordKeysPrev():525
>
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputToBatchPrev():322
>
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.doWork():182
>
> org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():170
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
>
> org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():144
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> at
>
[jira] [Created] (DRILL-5532) TPCH Query 16 failed when planner.width.max_per_node <10
Dechang Gu created DRILL-5532:
-
Summary: TPCH Query 16 failed when planner.width.max_per_node <10
Key: DRILL-5532
URL: https://issues.apache.org/jira/browse/DRILL-5532
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 1.11.0
Environment: 10 node cluster, rhel6.4
Reporter: Dechang Gu
Assignee: Jinfeng Ni
When set planner.width.max_per_node < 10 (tried 6, 8, and 9): TPCH query 16
failed with the following error:
{code}
java.sql.SQLException: SYSTEM ERROR: NullPointerException
Fragment 1:1
[Error Id: ba4661b4-2ff6-4eff-889c-fc4604a57418 on ucs-node8.perf.lab:31010]
(java.lang.NullPointerException) null
org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe():607
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe():560
org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputRecordKeysPrev():525
org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputToBatchPrev():322
org.apache.drill.exec.test.generated.StreamingAggregatorGen250.doWork():182
org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():170
org.apache.drill.exec.record.AbstractRecordBatch.next():162
org.apache.drill.exec.physical.impl.BaseRootExec.next():105
org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():144
org.apache.drill.exec.physical.impl.BaseRootExec.next():95
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():415
org.apache.hadoop.security.UserGroupInformation.doAs():1595
org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:593)
at
org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:215)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:140)
at PipSQueak.fetchRows(PipSQueak.java:420)
at PipSQueak.runTest(PipSQueak.java:116)
at PipSQueak.main(PipSQueak.java:556)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
ERROR: NullPointerException
Fragment 1:1
[Error Id: ba4661b4-2ff6-4eff-889c-fc4604a57418 on ucs-node8.perf.lab:31010]
(java.lang.NullPointerException) null
org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe():607
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe():560
org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputRecordKeysPrev():525
org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputToBatchPrev():322
org.apache.drill.exec.test.generated.StreamingAggregatorGen250.doWork():182
org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():170
org.apache.drill.exec.record.AbstractRecordBatch.next():162
org.apache.drill.exec.physical.impl.BaseRootExec.next():105
org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():144
org.apache.drill.exec.physical.impl.BaseRootExec.next():95
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():415
org.apache.hadoop.security.UserGroupInformation.doAs():1595
org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
at
org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:343)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:88)
at
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)
at
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at
[jira] [Closed] (DRILL-5266) Parquet Reader produces "low density" record batches - bits vs. bytes
[
https://issues.apache.org/jira/browse/DRILL-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu closed DRILL-5266.
-
verified. see closure comment in DRILL-5267
> Parquet Reader produces "low density" record batches - bits vs. bytes
> -
>
> Key: DRILL-5266
> URL: https://issues.apache.org/jira/browse/DRILL-5266
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Testing with the managed sort revealed that, for at least one file, Parquet
> produces "low-density" batches: batches in which only 5% of each value vector
> contains actual data, with the rest being unused space. When fed into the
> sort, we end up buffering 95% of wasted space, using only 5% of available
> memory to hold actual query data. The result is poor performance of the sort
> as it must spill far more frequently than expected.
> The managed sort analyzes incoming batches to prepare good memory use
> estimates. The following the the output from the Parquet file in question:
> {code}
> Actual batch schema & sizes {
> T1¦¦cs_sold_date_sk(std col. size: 4, actual col. size: 4, total size:
> 196608, vector size: 131072, data size: 4516, row capacity: 32768, density: 4)
> T1¦¦cs_sold_time_sk(std col. size: 4, actual col. size: 4, total size:
> 196608, vector size: 131072, data size: 4516, row capacity: 32768, density: 4)
> T1¦¦cs_ship_date_sk(std col. size: 4, actual col. size: 4, total size:
> 196608, vector size: 131072, data size: 4516, row capacity: 32768, density: 4)
> ...
> c_email_address(std col. size: 54, actual col. size: 27, total size: 53248,
> vector size: 49152, data size: 30327, row capacity: 4095, density: 62)
> Records: 1129, Total size: 32006144, Row width:28350, Density:5}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (DRILL-5267) Managed external sort spills too often with Parquet data
[ https://issues.apache.org/jira/browse/DRILL-5267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu closed DRILL-5267. - Verified and added test case to perf framework. Here are the numbers: Drill version (gitid) Query tm (ms) Max Batches note 41ffed5 44596 159 1.11.0 current master 8cded5a 37869 159 commit with the fix 33fc25c 77857 1271commit prior the fix 33fc25c 1277695 1271without setting "sort.external.disable_managed:false" in drill-override.conf > Managed external sort spills too often with Parquet data > > > Key: DRILL-5267 > URL: https://issues.apache.org/jira/browse/DRILL-5267 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.10.0 >Reporter: Paul Rogers >Assignee: Paul Rogers > Fix For: 1.10.0 > > > DRILL-5266 describes how Parquet produces low-density record batches. The > result of these batches is that the external sort spills more frequently than > it should because it sizes spill files based on batch size, not data content > of the batch. Since Parquet batches are 95% empty space, the spill files end > up far too small. > Adjust the spill calculations based on actual data content, not the size of > the overall record batch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-4982) Hive Queries degrade when queries switch between different formats
[
https://issues.apache.org/jira/browse/DRILL-4982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu closed DRILL-4982.
-
Verified and added testcase to perf test framework.
> Hive Queries degrade when queries switch between different formats
> --
>
> Key: DRILL-4982
> URL: https://issues.apache.org/jira/browse/DRILL-4982
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Chunhui Shi
>Assignee: Karthikeyan Manivannan
>Priority: Critical
> Fix For: 1.10.0
>
>
> We have seen degraded performance by doing these steps:
> 1) generate the repro data:
> python script repro.py as below:
> import string
> import random
>
> for i in range(3000):
> x1 = ''.join(random.choice(string.ascii_uppercase + string.digits) for _
> in range(random.randrange(19, 27)))
> x2 = ''.join(random.choice(string.ascii_uppercase + string.digits) for _
> in range(random.randrange(19, 27)))
> x3 = ''.join(random.choice(string.ascii_uppercase + string.digits) for _
> in range(random.randrange(19, 27)))
> x4 = ''.join(random.choice(string.ascii_uppercase + string.digits) for _
> in range(random.randrange(19, 27)))
> x5 = ''.join(random.choice(string.ascii_uppercase + string.digits) for _
> in range(random.randrange(19, 27)))
> x6 = ''.join(random.choice(string.ascii_uppercase + string.digits) for _
> in range(random.randrange(19, 27)))
> print
> "{0}".format(x1),"{0}".format(x2),"{0}".format(x3),"{0}".format(x4),"{0}".format(x5),"{0}".format(x6)
> python repro.py > repro.csv
> 2) put these files in a dfs directory e.g. '/tmp/hiveworkspace/plain'. Under
> hive prompt, use the following sql command to create an external table:
> CREATE EXTERNAL TABLE `hiveworkspace`.`plain` (`id1` string, `id2` string,
> `id3` string, `id4` string, `id5` string, `id6` string) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.OpenCSVSerde' STORED AS TEXTFILE LOCATION
> '/tmp/hiveworkspace/plain'
> 3) create Hive's table of ORC|PARQUET format:
> CREATE TABLE `hiveworkspace`.`plainorc` STORED AS ORC AS SELECT
> id1,id2,id3,id4,id5,id6 from `hiveworkspace`.`plain`;
> CREATE TABLE `hiveworkspace`.`plainparquet` STORED AS PARQUET AS SELECT
> id1,id2,id3,id4,id5,id6 from `hiveworkspace`.`plain`;
> 4) Query switch between these two tables, then the query time on the same
> table significantly lengthened. On my setup, for ORC, it was 15sec -> 26secs.
> Queries on table of other formats, after injecting a query to other formats,
> all have significant slow down.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (DRILL-5468) THCH Q18 regressed ~3x due to execution plan changes
[ https://issues.apache.org/jira/browse/DRILL-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu updated DRILL-5468: -- Attachment: Q18_profile_gitid_adbf363 Q18_profile_gitid_841ead4 > THCH Q18 regressed ~3x due to execution plan changes > > > Key: DRILL-5468 > URL: https://issues.apache.org/jira/browse/DRILL-5468 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.11.0 > Environment: 10+1 node ucs-micro cluster RHEL6.4 >Reporter: Dechang Gu >Assignee: Jinfeng Ni > Fix For: 1.11.0 > > Attachments: Q18_profile_gitid_841ead4, Q18_profile_gitid_adbf363 > > > In a regular regression test on Drill master (commit id 841ead4) TPCH Q18 on > SF100 parquet dataset took ~81 secs, while the same query on 1.10.0 took only > ~27 secs. The query time on the commit adbf363 which is right before 841ead4 > is ~32 secs. > Profiles shows the plans for the query changed quite a bit (profiles will be > uploaded) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5468) THCH Q18 regressed ~3x due to execution plan changes
Dechang Gu created DRILL-5468: - Summary: THCH Q18 regressed ~3x due to execution plan changes Key: DRILL-5468 URL: https://issues.apache.org/jira/browse/DRILL-5468 Project: Apache Drill Issue Type: Bug Components: Functions - Drill Affects Versions: 1.11.0 Environment: 10+1 node ucs-micro cluster RHEL6.4 Reporter: Dechang Gu Assignee: Jinfeng Ni Fix For: 1.11.0 In a regular regression test on Drill master (commit id 841ead4) TPCH Q18 on SF100 parquet dataset took ~81 secs, while the same query on 1.10.0 took only ~27 secs. The query time on the commit adbf363 which is right before 841ead4 is ~32 secs. Profiles shows the plans for the query changed quite a bit (profiles will be uploaded) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-4347) Planning time for query64 from TPCDS test suite has increased 10 times compared to 1.4 release
[
https://issues.apache.org/jira/browse/DRILL-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900247#comment-15900247
]
Dechang Gu commented on DRILL-4347:
---
Check it with the current AD1.10.0 master (gitid 3dfb497), it takes >4 minutes
for planning:
DURATION: 05 min 54.007 sec
PLANNING: 04 min 12.826 sec
EXECUTION: 01 min 41.181 sec
So someone need to chase the issue further
> Planning time for query64 from TPCDS test suite has increased 10 times
> compared to 1.4 release
> --
>
> Key: DRILL-4347
> URL: https://issues.apache.org/jira/browse/DRILL-4347
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.5.0
>Reporter: Victoria Markman
>Assignee: Aman Sinha
> Fix For: Future
>
> Attachments: 294e9fb9-cdda-a89f-d1a7-b852878926a1.sys.drill_1.4.0,
> 294ea418-9fb8-3082-1725-74e3cfe38fe9.sys.drill_1.5.0, drill4347_jstack.txt
>
>
> mapr-drill-1.5.0.201602012001-1.noarch.rpm
> {code}
> 0: jdbc:drill:schema=dfs> WITH cs_ui
> . . . . . . . . . . . . > AS (SELECT cs_item_sk,
> . . . . . . . . . . . . > Sum(cs_ext_list_price) AS sale,
> . . . . . . . . . . . . > Sum(cr_refunded_cash +
> cr_reversed_charge
> . . . . . . . . . . . . > + cr_store_credit) AS refund
> . . . . . . . . . . . . > FROM catalog_sales,
> . . . . . . . . . . . . > catalog_returns
> . . . . . . . . . . . . > WHERE cs_item_sk = cr_item_sk
> . . . . . . . . . . . . > AND cs_order_number =
> cr_order_number
> . . . . . . . . . . . . > GROUP BY cs_item_sk
> . . . . . . . . . . . . > HAVING Sum(cs_ext_list_price) > 2 * Sum(
> . . . . . . . . . . . . > cr_refunded_cash +
> cr_reversed_charge
> . . . . . . . . . . . . > + cr_store_credit)),
> . . . . . . . . . . . . > cross_sales
> . . . . . . . . . . . . > AS (SELECT i_product_name product_name,
> . . . . . . . . . . . . > i_item_sk item_sk,
> . . . . . . . . . . . . > s_store_name store_name,
> . . . . . . . . . . . . > s_zip store_zip,
> . . . . . . . . . . . . > ad1.ca_street_number
> b_street_number,
> . . . . . . . . . . . . > ad1.ca_street_name
> b_streen_name,
> . . . . . . . . . . . . > ad1.ca_cityb_city,
> . . . . . . . . . . . . > ad1.ca_zip b_zip,
> . . . . . . . . . . . . > ad2.ca_street_number
> c_street_number,
> . . . . . . . . . . . . > ad2.ca_street_name
> c_street_name,
> . . . . . . . . . . . . > ad2.ca_cityc_city,
> . . . . . . . . . . . . > ad2.ca_zip c_zip,
> . . . . . . . . . . . . > d1.d_year AS syear,
> . . . . . . . . . . . . > d2.d_year AS fsyear,
> . . . . . . . . . . . . > d3.d_year s2year,
> . . . . . . . . . . . . > Count(*) cnt,
> . . . . . . . . . . . . > Sum(ss_wholesale_cost) s1,
> . . . . . . . . . . . . > Sum(ss_list_price) s2,
> . . . . . . . . . . . . > Sum(ss_coupon_amt) s3
> . . . . . . . . . . . . > FROM store_sales,
> . . . . . . . . . . . . > store_returns,
> . . . . . . . . . . . . > cs_ui,
> . . . . . . . . . . . . > date_dim d1,
> . . . . . . . . . . . . > date_dim d2,
> . . . . . . . . . . . . > date_dim d3,
> . . . . . . . . . . . . > store,
> . . . . . . . . . . . . > customer,
> . . . . . . . . . . . . > customer_demographics cd1,
> . . . . . . . . . . . . > customer_demographics cd2,
> . . . . . . . . . . . . > promotion,
> . . . . . . . . . . . . > household_demographics hd1,
> . . . . . . . . . . . . > household_demographics hd2,
> . . . . . . . . . . . . > customer_address ad1,
> . . . . . . . . . . . . > customer_address ad2,
> . . . . . . . . . . . . > income_band ib1,
> . . . . . . . . . . . . > income_band ib2,
> . . . . . . . . . . . . > item
> . . . . . . . . . . . . > WHERE ss_store_sk = s_store_sk
> . . . . . . . . . . . . > AND ss_sold_date_sk = d1.d_date_sk
> . . . . . . . . . . . . > AND ss_customer_sk = c_customer_sk
> . . . . . . . . . . . . > AND
[jira] [Closed] (DRILL-4850) TPCDS Query 33 failed in the second and 3rd runs, but succeeded in the 1st run
[ https://issues.apache.org/jira/browse/DRILL-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu closed DRILL-4850. - Cannot repro in 1.10.0, so close it. > TPCDS Query 33 failed in the second and 3rd runs, but succeeded in the 1st run > -- > > Key: DRILL-4850 > URL: https://issues.apache.org/jira/browse/DRILL-4850 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.8.0 > Environment: REL 6.0 >Reporter: Dechang Gu >Assignee: Parth Chandra > Fix For: 1.10.0 > > > I run tpcds query 33 on SF100 database, 3 times consecutively. The first one > succeeded, but the 2nd and 3rd runs hit the following error: > > 2016-08-16 20:20:52,530 ucs-node6.perf.lab > [284c27f1-ee13-dfd0-6cbb-37f49810e93f:frag:3:9] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Failure while reading vector. Expected vector class of > org.apache.drill.exec.vector.NullableIntVector but was holding vector class > org.apache.drill.exec.vector.NullableBigIntVector, field= > i_manufact_id(BIGINT:OPTIONAL)[$bits$(UINT1:REQUIRED), > i_manufact_id(BIGINT:OPTIONAL)] > Fragment 3:9 > [Error Id: 7fc06ab9-6c63-402b-a1b4-465526aa7dc7 on ucs-node6.perf.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Failure while reading vector. Expected vector class > of org.apache.drill.exec.vector.NullableIntVector but was holding vector > class org.apache.drill.exec.vector.NullableBigIntVector, field= > i_manufact_id(BIGINT:OPTIONAL)[$bits$(UINT1:REQUIRED), > i_manufact_id(BIGINT:OPTIONAL)] > Fragment 3:9 > [Error Id: 7fc06ab9-6c63-402b-a1b4-465526aa7dc7 on ucs-node6.perf.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) > ~[drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_65] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] > Caused by: java.lang.IllegalStateException: Failure while reading vector. > Expected vector class of org.apache.drill.exec.vector.NullableIntVector but > was holding vector class org.apache.drill.exec.vector.NullableBigIntVector, > field= i_manufact_id(BIGINT:OPTIONAL)[$bits$(UINT1:REQUIRED), > i_manufact_id(BIGINT:OPTIONAL)] > at > org.apache.drill.exec.record.VectorContainer.getValueAccessorById(VectorContainer.java:290) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.RecordBatchLoader.getValueAccessorById(RecordBatchLoader.java:178) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.getValueAccessorById(UnorderedReceiverBatch.java:135) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.test.generated.PartitionerGen36655$OutgoingRecordBatch.doSetup(PartitionerTemplate.java:64) > ~[na:na] > at > org.apache.drill.exec.test.generated.PartitionerGen36655$OutgoingRecordBatch.initializeBatch(PartitionerTemplate.java:358) > ~[na:na] > at > org.apache.drill.exec.test.generated.PartitionerGen36655.flushOutgoingBatches(PartitionerTemplate.java:163) > ~[na:na] > at > org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator$FlushBatchesHandlingClass.execute(PartitionerDecorator.java:266) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator.executeMethodLogic(PartitionerDecorator.java:138) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator.flushOutgoingBatches(PartitionerDecorator.java:82) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at >
[jira] [Resolved] (DRILL-4850) TPCDS Query 33 failed in the second and 3rd runs, but succeeded in the 1st run
[ https://issues.apache.org/jira/browse/DRILL-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu resolved DRILL-4850. --- Resolution: Cannot Reproduce Fix Version/s: 1.10.0 Cannot repro in 1.10.0 (gitid: 3bfb497), hence close it. > TPCDS Query 33 failed in the second and 3rd runs, but succeeded in the 1st run > -- > > Key: DRILL-4850 > URL: https://issues.apache.org/jira/browse/DRILL-4850 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.8.0 > Environment: REL 6.0 >Reporter: Dechang Gu >Assignee: Parth Chandra > Fix For: 1.10.0 > > > I run tpcds query 33 on SF100 database, 3 times consecutively. The first one > succeeded, but the 2nd and 3rd runs hit the following error: > > 2016-08-16 20:20:52,530 ucs-node6.perf.lab > [284c27f1-ee13-dfd0-6cbb-37f49810e93f:frag:3:9] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Failure while reading vector. Expected vector class of > org.apache.drill.exec.vector.NullableIntVector but was holding vector class > org.apache.drill.exec.vector.NullableBigIntVector, field= > i_manufact_id(BIGINT:OPTIONAL)[$bits$(UINT1:REQUIRED), > i_manufact_id(BIGINT:OPTIONAL)] > Fragment 3:9 > [Error Id: 7fc06ab9-6c63-402b-a1b4-465526aa7dc7 on ucs-node6.perf.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Failure while reading vector. Expected vector class > of org.apache.drill.exec.vector.NullableIntVector but was holding vector > class org.apache.drill.exec.vector.NullableBigIntVector, field= > i_manufact_id(BIGINT:OPTIONAL)[$bits$(UINT1:REQUIRED), > i_manufact_id(BIGINT:OPTIONAL)] > Fragment 3:9 > [Error Id: 7fc06ab9-6c63-402b-a1b4-465526aa7dc7 on ucs-node6.perf.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) > ~[drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_65] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] > Caused by: java.lang.IllegalStateException: Failure while reading vector. > Expected vector class of org.apache.drill.exec.vector.NullableIntVector but > was holding vector class org.apache.drill.exec.vector.NullableBigIntVector, > field= i_manufact_id(BIGINT:OPTIONAL)[$bits$(UINT1:REQUIRED), > i_manufact_id(BIGINT:OPTIONAL)] > at > org.apache.drill.exec.record.VectorContainer.getValueAccessorById(VectorContainer.java:290) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.RecordBatchLoader.getValueAccessorById(RecordBatchLoader.java:178) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.getValueAccessorById(UnorderedReceiverBatch.java:135) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.test.generated.PartitionerGen36655$OutgoingRecordBatch.doSetup(PartitionerTemplate.java:64) > ~[na:na] > at > org.apache.drill.exec.test.generated.PartitionerGen36655$OutgoingRecordBatch.initializeBatch(PartitionerTemplate.java:358) > ~[na:na] > at > org.apache.drill.exec.test.generated.PartitionerGen36655.flushOutgoingBatches(PartitionerTemplate.java:163) > ~[na:na] > at > org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator$FlushBatchesHandlingClass.execute(PartitionerDecorator.java:266) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator.executeMethodLogic(PartitionerDecorator.java:138) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator.flushOutgoingBatches(PartitionerDecorator.java:82) >
[jira] [Closed] (DRILL-4851) TPCDS Query 64 just hang there at "starting" state
[ https://issues.apache.org/jira/browse/DRILL-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu closed DRILL-4851. - Duplicate of DRILL-4347, hence close this one > TPCDS Query 64 just hang there at "starting" state > -- > > Key: DRILL-4851 > URL: https://issues.apache.org/jira/browse/DRILL-4851 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.8.0 > Environment: REL 6.0 >Reporter: Dechang Gu >Assignee: Jinfeng Ni > > TPC-DS Query 64 on SF100 (views on parquet files) hung there at "starting" > state. drillbit.log on the foreman node show the following info: > 2016-08-17 14:26:00,785 ucs-node4.perf.lab > [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 284b2996-d25a-d9af-178c-143b32ea6969: WITH cs_ui AS (SELECT cs_item_sk, > Sum(cs_ext_list_price) AS sale, Sum(cr_refunded_cash + cr_reversed_charge + > cr_store_credit) AS refund FROM catalog_sales, catalog_returns WHERE > cs_item_sk = cr_item_sk AND cs_order_number = cr_order_number GROUP BY > cs_item_sk HAVING Sum(cs_ext_list_price) > 2 * Sum( cr_refunded_cash + > cr_reversed_charge + cr_store_credit)), cross_sales AS (SELECT i_product_name > product_name, i_item_sk item_sk, s_store_name > store_name, s_zip store_zip, ad1.ca_street_number > b_street_number, ad1.ca_street_name b_streen_name, ad1.ca_city > b_city, ad1.ca_zip b_zip, ad2.ca_street_number c_street_number, > ad2.ca_street_name c_street_name, ad2.ca_cityc_city, > ad2.ca_zip c_zip, d1.d_year AS syear, d2.d_year > AS fsyear, d3.d_year s2year, Count(*) cnt, > Sum(ss_wholesale_cost) s1, Sum(ss_list_price) s2, Sum(ss_coupon_amt) > s3 FROM store_sales, store_returns, cs_ui, date_dim d1, date_dim d2, > date_dim d3, store, customer, customer_dd > emographics cd1, customer_demographics cd2, promotion, household_demographics > hd1, household_demographics hd2, customer_address ad1, customer_address ad2, > income_band ib1, income_band ib2, item WHERE ss_store_sk = s_stt > ore_sk AND ss_sold_date_sk = d1.d_date_sk AND ss_customer_sk = c_customer_sk > AND ss_cdemo_sk = cd1.cd_demo_sk AND ss_hdemo_sk = hd1.hd_demo_sk AND > ss_addr_sk = ad1.ca_address_sk AND ss_item_sk = i_item_sk AND ss_item_skk > = sr_item_sk AND ss_ticket_number = sr_ticket_number AND ss_item_sk = > cs_ui.cs_item_sk AND c_current_cdemo_sk = cd2.cd_demo_sk AND > c_current_hdemo_sk = hd2.hd_demo_sk AND c_current_addr_sk = ad2.ca_address_sk > AND c_firr > st_sales_date_sk = d2.d_date_sk AND c_first_shipto_date_sk = d3.d_date_sk AND > ss_promo_sk = p_promo_sk AND hd1.hd_income_band_sk = ib1.ib_income_band_sk > AND hd2.hd_income_band_sk = ib2.ib_income_band_sk AND cd1.cd_maritt > al_status <> cd2.cd_marital_status AND i_color IN ( 'cyan', 'peach', 'blush', > 'frosted', 'powder', 'orange' ) AND i_current_price BETWEEN 58 AND 58 + 10 > AND i_current_price BETWEEN 58 + 1 AND 58 + 15 GROUP BY i_productt > _name, i_item_sk, s_store_name, s_zip, ad1.ca_street_number, > ad1.ca_street_name, ad1.ca_city, ad1.ca_zip, ad2.ca_street_number, > ad2.ca_street_name, ad2.ca_city, ad2.ca_zip, d1.d_year, d2.d_year, d3.d_year) > SELECT cs1.prr > oduct_name, cs1.store_name, cs1.store_zip, cs1.b_street_number, > cs1.b_streen_name, cs1.b_city, cs1.b_zip, cs1.c_street_number, > cs1.c_street_name, cs1.c_city, cs1.c_zip, cs1.syear, cs1.cnt, cs1.s1, cs1.s2, > cs1.s3, cs2.s11 > , cs2.s2, cs2.s3, cs2.syear, cs2.cnt FROM cross_sales cs1, cross_sales cs2 > WHERE cs1.item_sk = cs2.item_sk AND cs1.syear = 2001 AND cs2.syear = 2001 + > 1 AND cs2.cnt <= cs1.cnt AND cs1.store_name = cs2.store_name AND > cs1.store_zip = cs2.store_zip ORDER BY cs1.product_name, cs1.store_name, > cs2.cnt > 2016-08-17 14:26:04,045 ucs-node4.perf.lab > [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO > o.a.d.exec.store.parquet.Metadata - Took 1 ms to get file statuses > 2016-08-17 14:26:04,051 ucs-node4.perf.lab > [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO > o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of > 1 using 1 threads. Time: 4ms total, 4.753323mm > s avg, 4ms max. > 2016-08-17 14:26:04,051 ucs-node4.perf.lab > [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO > o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of > 1 using 1 threads. Earliest start: 12.261000 > μs, Latest start: 12.261000 μs, Average start: 12.261000 μs . > 2016-08-17 14:26:04,051 ucs-node4.perf.lab > [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO > o.a.d.exec.store.parquet.Metadata - Took 6 ms to
[jira] [Assigned] (DRILL-4827) Checking modification time of directories takes too long, needs to be improved
[ https://issues.apache.org/jira/browse/DRILL-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu reassigned DRILL-4827: - Assignee: Aman Sinha > Checking modification time of directories takes too long, needs to be improved > -- > > Key: DRILL-4827 > URL: https://issues.apache.org/jira/browse/DRILL-4827 > Project: Apache Drill > Issue Type: Improvement > Components: Functions - Drill >Affects Versions: 1.8.0 > Environment: RHEL 6 >Reporter: Dechang Gu >Assignee: Aman Sinha > > This is tracking bug for metadata cache performance for directory checking. > When evaluating the fix for Drill-4530, we run the following two queries on > 50K parquet files in a 3-layer directory hierarchy: > Query1: explain plan for select * from > dfs.`/tpchMetaParquet/tpch100_dir_partitioned_5files/lineitem` where > dir0=2006 and dir1=12 and dir2=15; > Query2: explain plan for select * from > dfs.`/tpchMetaParquet/tpch100_dir_partitioned_5files/lineitem/2006/12/15`; > Query1 takes 3.254 secs. Query2 0.505 secs. > Drillbit.log shows that for Query1, 2.5 secs spent after metadata cache was > read and before partition pruning: > 2016-08-02 15:43:43,051 ucs-node7.perf.lab > [285edddf-b1f3-cd74-e826-84cb91ebc6e1:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 285edddf-b1f3-cd74-e826-84cb91ebc6e1: explain plan for select * from > dfs.`/tpchMetaParquet/tpch100_dir_partitioned_5files/lineitem` where > dir0=2006 and dir1=12 and dir2=15 > 2016-08-02 15:43:43,193 ucs-node7.perf.lab > [285edddf-b1f3-cd74-e826-84cb91ebc6e1:foreman] INFO > o.a.d.exec.store.parquet.Metadata - Took 6 ms to read directories from > directory cache file > 2016-08-02 15:43:45,745 ucs-node7.perf.lab > [285edddf-b1f3-cd74-e826-84cb91ebc6e1:foreman] INFO > o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning > class: > org.apache.drill.exec.planner.logical.partition.PruneScanRule$DirPruneScanFilterOnScanRule > Further investigation shows that the 2.5 secs was for checking modification > time of directories, which is proportional to the number of directories to be > checked. > Looks like this can be improved by only checking the top level directory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-4877) max(dir0), max(dir1) query against parquet data slower by 2X
[
https://issues.apache.org/jira/browse/DRILL-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536289#comment-15536289
]
Dechang Gu commented on DRILL-4877:
---
I run the query on the latest 1.9.0 (gitid: 2295715), and the query times for 3
runs:
Run1:TOTAL TIME : 33291 msec
Run2: TOTAL TIME : 24709 msec
Run3: TOTAL TIME : 26906 msec
which show no regression.
[~khfaraaz] So the jira is verified and can be closed.
> max(dir0), max(dir1) query against parquet data slower by 2X
>
>
> Key: DRILL-4877
> URL: https://issues.apache.org/jira/browse/DRILL-4877
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.9.0
> Environment: 4 node cluster centos
>Reporter: Khurram Faraaz
>Assignee: Aman Sinha
>Priority: Critical
> Fix For: 1.9.0
>
>
> max(dir0), max(dir1) query against parquet data slower by 2X
> test was run with meta data cache on both 1.7.0 and 1.9.0
> there is a difference in query plan and also execution time on 1.9.0 is close
> to 2X that on 1.7.0
> Test from Drill 1.9.0 git commit id: 28d315bb
> on 4 node Centos cluster
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select max(dir0), max(dir1), max(dir2) from
> `DRILL_4589`;
> +-+-+-+
> | EXPR$0 | EXPR$1 | EXPR$2 |
> +-+-+-+
> | 2015| Q4 | null|
> +-+-+-+
> 1 row selected (70.644 seconds)
> {noformat}
> Query plan for the above query, note than in Drill 1.9.0 usedMetadataFile is
> not available is the query plan text.
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> explain plan for select max(dir0), max(dir1),
> max(dir2) from `DRILL_4589`;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 Project(EXPR$0=[$0], EXPR$1=[$1], EXPR$2=[$2])
> 00-02StreamAgg(group=[{}], EXPR$0=[MAX($0)], EXPR$1=[MAX($1)],
> EXPR$2=[MAX($2)])
> 00-03 UnionExchange
> 01-01StreamAgg(group=[{}], EXPR$0=[MAX($0)], EXPR$1=[MAX($1)],
> EXPR$2=[MAX($2)])
> 01-02 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=/tmp/DRILL_4589/1990/Q1/f672.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2011/Q4/f162.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2000/Q2/f1101.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1996/Q2/f110.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2006/Q3/f1192.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1999/Q2/f174.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2006/Q4/f885.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2001/Q3/f1720.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2001/Q1/f1779.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1991/Q2/f629.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2003/Q4/f821.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2015/Q3/f896.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2002/Q2/f1458.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2004/Q4/f1756.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2001/Q2/f1490.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2003/Q3/f1137.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2013/Q1/f561.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1990/Q3/f1562.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2003/Q1/f1445.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2006/Q1/f236.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1992/Q4/f1209.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2014/Q2/f518.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1993/Q4/f1598.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2008/Q1/f780.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1999/Q1/f1763.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1990/Q4/f381.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1990/Q1/f1870.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2014/Q1/f915.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2001/Q2/f673.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1998/Q1/f736.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2013/Q2/f749.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2007/Q3/f111.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1993/Q3/f776.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2002/Q1/f403.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2005/Q2/f904.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2000/Q4/f944.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1994/Q2/f506.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1994/Q4/f612.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1991/Q1/f1838.parquet],
> ReadEntryWithPath
[jira] [Closed] (DRILL-4857) When no partition pruning occurs with metadata caching there's a performance regression
[ https://issues.apache.org/jira/browse/DRILL-4857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu closed DRILL-4857. - Verified, and added test cases to Perf Test Framework. > When no partition pruning occurs with metadata caching there's a performance > regression > --- > > Key: DRILL-4857 > URL: https://issues.apache.org/jira/browse/DRILL-4857 > Project: Apache Drill > Issue Type: Bug > Components: Metadata, Query Planning & Optimization >Affects Versions: 1.7.0 >Reporter: Aman Sinha >Assignee: Aman Sinha > Fix For: 1.8.0 > > > After DRILL-4530, we see the (expected) performance improvements in planning > time with metadata cache for cases where partition pruning got applied. > However, in cases where it did not get applied and for sufficiently large > number of files (tested with up to 400K files), there's performance > regression. Part of this was addressed by DRILL-4846. This JIRA is to > track some remaining fixes to address the regression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-4846) Eliminate extra operations during metadata cache pruning
[ https://issues.apache.org/jira/browse/DRILL-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu closed DRILL-4846. - Verified, and added test cases to Perf Test Framework. So close it. > Eliminate extra operations during metadata cache pruning > > > Key: DRILL-4846 > URL: https://issues.apache.org/jira/browse/DRILL-4846 > Project: Apache Drill > Issue Type: Bug > Components: Metadata >Affects Versions: 1.7.0 >Reporter: Aman Sinha >Assignee: Aman Sinha > Fix For: 1.8.0 > > > While doing performance testing for DRILL-4530 using a new data set and > queries, we found two potential performance issues: (a) the metadata cache > was being read twice in some cases and (b) the checking for directory > modification time was being done twice, once as part of the first phase of > directory-based pruning and subsequently after the second phase pruning. > This check gets expensive for large number of directories. Creating this > JIRA to track fixes for these issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4877) max(dir0), max(dir1) query against parquet data slower by 2X
[
https://issues.apache.org/jira/browse/DRILL-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15472053#comment-15472053
]
Dechang Gu commented on DRILL-4877:
---
Run the same queries on the commit (DRILL-4715: Fix IOBE for concurrent query
planning by applying fix of…, 70aba772a9434e0703078bddb47008f35cffb8bf),which
is right before commit 4f818d074373f3572cb3c2e99d1c9c43df2090aa DRILL-4530, it
behaves similar to 1.7.0:
{noformat}
Query Id AD
1.8.0 (gitid 70aba77)
Run1
Run2Run3Avg
DRILL4589_4877_64 53563 31403 30583
38516
DRILL4589_4877_EXPPLAN_63 11473 901488999795
{noformat}
and on the commit 4f818d074373f3572cb3c2e99d1c9c43df2090aa DRILL-4530, it
behaves like 1.8.0:
{noformat}
Query IdAD
1.8.0 (gitid 4f818d0)
Run1
Run2 Run3Avg
DRILL4589_4877_6436365 32699
32788 33951
DRILL4589_4877_EXPPLAN_6321376 1646517265
18369
{noformat}
So the regression is introduced in the commit
4f818d074373f3572cb3c2e99d1c9c43df2090aa
> max(dir0), max(dir1) query against parquet data slower by 2X
>
>
> Key: DRILL-4877
> URL: https://issues.apache.org/jira/browse/DRILL-4877
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.9.0
> Environment: 4 node cluster centos
>Reporter: Khurram Faraaz
>Assignee: Aman Sinha
>Priority: Critical
>
> max(dir0), max(dir1) query against parquet data slower by 2X
> test was run with meta data cache on both 1.7.0 and 1.9.0
> there is a difference in query plan and also execution time on 1.9.0 is close
> to 2X that on 1.7.0
> Test from Drill 1.9.0 git commit id: 28d315bb
> on 4 node Centos cluster
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select max(dir0), max(dir1), max(dir2) from
> `DRILL_4589`;
> +-+-+-+
> | EXPR$0 | EXPR$1 | EXPR$2 |
> +-+-+-+
> | 2015| Q4 | null|
> +-+-+-+
> 1 row selected (70.644 seconds)
> {noformat}
> Query plan for the above query, note than in Drill 1.9.0 usedMetadataFile is
> not available is the query plan text.
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> explain plan for select max(dir0), max(dir1),
> max(dir2) from `DRILL_4589`;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 Project(EXPR$0=[$0], EXPR$1=[$1], EXPR$2=[$2])
> 00-02StreamAgg(group=[{}], EXPR$0=[MAX($0)], EXPR$1=[MAX($1)],
> EXPR$2=[MAX($2)])
> 00-03 UnionExchange
> 01-01StreamAgg(group=[{}], EXPR$0=[MAX($0)], EXPR$1=[MAX($1)],
> EXPR$2=[MAX($2)])
> 01-02 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=/tmp/DRILL_4589/1990/Q1/f672.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2011/Q4/f162.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2000/Q2/f1101.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1996/Q2/f110.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2006/Q3/f1192.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1999/Q2/f174.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2006/Q4/f885.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2001/Q3/f1720.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2001/Q1/f1779.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1991/Q2/f629.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2003/Q4/f821.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2015/Q3/f896.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2002/Q2/f1458.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2004/Q4/f1756.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2001/Q2/f1490.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2003/Q3/f1137.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2013/Q1/f561.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1990/Q3/f1562.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2003/Q1/f1445.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2006/Q1/f236.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1992/Q4/f1209.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2014/Q2/f518.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/1993/Q4/f1598.parquet],
> ReadEntryWithPath [path=/tmp/DRILL_4589/2008/Q1/f780.parquet],
> ReadEntryWithPath
[jira] [Closed] (DRILL-4816) sqlline -f failed to read the query file
[ https://issues.apache.org/jira/browse/DRILL-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu closed DRILL-4816. - Verified on commit 6f5864cfbdcdccd67ba8f031126568d8edbf0c42. LGTM. > sqlline -f failed to read the query file > > > Key: DRILL-4816 > URL: https://issues.apache.org/jira/browse/DRILL-4816 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.8.0 > Environment: redha 2.6.32-358.el6 >Reporter: Dechang Gu >Assignee: Parth Chandra > Fix For: 1.8.0 > > > Installed Apache Drill master (commit id: 4e1bdac) on a 10 node cluster. > sqlline -u "jdbc:drill:schema=dfs.xxxParquet" -f refresh_meta_dirs.sql > hit the "No such file or directory" error: > 16:34:47,956 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could > NOT find resource [logback.groovy] > 16:34:47,956 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could > NOT find resource [logback-test.xml] > 16:34:47,960 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found > resource [logback.xml] at [file:/mapr/drillPerf/ > drillbit/conf/logback.xml] > 16:34:47,961 |-WARN in ch.qos.logback.classic.LoggerContext[default] - > Resource [logback.xml] occurs multiple times on the cl > asspath. > 16:34:47,961 |-WARN in ch.qos.logback.classic.LoggerContext[default] - > Resource [logback.xml] occurs at [file:/opt/mapr/drill > /drill-1.8.0/conf/logback.xml] > 16:34:47,961 |-WARN in ch.qos.logback.classic.LoggerContext[default] - > Resource [logback.xml] occurs at [file:/mapr/drillPerf > /drillbit/conf/logback.xml] > 16:34:48,163 |-INFO in > ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not > set > 16:34:48,168 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - > About to instantiate appender of type [ch.qos.logbac > k.core.rolling.RollingFileAppender] > 16:34:48,182 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - > Naming appender as [FILE] > 16:34:48,246 |-INFO in > ch.qos.logback.core.rolling.FixedWindowRollingPolicy@29989e7c - No > compression will be used > 16:34:48,246 |-WARN in > ch.qos.logback.core.rolling.FixedWindowRollingPolicy@29989e7c - Large window > sizes are not allowed. > 16:34:48,246 |-WARN in > ch.qos.logback.core.rolling.FixedWindowRollingPolicy@29989e7c - MaxIndex > reduced to 21 > 16:34:48,257 |-INFO in > ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default > type [ch.qos.logback.class > ic.encoder.PatternLayoutEncoder] for [encoder] property > 16:34:48,319 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] > - Active log file name: /var/log/drill/drillbit_ > ucs-node1.perf.lab.log > 16:34:48,319 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] > - File property is set to [/var/log/drill/drillb > it_ucs-node1.perf.lab.log] > 16:34:48,321 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - > Setting additivity of logger [org.apache.drill] to > false > 16:34:48,321 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - > org.apache.drill level set to INFO > 16:34:48,322 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - > Attaching appender named [FILE] to Logger[org.apa > che.drill] > 16:34:48,323 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - ROOT > level set to INFO > 16:34:48,323 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - > Attaching appender named [FILE] to Logger[ROOT] > 16:34:48,323 |-INFO in > ch.qos.logback.classic.joran.action.ConfigurationAction - End of > configuration. > 16:34:48,324 |-INFO in > ch.qos.logback.classic.joran.JoranConfigurator@62ccf439 - Registering current > configuration as safe fa > llback point > -u[i+1] (No such file or directory) > Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl > running the query from inside the sqlline connection is ok. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3714) Query runs out of memory and remains in CANCELLATION_REQUESTED state until drillbit is restarted
[
https://issues.apache.org/jira/browse/DRILL-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431205#comment-15431205
]
Dechang Gu commented on DRILL-3714:
---
Verified on TPCDS SF100, LGTM:
0: jdbc:drill:schema=dfs.tpcdsPar100> SELECT SUM(ss.ss_net_paid_inc_tax) OVER
(PARTITION BY ss.ss_store_sk) AS TotalSpend FROM store_sales ss WHERE
ss.ss_store_sk IS NOT NULL ORDER BY 1 LIMIT 10;
+---+
| TotalSpend |
+---+
| 2.383345486854E9 |
| 2.383345486854E9 |
| 2.383345486854E9 |
| 2.383345486854E9 |
| 2.383345486854E9 |
| 2.383345486854E9 |
| 2.383345486854E9 |
| 2.383345486854E9 |
| 2.383345486854E9 |
| 2.383345486854E9 |
+---+
10 rows selected (29.575 seconds)
> Query runs out of memory and remains in CANCELLATION_REQUESTED state until
> drillbit is restarted
>
>
> Key: DRILL-3714
> URL: https://issues.apache.org/jira/browse/DRILL-3714
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.2.0
>Reporter: Victoria Markman
>Assignee: Jacques Nadeau
>Priority: Critical
> Fix For: 1.7.0
>
> Attachments: Screen Shot 2015-08-26 at 10.36.33 AM.png, drillbit.log,
> jstack.txt, query_profile_2a2210a7-7a78-c774-d54c-c863d0b77bb0.json
>
>
> This is a variation of DRILL-3705 with the difference of drill behavior when
> hitting OOM condition.
> Query runs out of memory during execution and remains in
> "CANCELLATION_REQUESTED" state until drillbit is bounced.
> Client (sqlline in this case) never gets a response from the server.
> Reproduction details:
> Single node drillbit installation.
> DRILL_MAX_DIRECT_MEMORY="8G"
> DRILL_HEAP="4G"
> Run this query on TPCDS SF100 data set
> {code}
> SELECT SUM(ss.ss_net_paid_inc_tax) OVER (PARTITION BY ss.ss_store_sk) AS
> TotalSpend FROM store_sales ss WHERE ss.ss_store_sk IS NOT NULL ORDER BY 1
> LIMIT 10;
> {code}
> drillbit.log
> {code}
> 2015-08-26 16:54:58,469 [2a2210a7-7a78-c774-d54c-c863d0b77bb0:frag:3:22] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 2a2210a7-7a78-c774-d54c-c863d0b77bb0:3:22: State to report: RUNNING
> 2015-08-26 16:55:50,498 [BitServer-5] WARN
> o.a.drill.exec.rpc.data.DataServer - Message of mode REQUEST of rpc type 3
> took longer than 500ms. Actual duration was 2569ms.
> 2015-08-26 16:56:31,086 [BitServer-5] ERROR
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.
> Connection: /10.10.88.133:31012 <--> /10.10.88.133:54554 (data server).
> Closing connection.
> io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Direct
> buffer memory
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:233)
> ~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:618)
> [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:329)
> [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:250)
> [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> [netty-common-4.0.27.Final.jar:4.0.27.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_71]
> at
[jira] [Commented] (DRILL-4851) TPCDS Query 64 just hang there at "starting" state
[ https://issues.apache.org/jira/browse/DRILL-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428771#comment-15428771 ] Dechang Gu commented on DRILL-4851: --- The issue also is shown in Drill 1.6.0 release, commit id: d51f7fc > TPCDS Query 64 just hang there at "starting" state > -- > > Key: DRILL-4851 > URL: https://issues.apache.org/jira/browse/DRILL-4851 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.8.0 > Environment: REL 6.0 >Reporter: Dechang Gu > > TPC-DS Query 64 on SF100 (views on parquet files) hung there at "starting" > state. drillbit.log on the foreman node show the following info: > 2016-08-17 14:26:00,785 ucs-node4.perf.lab > [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 284b2996-d25a-d9af-178c-143b32ea6969: WITH cs_ui AS (SELECT cs_item_sk, > Sum(cs_ext_list_price) AS sale, Sum(cr_refunded_cash + cr_reversed_charge + > cr_store_credit) AS refund FROM catalog_sales, catalog_returns WHERE > cs_item_sk = cr_item_sk AND cs_order_number = cr_order_number GROUP BY > cs_item_sk HAVING Sum(cs_ext_list_price) > 2 * Sum( cr_refunded_cash + > cr_reversed_charge + cr_store_credit)), cross_sales AS (SELECT i_product_name > product_name, i_item_sk item_sk, s_store_name > store_name, s_zip store_zip, ad1.ca_street_number > b_street_number, ad1.ca_street_name b_streen_name, ad1.ca_city > b_city, ad1.ca_zip b_zip, ad2.ca_street_number c_street_number, > ad2.ca_street_name c_street_name, ad2.ca_cityc_city, > ad2.ca_zip c_zip, d1.d_year AS syear, d2.d_year > AS fsyear, d3.d_year s2year, Count(*) cnt, > Sum(ss_wholesale_cost) s1, Sum(ss_list_price) s2, Sum(ss_coupon_amt) > s3 FROM store_sales, store_returns, cs_ui, date_dim d1, date_dim d2, > date_dim d3, store, customer, customer_dd > emographics cd1, customer_demographics cd2, promotion, household_demographics > hd1, household_demographics hd2, customer_address ad1, customer_address ad2, > income_band ib1, income_band ib2, item WHERE ss_store_sk = s_stt > ore_sk AND ss_sold_date_sk = d1.d_date_sk AND ss_customer_sk = c_customer_sk > AND ss_cdemo_sk = cd1.cd_demo_sk AND ss_hdemo_sk = hd1.hd_demo_sk AND > ss_addr_sk = ad1.ca_address_sk AND ss_item_sk = i_item_sk AND ss_item_skk > = sr_item_sk AND ss_ticket_number = sr_ticket_number AND ss_item_sk = > cs_ui.cs_item_sk AND c_current_cdemo_sk = cd2.cd_demo_sk AND > c_current_hdemo_sk = hd2.hd_demo_sk AND c_current_addr_sk = ad2.ca_address_sk > AND c_firr > st_sales_date_sk = d2.d_date_sk AND c_first_shipto_date_sk = d3.d_date_sk AND > ss_promo_sk = p_promo_sk AND hd1.hd_income_band_sk = ib1.ib_income_band_sk > AND hd2.hd_income_band_sk = ib2.ib_income_band_sk AND cd1.cd_maritt > al_status <> cd2.cd_marital_status AND i_color IN ( 'cyan', 'peach', 'blush', > 'frosted', 'powder', 'orange' ) AND i_current_price BETWEEN 58 AND 58 + 10 > AND i_current_price BETWEEN 58 + 1 AND 58 + 15 GROUP BY i_productt > _name, i_item_sk, s_store_name, s_zip, ad1.ca_street_number, > ad1.ca_street_name, ad1.ca_city, ad1.ca_zip, ad2.ca_street_number, > ad2.ca_street_name, ad2.ca_city, ad2.ca_zip, d1.d_year, d2.d_year, d3.d_year) > SELECT cs1.prr > oduct_name, cs1.store_name, cs1.store_zip, cs1.b_street_number, > cs1.b_streen_name, cs1.b_city, cs1.b_zip, cs1.c_street_number, > cs1.c_street_name, cs1.c_city, cs1.c_zip, cs1.syear, cs1.cnt, cs1.s1, cs1.s2, > cs1.s3, cs2.s11 > , cs2.s2, cs2.s3, cs2.syear, cs2.cnt FROM cross_sales cs1, cross_sales cs2 > WHERE cs1.item_sk = cs2.item_sk AND cs1.syear = 2001 AND cs2.syear = 2001 + > 1 AND cs2.cnt <= cs1.cnt AND cs1.store_name = cs2.store_name AND > cs1.store_zip = cs2.store_zip ORDER BY cs1.product_name, cs1.store_name, > cs2.cnt > 2016-08-17 14:26:04,045 ucs-node4.perf.lab > [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO > o.a.d.exec.store.parquet.Metadata - Took 1 ms to get file statuses > 2016-08-17 14:26:04,051 ucs-node4.perf.lab > [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO > o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of > 1 using 1 threads. Time: 4ms total, 4.753323mm > s avg, 4ms max. > 2016-08-17 14:26:04,051 ucs-node4.perf.lab > [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO > o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of > 1 using 1 threads. Earliest start: 12.261000 > μs, Latest start: 12.261000 μs, Average start: 12.261000 μs . > 2016-08-17 14:26:04,051 ucs-node4.perf.lab > [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO >
[jira] [Commented] (DRILL-4851) TPCDS Query 64 just hang there at "starting" state
[ https://issues.apache.org/jira/browse/DRILL-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428390#comment-15428390 ] Dechang Gu commented on DRILL-4851: --- The same issue is also shown in Drill 1.7.0 release, commit id: 7800aeb > TPCDS Query 64 just hang there at "starting" state > -- > > Key: DRILL-4851 > URL: https://issues.apache.org/jira/browse/DRILL-4851 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.8.0 > Environment: REL 6.0 >Reporter: Dechang Gu > > TPC-DS Query 64 on SF100 (views on parquet files) hung there at "starting" > state. drillbit.log on the foreman node show the following info: > 2016-08-17 14:26:00,785 ucs-node4.perf.lab > [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 284b2996-d25a-d9af-178c-143b32ea6969: WITH cs_ui AS (SELECT cs_item_sk, > Sum(cs_ext_list_price) AS sale, Sum(cr_refunded_cash + cr_reversed_charge + > cr_store_credit) AS refund FROM catalog_sales, catalog_returns WHERE > cs_item_sk = cr_item_sk AND cs_order_number = cr_order_number GROUP BY > cs_item_sk HAVING Sum(cs_ext_list_price) > 2 * Sum( cr_refunded_cash + > cr_reversed_charge + cr_store_credit)), cross_sales AS (SELECT i_product_name > product_name, i_item_sk item_sk, s_store_name > store_name, s_zip store_zip, ad1.ca_street_number > b_street_number, ad1.ca_street_name b_streen_name, ad1.ca_city > b_city, ad1.ca_zip b_zip, ad2.ca_street_number c_street_number, > ad2.ca_street_name c_street_name, ad2.ca_cityc_city, > ad2.ca_zip c_zip, d1.d_year AS syear, d2.d_year > AS fsyear, d3.d_year s2year, Count(*) cnt, > Sum(ss_wholesale_cost) s1, Sum(ss_list_price) s2, Sum(ss_coupon_amt) > s3 FROM store_sales, store_returns, cs_ui, date_dim d1, date_dim d2, > date_dim d3, store, customer, customer_dd > emographics cd1, customer_demographics cd2, promotion, household_demographics > hd1, household_demographics hd2, customer_address ad1, customer_address ad2, > income_band ib1, income_band ib2, item WHERE ss_store_sk = s_stt > ore_sk AND ss_sold_date_sk = d1.d_date_sk AND ss_customer_sk = c_customer_sk > AND ss_cdemo_sk = cd1.cd_demo_sk AND ss_hdemo_sk = hd1.hd_demo_sk AND > ss_addr_sk = ad1.ca_address_sk AND ss_item_sk = i_item_sk AND ss_item_skk > = sr_item_sk AND ss_ticket_number = sr_ticket_number AND ss_item_sk = > cs_ui.cs_item_sk AND c_current_cdemo_sk = cd2.cd_demo_sk AND > c_current_hdemo_sk = hd2.hd_demo_sk AND c_current_addr_sk = ad2.ca_address_sk > AND c_firr > st_sales_date_sk = d2.d_date_sk AND c_first_shipto_date_sk = d3.d_date_sk AND > ss_promo_sk = p_promo_sk AND hd1.hd_income_band_sk = ib1.ib_income_band_sk > AND hd2.hd_income_band_sk = ib2.ib_income_band_sk AND cd1.cd_maritt > al_status <> cd2.cd_marital_status AND i_color IN ( 'cyan', 'peach', 'blush', > 'frosted', 'powder', 'orange' ) AND i_current_price BETWEEN 58 AND 58 + 10 > AND i_current_price BETWEEN 58 + 1 AND 58 + 15 GROUP BY i_productt > _name, i_item_sk, s_store_name, s_zip, ad1.ca_street_number, > ad1.ca_street_name, ad1.ca_city, ad1.ca_zip, ad2.ca_street_number, > ad2.ca_street_name, ad2.ca_city, ad2.ca_zip, d1.d_year, d2.d_year, d3.d_year) > SELECT cs1.prr > oduct_name, cs1.store_name, cs1.store_zip, cs1.b_street_number, > cs1.b_streen_name, cs1.b_city, cs1.b_zip, cs1.c_street_number, > cs1.c_street_name, cs1.c_city, cs1.c_zip, cs1.syear, cs1.cnt, cs1.s1, cs1.s2, > cs1.s3, cs2.s11 > , cs2.s2, cs2.s3, cs2.syear, cs2.cnt FROM cross_sales cs1, cross_sales cs2 > WHERE cs1.item_sk = cs2.item_sk AND cs1.syear = 2001 AND cs2.syear = 2001 + > 1 AND cs2.cnt <= cs1.cnt AND cs1.store_name = cs2.store_name AND > cs1.store_zip = cs2.store_zip ORDER BY cs1.product_name, cs1.store_name, > cs2.cnt > 2016-08-17 14:26:04,045 ucs-node4.perf.lab > [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO > o.a.d.exec.store.parquet.Metadata - Took 1 ms to get file statuses > 2016-08-17 14:26:04,051 ucs-node4.perf.lab > [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO > o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of > 1 using 1 threads. Time: 4ms total, 4.753323mm > s avg, 4ms max. > 2016-08-17 14:26:04,051 ucs-node4.perf.lab > [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO > o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of > 1 using 1 threads. Earliest start: 12.261000 > μs, Latest start: 12.261000 μs, Average start: 12.261000 μs . > 2016-08-17 14:26:04,051 ucs-node4.perf.lab > [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO >
[jira] [Created] (DRILL-4851) TPCDS Query 64 just hang there at "starting" state
Dechang Gu created DRILL-4851: - Summary: TPCDS Query 64 just hang there at "starting" state Key: DRILL-4851 URL: https://issues.apache.org/jira/browse/DRILL-4851 Project: Apache Drill Issue Type: Bug Components: Functions - Drill Affects Versions: 1.8.0 Environment: REL 6.0 Reporter: Dechang Gu TPC-DS Query 64 on SF100 (views on parquet files) hung there at "starting" state. drillbit.log on the foreman node show the following info: 2016-08-17 14:26:00,785 ucs-node4.perf.lab [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 284b2996-d25a-d9af-178c-143b32ea6969: WITH cs_ui AS (SELECT cs_item_sk, Sum(cs_ext_list_price) AS sale, Sum(cr_refunded_cash + cr_reversed_charge + cr_store_credit) AS refund FROM catalog_sales, catalog_returns WHERE cs_item_sk = cr_item_sk AND cs_order_number = cr_order_number GROUP BY cs_item_sk HAVING Sum(cs_ext_list_price) > 2 * Sum( cr_refunded_cash + cr_reversed_charge + cr_store_credit)), cross_sales AS (SELECT i_product_name product_name, i_item_sk item_sk, s_store_name store_name, s_zip store_zip, ad1.ca_street_number b_street_number, ad1.ca_street_name b_streen_name, ad1.ca_city b_city, ad1.ca_zip b_zip, ad2.ca_street_number c_street_number, ad2.ca_street_name c_street_name, ad2.ca_cityc_city, ad2.ca_zip c_zip, d1.d_year AS syear, d2.d_year AS fsyear, d3.d_year s2year, Count(*) cnt, Sum(ss_wholesale_cost) s1, Sum(ss_list_price) s2, Sum(ss_coupon_amt) s3 FROM store_sales, store_returns, cs_ui, date_dim d1, date_dim d2, date_dim d3, store, customer, customer_dd emographics cd1, customer_demographics cd2, promotion, household_demographics hd1, household_demographics hd2, customer_address ad1, customer_address ad2, income_band ib1, income_band ib2, item WHERE ss_store_sk = s_stt ore_sk AND ss_sold_date_sk = d1.d_date_sk AND ss_customer_sk = c_customer_sk AND ss_cdemo_sk = cd1.cd_demo_sk AND ss_hdemo_sk = hd1.hd_demo_sk AND ss_addr_sk = ad1.ca_address_sk AND ss_item_sk = i_item_sk AND ss_item_skk = sr_item_sk AND ss_ticket_number = sr_ticket_number AND ss_item_sk = cs_ui.cs_item_sk AND c_current_cdemo_sk = cd2.cd_demo_sk AND c_current_hdemo_sk = hd2.hd_demo_sk AND c_current_addr_sk = ad2.ca_address_sk AND c_firr st_sales_date_sk = d2.d_date_sk AND c_first_shipto_date_sk = d3.d_date_sk AND ss_promo_sk = p_promo_sk AND hd1.hd_income_band_sk = ib1.ib_income_band_sk AND hd2.hd_income_band_sk = ib2.ib_income_band_sk AND cd1.cd_maritt al_status <> cd2.cd_marital_status AND i_color IN ( 'cyan', 'peach', 'blush', 'frosted', 'powder', 'orange' ) AND i_current_price BETWEEN 58 AND 58 + 10 AND i_current_price BETWEEN 58 + 1 AND 58 + 15 GROUP BY i_productt _name, i_item_sk, s_store_name, s_zip, ad1.ca_street_number, ad1.ca_street_name, ad1.ca_city, ad1.ca_zip, ad2.ca_street_number, ad2.ca_street_name, ad2.ca_city, ad2.ca_zip, d1.d_year, d2.d_year, d3.d_year) SELECT cs1.prr oduct_name, cs1.store_name, cs1.store_zip, cs1.b_street_number, cs1.b_streen_name, cs1.b_city, cs1.b_zip, cs1.c_street_number, cs1.c_street_name, cs1.c_city, cs1.c_zip, cs1.syear, cs1.cnt, cs1.s1, cs1.s2, cs1.s3, cs2.s11 , cs2.s2, cs2.s3, cs2.syear, cs2.cnt FROM cross_sales cs1, cross_sales cs2 WHERE cs1.item_sk = cs2.item_sk AND cs1.syear = 2001 AND cs2.syear = 2001 + 1 AND cs2.cnt <= cs1.cnt AND cs1.store_name = cs2.store_name AND cs1.store_zip = cs2.store_zip ORDER BY cs1.product_name, cs1.store_name, cs2.cnt 2016-08-17 14:26:04,045 ucs-node4.perf.lab [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO o.a.d.exec.store.parquet.Metadata - Took 1 ms to get file statuses 2016-08-17 14:26:04,051 ucs-node4.perf.lab [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1 using 1 threads. Time: 4ms total, 4.753323mm s avg, 4ms max. 2016-08-17 14:26:04,051 ucs-node4.perf.lab [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1 using 1 threads. Earliest start: 12.261000 μs, Latest start: 12.261000 μs, Average start: 12.261000 μs . 2016-08-17 14:26:04,051 ucs-node4.perf.lab [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO o.a.d.exec.store.parquet.Metadata - Took 6 ms to read file metadata 2016-08-17 14:26:04,064 ucs-node4.perf.lab [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO o.a.d.exec.store.parquet.Metadata - Took 1 ms to get file statuses 2016-08-17 14:26:04,068 ucs-node4.perf.lab [284b2996-d25a-d9af-178c-143b32ea6969:foreman] INFO o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1 using 1 threads.
[jira] [Created] (DRILL-4850) TPCDS Query 33 failed in the second and 3rd runs, but succeeded in the 1st run
Dechang Gu created DRILL-4850: - Summary: TPCDS Query 33 failed in the second and 3rd runs, but succeeded in the 1st run Key: DRILL-4850 URL: https://issues.apache.org/jira/browse/DRILL-4850 Project: Apache Drill Issue Type: Bug Components: Functions - Drill Affects Versions: 1.8.0 Environment: REL 6.0 Reporter: Dechang Gu I run tpcds query 33 on SF100 database, 3 times consecutively. The first one succeeded, but the 2nd and 3rd runs hit the following error: 2016-08-16 20:20:52,530 ucs-node6.perf.lab [284c27f1-ee13-dfd0-6cbb-37f49810e93f:frag:3:9] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: Failure while reading vector. Expected vector class of org.apache.drill.exec.vector.NullableIntVector but was holding vector class org.apache.drill.exec.vector.NullableBigIntVector, field= i_manufact_id(BIGINT:OPTIONAL)[$bits$(UINT1:REQUIRED), i_manufact_id(BIGINT:OPTIONAL)] Fragment 3:9 [Error Id: 7fc06ab9-6c63-402b-a1b4-465526aa7dc7 on ucs-node6.perf.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Failure while reading vector. Expected vector class of org.apache.drill.exec.vector.NullableIntVector but was holding vector class org.apache.drill.exec.vector.NullableBigIntVector, field= i_manufact_id(BIGINT:OPTIONAL)[$bits$(UINT1:REQUIRED), i_manufact_id(BIGINT:OPTIONAL)] Fragment 3:9 [Error Id: 7fc06ab9-6c63-402b-a1b4-465526aa7dc7 on ucs-node6.perf.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) ~[drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293) [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262) [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_65] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] Caused by: java.lang.IllegalStateException: Failure while reading vector. Expected vector class of org.apache.drill.exec.vector.NullableIntVector but was holding vector class org.apache.drill.exec.vector.NullableBigIntVector, field= i_manufact_id(BIGINT:OPTIONAL)[$bits$(UINT1:REQUIRED), i_manufact_id(BIGINT:OPTIONAL)] at org.apache.drill.exec.record.VectorContainer.getValueAccessorById(VectorContainer.java:290) ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] at org.apache.drill.exec.record.RecordBatchLoader.getValueAccessorById(RecordBatchLoader.java:178) ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.getValueAccessorById(UnorderedReceiverBatch.java:135) ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] at org.apache.drill.exec.test.generated.PartitionerGen36655$OutgoingRecordBatch.doSetup(PartitionerTemplate.java:64) ~[na:na] at org.apache.drill.exec.test.generated.PartitionerGen36655$OutgoingRecordBatch.initializeBatch(PartitionerTemplate.java:358) ~[na:na] at org.apache.drill.exec.test.generated.PartitionerGen36655.flushOutgoingBatches(PartitionerTemplate.java:163) ~[na:na] at org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator$FlushBatchesHandlingClass.execute(PartitionerDecorator.java:266) ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator.executeMethodLogic(PartitionerDecorator.java:138) ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator.flushOutgoingBatches(PartitionerDecorator.java:82) ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext(PartitionSenderRootExec.java:183) ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:232) ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] at
[jira] [Closed] (DRILL-4623) Disable Epoll by Default
[ https://issues.apache.org/jira/browse/DRILL-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu closed DRILL-4623. - Verified with gitid 39406d6 on tpch concurrency test with 28 and 32 threads. There is no error message like "Connection reset by peer" . > Disable Epoll by Default > > > Key: DRILL-4623 > URL: https://issues.apache.org/jira/browse/DRILL-4623 > Project: Apache Drill > Issue Type: Bug >Reporter: Sudheesh Katkam >Assignee: Sudheesh Katkam > Fix For: 1.8.0 > > > At higher concurrency (and/or spuriously), we hit [netty issue > #3539|https://github.com/netty/netty/issues/3539]. This is an issue with the > version of Netty Drill currently uses. Once Drill moves to a later version of > Netty, epoll should be reenabled by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-2330) Add support for nested aggregate expressions for window aggregates
[
https://issues.apache.org/jira/browse/DRILL-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413801#comment-15413801
]
Dechang Gu commented on DRILL-2330:
---
The Queries are added to the Drill Perf Test Framework, and test on 1.8.0
master shows the following results:
Env: UCS micro cluster MapR 5.1.0.37549.GA, TPCDS SF100 parquet files, Apache
Drill 1.8.0 Master (gitid aaf220f)
Query time (ms)
tpcds query Run 1 Run 2 Run 3
12 DRILL-4525 Validataion Error
20 DRILL-4525 Validataion Error
47 65,515 27,391 26,247
51 DRILL-4811 IllegalArgument
53 7,722 4,310 3,518
57 22,968 15,956 15,578
63 7,2924,012 3,526
89 5,5923,777 3,585
98DRILL-4525 Validataion Error
> Add support for nested aggregate expressions for window aggregates
> --
>
> Key: DRILL-2330
> URL: https://issues.apache.org/jira/browse/DRILL-2330
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 0.8.0
>Reporter: Abhishek Girish
>Assignee: Gautam Kumar Parai
> Fix For: 1.8.0
>
> Attachments: drillbit.log
>
>
> Aggregate expressions currently cannot be nested.
> *The following query fails to validate:*
> {code:sql}
> select avg(sum(i_item_sk)) from item;
> {code}
> Error:
> Query failed: SqlValidatorException: Aggregate expressions cannot be nested
> Log attached.
> Reference: TPCDS queries (20, 63, 98, ...) fail to execute.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-4827) Checking modification time of directories takes too long, needs to be improved
Dechang Gu created DRILL-4827: - Summary: Checking modification time of directories takes too long, needs to be improved Key: DRILL-4827 URL: https://issues.apache.org/jira/browse/DRILL-4827 Project: Apache Drill Issue Type: Improvement Components: Functions - Drill Affects Versions: 1.8.0 Environment: RHEL 6 Reporter: Dechang Gu This is tracking bug for metadata cache performance for directory checking. When evaluating the fix for Drill-4530, we run the following two queries on 50K parquet files in a 3-layer directory hierarchy: Query1: explain plan for select * from dfs.`/tpchMetaParquet/tpch100_dir_partitioned_5files/lineitem` where dir0=2006 and dir1=12 and dir2=15; Query2: explain plan for select * from dfs.`/tpchMetaParquet/tpch100_dir_partitioned_5files/lineitem/2006/12/15`; Query1 takes 3.254 secs. Query2 0.505 secs. Drillbit.log shows that for Query1, 2.5 secs spent after metadata cache was read and before partition pruning: 2016-08-02 15:43:43,051 ucs-node7.perf.lab [285edddf-b1f3-cd74-e826-84cb91ebc6e1:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 285edddf-b1f3-cd74-e826-84cb91ebc6e1: explain plan for select * from dfs.`/tpchMetaParquet/tpch100_dir_partitioned_5files/lineitem` where dir0=2006 and dir1=12 and dir2=15 2016-08-02 15:43:43,193 ucs-node7.perf.lab [285edddf-b1f3-cd74-e826-84cb91ebc6e1:foreman] INFO o.a.d.exec.store.parquet.Metadata - Took 6 ms to read directories from directory cache file 2016-08-02 15:43:45,745 ucs-node7.perf.lab [285edddf-b1f3-cd74-e826-84cb91ebc6e1:foreman] INFO o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning class: org.apache.drill.exec.planner.logical.partition.PruneScanRule$DirPruneScanFilterOnScanRule Further investigation shows that the 2.5 secs was for checking modification time of directories, which is proportional to the number of directories to be checked. Looks like this can be improved by only checking the top level directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4816) sqlline -f failed to read the query file
[ https://issues.apache.org/jira/browse/DRILL-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu updated DRILL-4816: -- Description: Installed Apache Drill master (commit id: 4e1bdac) on a 10 node cluster. sqlline -u "jdbc:drill:schema=dfs.xxxParquet" -f refresh_meta_dirs.sql hit the "No such file or directory" error: 16:34:47,956 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy] 16:34:47,956 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml] 16:34:47,960 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/mapr/drillPerf/ drillbit/conf/logback.xml] 16:34:47,961 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the cl asspath. 16:34:47,961 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/opt/mapr/drill /drill-1.8.0/conf/logback.xml] 16:34:47,961 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/mapr/drillPerf /drillbit/conf/logback.xml] 16:34:48,163 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set 16:34:48,168 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logbac k.core.rolling.RollingFileAppender] 16:34:48,182 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [FILE] 16:34:48,246 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@29989e7c - No compression will be used 16:34:48,246 |-WARN in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@29989e7c - Large window sizes are not allowed. 16:34:48,246 |-WARN in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@29989e7c - MaxIndex reduced to 21 16:34:48,257 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.class ic.encoder.PatternLayoutEncoder] for [encoder] property 16:34:48,319 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file name: /var/log/drill/drillbit_ ucs-node1.perf.lab.log 16:34:48,319 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is set to [/var/log/drill/drillb it_ucs-node1.perf.lab.log] 16:34:48,321 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.apache.drill] to false 16:34:48,321 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - org.apache.drill level set to INFO 16:34:48,322 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [FILE] to Logger[org.apa che.drill] 16:34:48,323 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - ROOT level set to INFO 16:34:48,323 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [FILE] to Logger[ROOT] 16:34:48,323 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration. 16:34:48,324 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@62ccf439 - Registering current configuration as safe fa llback point -u[i+1] (No such file or directory) Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl running the query from inside the sqlline connection is ok. was: Installed Apache Drill master (commit id: 4e1bdac) on a 10 node cluster. sqlline -u "jdbc:drill:schema=dfs.comscoreParquet" -f refresh_meta_dirs.sql hit the "No such file or directory" error: 16:34:47,956 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy] 16:34:47,956 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml] 16:34:47,960 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/mapr/drillPerf/ drillbit/conf/logback.xml] 16:34:47,961 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the cl asspath. 16:34:47,961 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/opt/mapr/drill /drill-1.8.0/conf/logback.xml] 16:34:47,961 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/mapr/drillPerf /drillbit/conf/logback.xml] 16:34:48,163 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set 16:34:48,168 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logbac k.core.rolling.RollingFileAppender] 16:34:48,182 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [FILE] 16:34:48,246 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@29989e7c - No compression will be used 16:34:48,246 |-WARN in
[jira] [Created] (DRILL-4816) sqlline -f failed to read the query file
Dechang Gu created DRILL-4816: - Summary: sqlline -f failed to read the query file Key: DRILL-4816 URL: https://issues.apache.org/jira/browse/DRILL-4816 Project: Apache Drill Issue Type: Bug Components: Functions - Drill Affects Versions: 1.8.0 Environment: redha 2.6.32-358.el6 Reporter: Dechang Gu Fix For: 1.8.0 Installed Apache Drill master (commit id: 4e1bdac) on a 10 node cluster. sqlline -u "jdbc:drill:schema=dfs.comscoreParquet" -f refresh_meta_dirs.sql hit the "No such file or directory" error: 16:34:47,956 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy] 16:34:47,956 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml] 16:34:47,960 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/mapr/drillPerf/ drillbit/conf/logback.xml] 16:34:47,961 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the cl asspath. 16:34:47,961 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/opt/mapr/drill /drill-1.8.0/conf/logback.xml] 16:34:47,961 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/mapr/drillPerf /drillbit/conf/logback.xml] 16:34:48,163 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set 16:34:48,168 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logbac k.core.rolling.RollingFileAppender] 16:34:48,182 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [FILE] 16:34:48,246 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@29989e7c - No compression will be used 16:34:48,246 |-WARN in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@29989e7c - Large window sizes are not allowed. 16:34:48,246 |-WARN in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@29989e7c - MaxIndex reduced to 21 16:34:48,257 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.class ic.encoder.PatternLayoutEncoder] for [encoder] property 16:34:48,319 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file name: /var/log/drill/drillbit_ ucs-node1.perf.lab.log 16:34:48,319 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is set to [/var/log/drill/drillb it_ucs-node1.perf.lab.log] 16:34:48,321 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.apache.drill] to false 16:34:48,321 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - org.apache.drill level set to INFO 16:34:48,322 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [FILE] to Logger[org.apa che.drill] 16:34:48,323 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - ROOT level set to INFO 16:34:48,323 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [FILE] to Logger[ROOT] 16:34:48,323 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration. 16:34:48,324 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@62ccf439 - Registering current configuration as safe fa llback point -u[i+1] (No such file or directory) Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl running the query from inside the sqlline connection is ok. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-4126) Adding HiveMetaStore caching when impersonation is enabled.
[ https://issues.apache.org/jira/browse/DRILL-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu closed DRILL-4126. - Verified with Apache Drill 1.5.0 (git id 3f228d3) against the commit (git id 539cbba) prior to the patch, querying INFORTION_SCHEMA. Significant reduction in the function calls to HIVE API. Before the patch (git id 539cbba): -- get_all_databases was called 340 times -- get_all_tables was called 336 times. with the patch (AD 1.5.0 git id 3f228d3), for the same query and same databases: -- get_all_databases was only called 2 times, and -- get_all_tables was called 38 times. So the fixed LGTM, and the jira is closed. > Adding HiveMetaStore caching when impersonation is enabled. > > > Key: DRILL-4126 > URL: https://issues.apache.org/jira/browse/DRILL-4126 > Project: Apache Drill > Issue Type: Bug >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > Fix For: 1.5.0 > > > Currently, HiveMetastore caching is used only when impersonation is disabled, > such that all the hivemetastore call goes through > NonCloseableHiveClientWithCaching [1]. However, if impersonation is enabled, > caching is not used for HiveMetastore access. > This could significantly increase the planning time when hive storage plugin > is enabled, or when running a query against INFORMATION_SCHEMA. Depending on > the # of databases/tables in Hive storage plugin, the planning time or > INFORMATION_SCHEMA query could become unacceptable. This becomes even worse > if the hive metastore is running on a different node from drillbit, making > the access of hivemetastore even slower. > We are seeing that it could takes 30~60 seconds for planning time, or > execution time for INFORMATION_SCHEMA query. The long planning or execution > time for INFORMATION_SCHEMA query prevents Drill from acting "interactively" > for such queries. > We should enable caching when impersonation is used. As long as the > authorizer verifies the user has the access to databases/tables, we should > get the data from caching. By doing that, we should see reduced number of api > call to HiveMetaStore. > [1] > https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java#L299 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-4127) HiveSchema.getSubSchema() should use lazy loading of all the table names
[ https://issues.apache.org/jira/browse/DRILL-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu closed DRILL-4127. - verified with perf test framework. without the patch (commit id: 539cbba): 91_539cbba_HIVE_20160720_113024/HIVE_limit1_02/HIVE_limit1_02.log:[STAT] TOTAL TIME : 126599 msec 91_539cbba_HIVE_20160720_113024/HIVE_limit1_02/HIVE_limit1_02.log:[STAT] TOTAL TIME : 165969 msec 91_539cbba_HIVE_20160720_113024/HIVE_limit1_02/HIVE_limit1_02.log:[STAT] TOTAL TIME : 163977 msec with the patch (Apache Drill 1.5.0 GA, commit id: 3f228d3), the same query: 95_3f228d3_HIVE_20160721_130712/HIVE_limit1_02/HIVE_limit1_02.log:[STAT] TOTAL TIME : 1664 msec 95_3f228d3_HIVE_20160721_130712/HIVE_limit1_02/HIVE_limit1_02.log:[STAT] TOTAL TIME : 157 msec 95_3f228d3_HIVE_20160721_130712/HIVE_limit1_02/HIVE_limit1_02.log:[STAT] TOTAL TIME : 167 msec So, LGTM. > HiveSchema.getSubSchema() should use lazy loading of all the table names > > > Key: DRILL-4127 > URL: https://issues.apache.org/jira/browse/DRILL-4127 > Project: Apache Drill > Issue Type: Bug >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > Fix For: 1.5.0 > > > Currently, HiveSchema.getSubSchema() will pre-load all the table names when > it constructs the subschema, even though those tables names are not requested > at all. This could cause considerably big performance overhead, especially > when the hive schema contains large # of objects (thousands of tables/views > are not un-common in some use case). > In stead, we should change the loading of table names to on-demand. Only when > there is a request of get all table names, we load them into hive schema. > This should help "show schemas", since it only requires the schema name, not > the table names in the schema. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-4785) Limit 0 queries regressed in Drill 1.7.0
[ https://issues.apache.org/jira/browse/DRILL-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu closed DRILL-4785. - verified with limit 0 queries. LGTM. > Limit 0 queries regressed in Drill 1.7.0 > - > > Key: DRILL-4785 > URL: https://issues.apache.org/jira/browse/DRILL-4785 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.7.0 > Environment: Redhat EL6 >Reporter: Dechang Gu >Assignee: Venki Korukanti > Fix For: 1.8.0 > > > We noticed a bunch of limit 0 queries regressed quite a bit: +2500ms, while > the same queries took ~400ms in Apache Drill 1.6.0. 5-6X regression. Further > investigation indicates that most likely the root cause of the regression is > in the commit: > vkorukanti committed with vkorukanti DRILL-4446: Support mandatory work > assignment to endpoint requirement… > commit id: 10afc708600ea9f4cb0e7c2cd981b5b1001fea0d > With drill build on this commit, query takes 3095ms > and in the drillbit.log: > 2016-07-15 17:27:55,048 ucs-node2.perf.lab > [28768074-4ed6-a70a-2e6a-add3201ab801:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 28768074-4ed6-a70a-2e6a-add3201ab801: SELECT * FROM (SELECT > CAST(EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) AS > INTEGER) AS `mn_business_date_ok`,AVG((CASE WHEN ((CAST(EXTRACT(YEAR FROM > CAST(`rfm_sales`.`business_date` AS DATE)) AS INTEGER) = 2014) AND > (CAST((EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) - 1) / 3 > + 1 AS INTEGER) <= 4)) THEN `rfm_sales`.`pos_netsales` ELSE NULL END)) AS > `avg_Calculation_CIDBACJBCCCBHDGB_ok`,SUM((CASE WHEN ((CAST(EXTRACT(YEAR FROM > CAST(`rfm_sales`.`business_date` AS DATE)) AS INTEGER) = 2014) AND > (CAST((EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) - 1) / 3 > + 1 AS INTEGER) <= 4)) THEN `rfm_sales`.`pos_netsales` ELSE NULL END)) AS > `sum_Calculation_CIDBACJBCCCBHDGB_ok`,SUM((CASE WHEN ((CAST(EXTRACT(YEAR FROM > CAST(`rfm_sales`.`business_date` AS DATE)) AS INTEGER) = 2014) AND > (CAST((EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) - 1) / 3 > + 1 AS INTEGER) <= 4)) THEN 1 ELSE NULL END)) AS > `sum_Calculation_CJEBBAEBBFADBDFJ_ok`,SUM((CASE WHEN ((CAST(EXTRACT(YEAR FROM > CAST(`rfm_sales`.`business_date` AS DATE)) AS INTEGER) = 2014) AND > (CAST((EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) - 1) / 3 > + 1 AS INTEGER) <= 4)) THEN (`rfm_sales`.`pos_comps` + > `rfm_sales`.`pos_promos`) ELSE NULL END)) AS `sum_Net_Sales__YTD___copy__ok` > FROM `dfs.xxx`.`views/rfm_sales` `rfm_sales` GROUP BY CAST(EXTRACT(MONTH FROM > CAST(`rfm_sales`.`business_date` AS DATE)) AS INTEGER)) T LIMIT 0 > 2016-07-15 17:27:55,664 ucs-node2.perf.lab > [28768074-4ed6-a70a-2e6a-add3201ab801:foreman] INFO > o.a.d.exec.store.parquet.Metadata - Took 208 ms to read metadata from cache > file > 2016-07-15 17:27:56,783 ucs-node2.perf.lab > [28768074-4ed6-a70a-2e6a-add3201ab801:foreman] INFO > o.a.d.exec.store.parquet.Metadata - Took 129 ms to read metadata from cache > file > 2016-07-15 17:27:57,960 ucs-node2.perf.lab > [28768074-4ed6-a70a-2e6a-add3201ab801:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 28768074-4ed6-a70a-2e6a-add3201ab801:0:0: State change requested > AWAITING_ALLOCATION --> RUNNING > 2016-07-15 17:27:57,961 ucs-node2.perf.lab > [28768074-4ed6-a70a-2e6a-add3201ab801:frag:0:0] INFO > o.a.d.e.w.f.FragmentStatusReporter - > 28768074-4ed6-a70a-2e6a-add3201ab801:0:0: State to report: RUNNING > 2016-07-15 17:27:57,989 ucs-node2.perf.lab > [28768074-4ed6-a70a-2e6a-add3201ab801:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 28768074-4ed6-a70a-2e6a-add3201ab801:0:0: State change requested RUNNING --> > FINISHED > 2016-07-15 17:27:57,989 ucs-node2.perf.lab > [28768074-4ed6-a70a-2e6a-add3201ab801:frag:0:0] INFO > o.a.d.e.w.f.FragmentStatusReporter - > 28768074-4ed6-a70a-2e6a-add3201ab801:0:0: State to report: FINISHED > while running the same query on the parent commit (commit id > 9f4fff800d128878094ae70b454201f79976135d), it only takes 492ms. > and in the drillbit.log: > 2016-07-15 17:19:27,309 ucs-node7.perf.lab > [2876826f-ee19-9466-0c0c-869f47c409f8:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 2876826f-ee19-9466-0c0c-869f47c409f8: SELECT * FROM (SELECT > CAST(EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) AS > INTEGER) AS `mn_business_date_ok`,AVG((CASE WHEN ((CAST(EXTRACT(YEAR FROM > CAST(`rfm_sales`.`business_date` AS DATE)) AS INTEGER) = 2014) AND > (CAST((EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) - 1) / 3 > + 1 AS INTEGER) <= 4)) THEN `rfm_sales`.`pos_netsales` ELSE NULL END)) AS >
[jira] [Created] (DRILL-4785) Limit 0 queries regressed in Drill 1.7.0
Dechang Gu created DRILL-4785: - Summary: Limit 0 queries regressed in Drill 1.7.0 Key: DRILL-4785 URL: https://issues.apache.org/jira/browse/DRILL-4785 Project: Apache Drill Issue Type: Bug Components: Functions - Drill Affects Versions: 1.7.0 Environment: Redhat EL6 Reporter: Dechang Gu Assignee: Venki Korukanti Fix For: 1.8.0 We noticed a bunch of limit 0 queries regressed quite a bit: +2500ms, while the same queries took ~400ms in Apache Drill 1.6.0. 5-6X regression. Further investigation indicates that most likely the root cause of the regression is in the commit: vkorukanti committed with vkorukanti DRILL-4446: Support mandatory work assignment to endpoint requirement… commit id: 10afc708600ea9f4cb0e7c2cd981b5b1001fea0d With drill build on this commit, query takes 3095ms and in the drillbit.log: 2016-07-15 17:27:55,048 ucs-node2.perf.lab [28768074-4ed6-a70a-2e6a-add3201ab801:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 28768074-4ed6-a70a-2e6a-add3201ab801: SELECT * FROM (SELECT CAST(EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) AS INTEGER) AS `mn_business_date_ok`,AVG((CASE WHEN ((CAST(EXTRACT(YEAR FROM CAST(`rfm_sales`.`business_date` AS DATE)) AS INTEGER) = 2014) AND (CAST((EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) - 1) / 3 + 1 AS INTEGER) <= 4)) THEN `rfm_sales`.`pos_netsales` ELSE NULL END)) AS `avg_Calculation_CIDBACJBCCCBHDGB_ok`,SUM((CASE WHEN ((CAST(EXTRACT(YEAR FROM CAST(`rfm_sales`.`business_date` AS DATE)) AS INTEGER) = 2014) AND (CAST((EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) - 1) / 3 + 1 AS INTEGER) <= 4)) THEN `rfm_sales`.`pos_netsales` ELSE NULL END)) AS `sum_Calculation_CIDBACJBCCCBHDGB_ok`,SUM((CASE WHEN ((CAST(EXTRACT(YEAR FROM CAST(`rfm_sales`.`business_date` AS DATE)) AS INTEGER) = 2014) AND (CAST((EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) - 1) / 3 + 1 AS INTEGER) <= 4)) THEN 1 ELSE NULL END)) AS `sum_Calculation_CJEBBAEBBFADBDFJ_ok`,SUM((CASE WHEN ((CAST(EXTRACT(YEAR FROM CAST(`rfm_sales`.`business_date` AS DATE)) AS INTEGER) = 2014) AND (CAST((EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) - 1) / 3 + 1 AS INTEGER) <= 4)) THEN (`rfm_sales`.`pos_comps` + `rfm_sales`.`pos_promos`) ELSE NULL END)) AS `sum_Net_Sales__YTD___copy__ok` FROM `dfs.xxx`.`views/rfm_sales` `rfm_sales` GROUP BY CAST(EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) AS INTEGER)) T LIMIT 0 2016-07-15 17:27:55,664 ucs-node2.perf.lab [28768074-4ed6-a70a-2e6a-add3201ab801:foreman] INFO o.a.d.exec.store.parquet.Metadata - Took 208 ms to read metadata from cache file 2016-07-15 17:27:56,783 ucs-node2.perf.lab [28768074-4ed6-a70a-2e6a-add3201ab801:foreman] INFO o.a.d.exec.store.parquet.Metadata - Took 129 ms to read metadata from cache file 2016-07-15 17:27:57,960 ucs-node2.perf.lab [28768074-4ed6-a70a-2e6a-add3201ab801:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 28768074-4ed6-a70a-2e6a-add3201ab801:0:0: State change requested AWAITING_ALLOCATION --> RUNNING 2016-07-15 17:27:57,961 ucs-node2.perf.lab [28768074-4ed6-a70a-2e6a-add3201ab801:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 28768074-4ed6-a70a-2e6a-add3201ab801:0:0: State to report: RUNNING 2016-07-15 17:27:57,989 ucs-node2.perf.lab [28768074-4ed6-a70a-2e6a-add3201ab801:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 28768074-4ed6-a70a-2e6a-add3201ab801:0:0: State change requested RUNNING --> FINISHED 2016-07-15 17:27:57,989 ucs-node2.perf.lab [28768074-4ed6-a70a-2e6a-add3201ab801:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 28768074-4ed6-a70a-2e6a-add3201ab801:0:0: State to report: FINISHED while running the same query on the parent commit (commit id 9f4fff800d128878094ae70b454201f79976135d), it only takes 492ms. and in the drillbit.log: 2016-07-15 17:19:27,309 ucs-node7.perf.lab [2876826f-ee19-9466-0c0c-869f47c409f8:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 2876826f-ee19-9466-0c0c-869f47c409f8: SELECT * FROM (SELECT CAST(EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) AS INTEGER) AS `mn_business_date_ok`,AVG((CASE WHEN ((CAST(EXTRACT(YEAR FROM CAST(`rfm_sales`.`business_date` AS DATE)) AS INTEGER) = 2014) AND (CAST((EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) - 1) / 3 + 1 AS INTEGER) <= 4)) THEN `rfm_sales`.`pos_netsales` ELSE NULL END)) AS `avg_Calculation_CIDBACJBCCCBHDGB_ok`,SUM((CASE WHEN ((CAST(EXTRACT(YEAR FROM CAST(`rfm_sales`.`business_date` AS DATE)) AS INTEGER) = 2014) AND (CAST((EXTRACT(MONTH FROM CAST(`rfm_sales`.`business_date` AS DATE)) - 1) / 3 + 1 AS INTEGER) <= 4)) THEN `rfm_sales`.`pos_netsales` ELSE NULL END)) AS `sum_Calculation_CIDBACJBCCCBHDGB_ok`,SUM((CASE WHEN
[jira] [Closed] (DRILL-4237) Skew in hash distribution
[
https://issues.apache.org/jira/browse/DRILL-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu closed DRILL-4237.
-
Verified with TPCH regression test, no regression is observed.
> Skew in hash distribution
> -
>
> Key: DRILL-4237
> URL: https://issues.apache.org/jira/browse/DRILL-4237
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Aman Sinha
>Assignee: Chunhui Shi
> Fix For: 1.7.0
>
>
> Apparently, the fix in DRILL-4119 did not fully resolve the data skew issue.
> It worked fine on the smaller sample of the data set but on another sample of
> the same data set, it still produces skewed values - see below the hash
> values which are all odd numbers.
> {noformat}
> 0: jdbc:drill:zk=local> select columns[0], hash32(columns[0]) from `test.csv`
> limit 10;
> +---+--+
> | EXPR$0 |EXPR$1|
> +---+--+
> | f71aaddec3316ae18d43cb1467e88a41 | 1506011089 |
> | 3f3a13bb45618542b5ac9d9536704d3a | 1105719049 |
> | 6935afd0c693c67bba482cedb7a2919b | -18137557|
> | ca2a938d6d7e57bda40501578f98c2a8 | -1372666789 |
> | fab7f08402c8836563b0a5c94dbf0aec | -1930778239 |
> | 9eb4620dcb68a84d17209da279236431 | -970026001 |
> | 16eed4a4e801b98550b4ff504242961e | 356133757|
> | a46f7935fea578ce61d8dd45bfbc2b3d | -94010449|
> | 7fdf5344536080c15deb2b5a2975a2b7 | -141361507 |
> | b82560a06e2e51b461c9fe134a8211bd | -375376717 |
> +---+--+
> {noformat}
> This indicates an underlying issue with the XXHash64 java implementation,
> which is Drill's implementation of the C version. One of the key difference
> as pointed out by [~jnadeau] was the use of unsigned int64 in the C version
> compared to the Java version which uses (signed) long. I created an XXHash
> version using com.google.common.primitives.UnsignedLong. However,
> UnsignedLong does not have bit-wise operations that are needed for XXHash
> such as rotateLeft(), XOR etc. One could write wrappers for these but at
> this point, the question is: should we think of an alternative hash function
> ?
> The alternative approach could be the murmur hash for numeric data types that
> we were using earlier and the Mahout version of hash function for string
> types
> (https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/HashHelper.java#L28).
> As a test, I reverted to this function and was getting good hash
> distribution for the test data.
> I could not find any performance comparisons of our perf tests (TPC-H or DS)
> with the original and newer (XXHash) hash functions. If performance is
> comparable, should we revert to the original function ?
> As an aside, I would like to remove the hash64 versions of the functions
> since these are not used anywhere.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-4577) Improve performance for query on INFORMATION_SCHEMA when HIVE is plugged in
[
https://issues.apache.org/jira/browse/DRILL-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu closed DRILL-4577.
-
Verified and add test cases to perf test framework
> Improve performance for query on INFORMATION_SCHEMA when HIVE is plugged in
> ---
>
> Key: DRILL-4577
> URL: https://issues.apache.org/jira/browse/DRILL-4577
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Hive
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.7.0
>
>
> A query such as
> {code}
> select * from INFORMATION_SCHEMA.`TABLES`
> {code}
> is converted as calls to fetch all tables from storage plugins.
> When users have Hive, the calls to hive metadata storage would be:
> 1) get_table
> 2) get_partitions
> However, the information regarding partitions is not used in this type of
> queries. Beside, a more efficient way is to fetch tables is to use
> get_multi_table call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-4589) Reduce planning time for file system partition pruning by reducing filter evaluation overhead
[ https://issues.apache.org/jira/browse/DRILL-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu closed DRILL-4589. - Reviewed and verified. LGTM. > Reduce planning time for file system partition pruning by reducing filter > evaluation overhead > - > > Key: DRILL-4589 > URL: https://issues.apache.org/jira/browse/DRILL-4589 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > Fix For: 1.7.0 > > > When Drill is used to query hundreds of thousands, or even millions of files > organized into multi-level directories, user typically will provide a > partition filter like : dir0 = something and dir1 = something2 and .. . > For such queries, we saw the query planning time could be unacceptable long, > due to three main overheads: 1) to expand and get the list of files, 2) to > evaluate the partition filter, 3) to get the metadata, in the case of parquet > files for which metadata cache file is not available. > DRILL-2517 targets at the 3rd part of overhead. As a follow-up work after > DRILL-2517, we plan to reduce the filter evaluation overhead. For now, the > partition filter evaluation is applied to file level. In many cases, we saw > that the number of leaf subdirectories is significantly lower than that of > files. Since all the files under the same leaf subdirecctory share the same > directory metadata, we should apply the filter evaluation at the leaf > subdirectory. By doing that, we could reduce the cpu overhead to evaluate the > filter, and the memory overhead as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (DRILL-4589) Reduce planning time for file system partition pruning by reducing filter evaluation overhead
[ https://issues.apache.org/jira/browse/DRILL-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15259189#comment-15259189 ] Dechang Gu edited comment on DRILL-4589 at 4/27/16 2:51 AM: Verified on ucs cluster with the test case. Overall, we see average 2x improvement in explain plan time, some got 5x improvement, comparing with the fix and without the fix: 10-node cluster MapR 4.0 (1 master, 10 data nodes) - 32 cores, 256G RAM, 10 disks, 208K parquet files (each 12KB) in 3-level directory structure Query Without 4589 Apache Drill 1.7.0 master GitId 9514cbe Exec Time (ms) With 4589 Apache Drill 1.7.0 master GitId 9f4fff8 Exec Time (ms) with 4589 vs w/o 4589 Run 1 Run 2 Run 3 Avg Query Time Run 1 Run 2 Run 3Avg Query Time diff in avg wo4589/with4589 (avg) wo4589/with4589 (best) DRILL4589_EXPPLAN_0121478 15491 14784 17251 863484319673 891383381.941.75 DRILL4589_EXPPLAN_0219168 15560 15168 16632 10391 10665 8343 980068321.701.82 DRILL4589_EXPPLAN_0315478 13606 14506 14530 932384129520 908554451.601.62 DRILL4589_EXPPLAN_0418792 15311 14197 16100 856285257720 826978311.951.84 DRILL4589_EXPPLAN_0518447 14852 14692 15997 933386007874 860273951.861.87 DRILL4589_EXPPLAN_0618249 14619 15113 15994 944081339474 901669781.771.80 DRILL4589_EXPPLAN_0717213 15377 14132 15574 819678508066 803775371.941.80 DRILL4589_EXPPLAN_0815884 13808 16767 15486 880582127978 833271551.861.73 DRILL4589_EXPPLAN_0914810 15947 14151 14969 861284718847 864363261.731.67 DRILL4589_EXPPLAN_1015995 15373 16091 15820 954188798203 887469451.781.87 DRILL4589_EXPPLAN_1118722 18239 18828 18596 967780407883 853310063 2.182.31 DRILL4589_EXPPLAN_1216725 16246 16772 16581 844278888285 820583762.022.06 DRILL4589_EXPPLAN_1317063 13647 15686 15465 928480509015 878366821.761.70 DRILL4589_EXPPLAN_1414831 15107 14873 14937 895493368944 907858591.651.66 DRILL4589_EXPPLAN_1515170 15548 15166 15295 889787398891 884264521.731.74 DRILL4589_EXPPLAN_1644969 41579 41880 42809 10238 97779029 968133128 4.424.61 DRILL4589_EXPPLAN_1743389 42351 41345 42362 931579328323
[jira] [Commented] (DRILL-4589) Reduce planning time for file system partition pruning by reducing filter evaluation overhead
[ https://issues.apache.org/jira/browse/DRILL-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15259189#comment-15259189 ] Dechang Gu commented on DRILL-4589: --- Verified on ucs cluster with the test case. Overall, we see average 2x improvement in explain plan time, some got 5x improvement, comparing with the fix and without the fix: https://docs.google.com/spreadsheets/d/1dh7w8yvQ4fHTt0Bcb9xROqeXU80REjBoPZNSyff49Bg/edit#gid=0 UCS 10-node cluster MapR 4.0 (1 master, 10 data nodes) - 32 cores, 256G RAM, 10 disks, 208K parquet files (each 12KB) in 3-level directory structure Query Without 4589 Apache Drill 1.7.0 master GitId 9514cbe Exec Time (ms) With 4589 Apache Drill 1.7.0 master GitId 9f4fff8 Exec Time (ms) with 4589 vs w/o 4589 Run 1 Run 2 Run 3 Avg Query Time Run 1 Run 2 Run 3Avg Query Time diff in avg wo4589/with4589 (avg) wo4589/with4589 (best) DRILL4589_EXPPLAN_0121478 15491 14784 17251 863484319673 891383381.941.75 DRILL4589_EXPPLAN_0219168 15560 15168 16632 10391 10665 8343 980068321.701.82 DRILL4589_EXPPLAN_0315478 13606 14506 14530 932384129520 908554451.601.62 DRILL4589_EXPPLAN_0418792 15311 14197 16100 856285257720 826978311.951.84 DRILL4589_EXPPLAN_0518447 14852 14692 15997 933386007874 860273951.861.87 DRILL4589_EXPPLAN_0618249 14619 15113 15994 944081339474 901669781.771.80 DRILL4589_EXPPLAN_0717213 15377 14132 15574 819678508066 803775371.941.80 DRILL4589_EXPPLAN_0815884 13808 16767 15486 880582127978 833271551.861.73 DRILL4589_EXPPLAN_0914810 15947 14151 14969 861284718847 864363261.731.67 DRILL4589_EXPPLAN_1015995 15373 16091 15820 954188798203 887469451.781.87 DRILL4589_EXPPLAN_1118722 18239 18828 18596 967780407883 853310063 2.182.31 DRILL4589_EXPPLAN_1216725 16246 16772 16581 844278888285 820583762.022.06 DRILL4589_EXPPLAN_1317063 13647 15686 15465 928480509015 878366821.761.70 DRILL4589_EXPPLAN_1414831 15107 14873 14937 895493368944 907858591.651.66 DRILL4589_EXPPLAN_1515170 15548 15166 15295 889787398891 884264521.731.74 DRILL4589_EXPPLAN_1644969 41579 41880 42809 10238 97779029 968133128 4.424.61 DRILL4589_EXPPLAN_1743389
[jira] [Closed] (DRILL-4287) Do lazy reading of parquet metadata cache file
[ https://issues.apache.org/jira/browse/DRILL-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu closed DRILL-4287. - Verified, looks good. > Do lazy reading of parquet metadata cache file > -- > > Key: DRILL-4287 > URL: https://issues.apache.org/jira/browse/DRILL-4287 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Affects Versions: 1.4.0 >Reporter: Aman Sinha >Assignee: Aman Sinha > Fix For: 1.6.0 > > > Currently, the parquet metadata cache file is read eagerly during creation of > the DrillTable (as part of ParquetFormatMatcher.isReadable()). This is not > desirable from performance standpoint since there are scenarios where we want > to do some up-front optimizations - e.g. directory-based partition pruning > (see DRILL-2517) or potential limit 0 optimization etc. - and in such > situations it is better to do lazy reading of the metadata cache file. > This is a placeholder to perform such delayed reading since it is needed for > the aforementioned optimizations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-4279) Improve performance for skipAll query against Text/JSON/Parquet table
[
https://issues.apache.org/jira/browse/DRILL-4279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu closed DRILL-4279.
-
Verified, and it looks good.
> Improve performance for skipAll query against Text/JSON/Parquet table
> -
>
> Key: DRILL-4279
> URL: https://issues.apache.org/jira/browse/DRILL-4279
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
> Fix For: 1.6.0
>
>
> When query does not specify any specific column to be returned SCAN, for
> instance,
> {code}
> Q1: select count(*) from T1;
> Q2: select 1 + 100 from T1;
> Q3: select 1.0 + random() from T1;
> {code}
> Drill's planner would use a ColumnList with * column, plus a SKIP_ALL mode.
> However, the MODE is not serialized / deserialized. This leads to two
> problems.
> 1). The EXPLAIN plan is confusing, since there is no way to different from a
> "SELECT * " query from this SKIP_ALL mode.
> For instance,
> {code}
> explain plan for select count(*) from dfs.`/Users/jni/work/data/yelp/t1`;
> 00-03 Project($f0=[0])
> 00-04Scan(groupscan=[EasyGroupScan
> [selectionRoot=file:/Users/jni/work/data/yelp/t1, numFiles=2, columns=[`*`],
> files= ...
> {code}
> 2) If the query is to be executed distributed / parallel, the missing
> serialization of mode would means some Fragment is fetching all the columns,
> while some Fragment is skipping all the columns. That will cause execution
> error.
> For instance, by changing slice_target to enforce the query to be executed in
> multiple fragments, it will hit execution error.
> {code}
> select count(*) from dfs.`/Users/jni/work/data/yelp/t1`;
> org.apache.drill.common.exceptions.UserRemoteException: DATA_READ ERROR:
> Error parsing JSON - You tried to start when you are using a ValueWriter of
> type NullableBitWriterImpl.
> {code}
> Directory "t1" just contains two yelp JSON files.
> Ideally, I think when no columns is required from SCAN, the explain plan
> should show an empty of column list. The MODE of SKIP_ALL together with star
> * column seems to be confusing and error prone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-4194) Improve the performance of metadata fetch operation in HiveScan
[
https://issues.apache.org/jira/browse/DRILL-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu closed DRILL-4194.
-
Verified with the test case HIVE in drill perf test framework, comparing this
commit (bc74629) and the commit right before it (2953fe5), the planning time is
improved ~20x:
[root@ucs-node1 private-drill-perf-test-framework]# grep TOTAL
log/49_2953fe5_HIVE_20160413_094226/*/*log
log/49_2953fe5_HIVE_20160413_094226/HIVE_expPlan_01/HIVE_expPlan_01.log:[STAT]
TOTAL TIME : 185351 msec
log/49_2953fe5_HIVE_20160413_094226/HIVE_expPlan_01/HIVE_expPlan_01.log:[STAT]
TOTAL TIME : 163891 msec
log/49_2953fe5_HIVE_20160413_094226/HIVE_expPlan_01/HIVE_expPlan_01.log:[STAT]
TOTAL TIME : 165902 msec
log/49_2953fe5_HIVE_20160413_094226/HIVE_limit1_02/HIVE_limit1_02.log:[STAT]
TOTAL TIME : 137410 msec
log/49_2953fe5_HIVE_20160413_094226/HIVE_limit1_02/HIVE_limit1_02.log:[STAT]
TOTAL TIME : 165910 msec
log/49_2953fe5_HIVE_20160413_094226/HIVE_limit1_02/HIVE_limit1_02.log:[STAT]
TOTAL TIME : 164624 msec
with this fix:
[root@ucs-node1 private-drill-perf-test-framework]# grep TOTAL
log/50_bc74629_HIVE_20160413_100625/*/*log
log/50_bc74629_HIVE_20160413_100625/HIVE_expPlan_01/HIVE_expPlan_01.log:[STAT]
TOTAL TIME : 8094 msec
log/50_bc74629_HIVE_20160413_100625/HIVE_expPlan_01/HIVE_expPlan_01.log:[STAT]
TOTAL TIME : 6801 msec
log/50_bc74629_HIVE_20160413_100625/HIVE_expPlan_01/HIVE_expPlan_01.log:[STAT]
TOTAL TIME : 6166 msec
log/50_bc74629_HIVE_20160413_100625/HIVE_limit1_02/HIVE_limit1_02.log:[STAT]
TOTAL TIME : 11229 msec
log/50_bc74629_HIVE_20160413_100625/HIVE_limit1_02/HIVE_limit1_02.log:[STAT]
TOTAL TIME : 7533 msec
log/50_bc74629_HIVE_20160413_100625/HIVE_limit1_02/HIVE_limit1_02.log:[STAT]
TOTAL TIME : 7764 msec
> Improve the performance of metadata fetch operation in HiveScan
> ---
>
> Key: DRILL-4194
> URL: https://issues.apache.org/jira/browse/DRILL-4194
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.4.0
>Reporter: Venki Korukanti
>Assignee: Venki Korukanti
> Fix For: 1.5.0
>
>
> Current HiveScan fetches the InputSplits for all partitions when {{HiveScan}}
> is created. This causes long delays when the table contains large number of
> partitions. If we end up pruning majority of partitions, this delay is
> unnecessary.
> We need this InputSplits info from the beginning of planning because
> * it is used in calculating the cost of the {{HiveScan}}. Currently when
> calculating the cost first we look at the rowCount (from Hive MetaStore), if
> it is available we use it in cost calculation. Otherwise we estimate the
> rowCount from InputSplits.
> * We also need the InputSplits for determining whether {{HiveScan}} is a
> singleton or distributed for adding appropriate traits in {{ScanPrule}}
> Fix is to delay the loading of the InputSplits until we need. There are two
> cases where we need it. If we end up fetching the InputSplits, store them
> until the query completes.
> * If the stats are not available, then we need InputSplits
> * If the partition is not pruned we need it for parallelization purposes.
> Regarding getting the parallelization info in {{ScanPrule}}: Had a discussion
> with [~amansinha100]. All we need at this point is whether the data is
> distributed or singleton at this point. Added a method {{isSingleton()}} to
> GroupScan. Returning {{false}} seems to work fine for HiveScan, but I am not
> sure of the implications here. We also have {{ExcessiveExchangeIdentifier}}
> which removes unnecessary exchanges by looking at the parallelization info. I
> think it is ok to return the parallelization info here as the pruning must
> have already completed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4590) TPC-H q17 returns wrong results when applied to views
[ https://issues.apache.org/jira/browse/DRILL-4590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15230677#comment-15230677 ] Dechang Gu commented on DRILL-4590: --- 1.2.0 release also show the same wrong results. > TPC-H q17 returns wrong results when applied to views > - > > Key: DRILL-4590 > URL: https://issues.apache.org/jira/browse/DRILL-4590 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.6.0 > Environment: RHEL 6.4 2.6.32-358.el6.x86_64 >Reporter: Dechang Gu >Priority: Critical > > When run tpch queries on views created from parquet tables, query 17 returned > wrong results: > [root@ucs-node1 bugs]# /opt/mapr/drill/drill-1.6.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tpchViews" -f /tmp/TPCH_17.sql > 1/1 select > sum(l.l_extendedprice) / 7.0 as avg_yearly > from > lineitem l, > part p > where > p.p_partkey = l.l_partkey > and p.p_brand = 'Brand#13' > and p.p_container = 'JUMBO CAN' > and l.l_quantity < ( > select > 0.2 * avg(l2.l_quantity) > from > lineitem l2 > where > l2.l_partkey = p.p_partkey > ); > +-+ > | avg_yearly | > +-+ > | 1139490.7042857148 | > +-+ > 1 row selected (20.364 seconds) > While the same query directly on the parquet tables shows the correct results: > [root@ucs-node1 bugs]# /opt/mapr/drill/drill-1.6.0/bin/sqlline -u > "jdbc:drill:schema=dfs.parquet" -f /tmp/17_par100.q > 1/1 select > sum(l.l_extendedprice) / 7.0 as avg_yearly > from > lineitem_par100 l, > part_par100 p > where > p.p_partkey = l.l_partkey > and p.p_brand = 'Brand#13' > and p.p_container = 'JUMBO CAN' > and l.l_quantity < ( > select > 0.2 * avg(l2.l_quantity) > from > lineitem_par100 l2 > where > l2.l_partkey = p.p_partkey > ); > +--+ > | avg_yearly | > +--+ > | 3.237333813714285E7 | > +--+ > 1 row selected (25.266 seconds) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4590) TPC-H q17 returns wrong results when applied to views
[ https://issues.apache.org/jira/browse/DRILL-4590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15230535#comment-15230535 ] Dechang Gu commented on DRILL-4590: --- Just check the runs with 1.4.0 (r1) against views, the plan looks the same as that of 1.6.0 on the views. Will check 1.2.0 later. > TPC-H q17 returns wrong results when applied to views > - > > Key: DRILL-4590 > URL: https://issues.apache.org/jira/browse/DRILL-4590 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.6.0 > Environment: RHEL 6.4 2.6.32-358.el6.x86_64 >Reporter: Dechang Gu >Priority: Critical > > When run tpch queries on views created from parquet tables, query 17 returned > wrong results: > [root@ucs-node1 bugs]# /opt/mapr/drill/drill-1.6.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tpchViews" -f /tmp/TPCH_17.sql > 1/1 select > sum(l.l_extendedprice) / 7.0 as avg_yearly > from > lineitem l, > part p > where > p.p_partkey = l.l_partkey > and p.p_brand = 'Brand#13' > and p.p_container = 'JUMBO CAN' > and l.l_quantity < ( > select > 0.2 * avg(l2.l_quantity) > from > lineitem l2 > where > l2.l_partkey = p.p_partkey > ); > +-+ > | avg_yearly | > +-+ > | 1139490.7042857148 | > +-+ > 1 row selected (20.364 seconds) > While the same query directly on the parquet tables shows the correct results: > [root@ucs-node1 bugs]# /opt/mapr/drill/drill-1.6.0/bin/sqlline -u > "jdbc:drill:schema=dfs.parquet" -f /tmp/17_par100.q > 1/1 select > sum(l.l_extendedprice) / 7.0 as avg_yearly > from > lineitem_par100 l, > part_par100 p > where > p.p_partkey = l.l_partkey > and p.p_brand = 'Brand#13' > and p.p_container = 'JUMBO CAN' > and l.l_quantity < ( > select > 0.2 * avg(l2.l_quantity) > from > lineitem_par100 l2 > where > l2.l_partkey = p.p_partkey > ); > +--+ > | avg_yearly | > +--+ > | 3.237333813714285E7 | > +--+ > 1 row selected (25.266 seconds) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4590) Queries return wrong results when applied to views
[ https://issues.apache.org/jira/browse/DRILL-4590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15230441#comment-15230441 ] Dechang Gu commented on DRILL-4590: --- TPCH query 17 on views created from CSV files also shows the same wrong results. > Queries return wrong results when applied to views > -- > > Key: DRILL-4590 > URL: https://issues.apache.org/jira/browse/DRILL-4590 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.6.0 > Environment: RHEL 6.4 2.6.32-358.el6.x86_64 >Reporter: Dechang Gu >Priority: Critical > > When run tpch queries on views created from parquet tables, query 17 returned > wrong results: > [root@ucs-node1 bugs]# /opt/mapr/drill/drill-1.6.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tpchViews" -f /tmp/TPCH_17.sql > 1/1 select > sum(l.l_extendedprice) / 7.0 as avg_yearly > from > lineitem l, > part p > where > p.p_partkey = l.l_partkey > and p.p_brand = 'Brand#13' > and p.p_container = 'JUMBO CAN' > and l.l_quantity < ( > select > 0.2 * avg(l2.l_quantity) > from > lineitem l2 > where > l2.l_partkey = p.p_partkey > ); > +-+ > | avg_yearly | > +-+ > | 1139490.7042857148 | > +-+ > 1 row selected (20.364 seconds) > While the same query directly on the parquet tables shows the correct results: > [root@ucs-node1 bugs]# /opt/mapr/drill/drill-1.6.0/bin/sqlline -u > "jdbc:drill:schema=dfs.parquet" -f /tmp/17_par100.q > 1/1 select > sum(l.l_extendedprice) / 7.0 as avg_yearly > from > lineitem_par100 l, > part_par100 p > where > p.p_partkey = l.l_partkey > and p.p_brand = 'Brand#13' > and p.p_container = 'JUMBO CAN' > and l.l_quantity < ( > select > 0.2 * avg(l2.l_quantity) > from > lineitem_par100 l2 > where > l2.l_partkey = p.p_partkey > ); > +--+ > | avg_yearly | > +--+ > | 3.237333813714285E7 | > +--+ > 1 row selected (25.266 seconds) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4590) Queries return wrong results when applied to views
[
https://issues.apache.org/jira/browse/DRILL-4590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15230424#comment-15230424
]
Dechang Gu commented on DRILL-4590:
---
This issue can be easily reproduced as follows (using the drill built-in sample
database):
1. create the views:
0: jdbc:drill:schema=dfs.tpchViews> create or replace view cview as select
c_custkey, c_name, c_acctbal, c_nationkey from cp.`tpch/customer.parquet`;
+---+---+
| ok |summary|
+---+---+
| true | View 'cview' replaced successfully in 'dfs.tpchViews' schema |
+---+---+
1 row selected (0.286 seconds)
0: jdbc:drill:schema=dfs.tpchViews> create or replace view nview as select
n_nationkey, n_name, n_regionkey from cp.`tpch/nation.parquet`;
+---+---+
| ok |summary|
+---+---+
| true | View 'nview' replaced successfully in 'dfs.tpchViews' schema |
+---+---+
1 row selected (0.282 seconds)
2. Check the execution plans for the query on the views, which shows what
causes the wrong results:
0: jdbc:drill:schema=dfs.tpchViews> explain plan for select count(*) from cview
c, nview n where c.c_nationkey = n.n_nationkey and 10.0 > (select
min(c1.c_acctbal) from cview c1 where c1.c_nationkey = n.n_nationkey);
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(EXPR$0=[$0])
00-02StreamAgg(group=[{}], EXPR$0=[COUNT()])
00-03 Project($f0=[0])
00-04HashJoin(condition=[=($0, $1)], joinType=[inner])
00-06 Project(c_custkey=[$0])
00-08HashJoin(condition=[=($1, $2)], joinType=[inner])
00-11 Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=classpath:/tpch/customer.parquet]],
selectionRoot=classpath:/tpch/customer.parquet, numFiles=1,
usedMetadataFile=false, columns=[`c_custkey`, `c_nationkey`]]])
00-10 Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=classpath:/tpch/nation.parquet]],
selectionRoot=classpath:/tpch/nation.parquet, numFiles=1,
usedMetadataFile=false, columns=[`n_nationkey`]]])
00-05 Project(c_custkey0=[$0])
00-07SelectionVectorRemover
00-09 Filter(condition=[>(10.0, $1)])
00-12HashAgg(group=[{0}], EXPR$0=[MIN($1)])
00-13 Project(c_custkey0=[$2], c_acctbal=[$0])
00-14HashJoin(condition=[=($1, $2)], joinType=[inner])
00-16 Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=classpath:/tpch/customer.parquet]],
selectionRoot=classpath:/tpch/customer.parquet, numFiles=1,
usedMetadataFile=false, columns=[`c_acctbal`, `c_nationkey`]]])
00-15 HashAgg(group=[{0}])
00-17Project(c_custkey=[$0])
00-18 HashJoin(condition=[=($1, $2)],
joinType=[inner])
00-20Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=classpath:/tpch/customer.parquet]],
selectionRoot=classpath:/tpch/customer.parquet, numFiles=1,
usedMetadataFile=false, columns=[`c_custkey`, `c_nationkey`]]])
00-19Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=classpath:/tpch/nation.parquet]],
selectionRoot=classpath:/tpch/nation.parquet, numFiles=1,
usedMetadataFile=false, columns=[`n_nationkey`]]])
3. run the simple query against the views:
0: jdbc:drill:schema=dfs.tpchViews> select count(*) from cview c, nview n where
c.c_nationkey = n.n_nationkey and 10.0 > (select min(c1.c_acctbal) from cview
c1 where c1.c_nationkey = n.n_nationkey);
+-+
| EXPR$0 |
+-+
| 24 |
+-+
1 row selected (0.888 seconds)
4. run the same query against the parquet tables:
0: jdbc:drill:schema=dfs.tpchViews> select count(*) from
cp.`tpch/customer.parquet` c, cp.`tpch/nation.parquet` n where c.c_nationkey =
n.n_nationkey and 10.0 > (select min(c1.c_acctbal) from
cp.`tpch/customer.parquet` c1 where c1.c_nationkey = n.n_nationkey);
+-+
| EXPR$0 |
+-+
| 1500|
+-+
1 row selected (0.534 seconds)
and
5. show the plan on parquet tables:
0: jdbc:drill:schema=dfs.tpchViews> explain plan for select count(*) from
cp.`tpch/customer.parquet` c, cp.`tpch/nation.parquet` n where c.c_nationkey =
n.n_nationkey and 10.0 > (select min(c1.c_acctbal) from
cp.`tpch/customer.parquet` c1 where c1.c_nationkey =
[jira] [Created] (DRILL-4590) Queries return wrong results when applied to views
Dechang Gu created DRILL-4590: - Summary: Queries return wrong results when applied to views Key: DRILL-4590 URL: https://issues.apache.org/jira/browse/DRILL-4590 Project: Apache Drill Issue Type: Bug Components: Functions - Drill Affects Versions: 1.6.0 Environment: RHEL 6.4 2.6.32-358.el6.x86_64 Reporter: Dechang Gu Priority: Critical When run tpch queries on views created from parquet tables, query 17 returned wrong results: [root@ucs-node1 bugs]# /opt/mapr/drill/drill-1.6.0/bin/sqlline -u "jdbc:drill:schema=dfs.tpchViews" -f /tmp/TPCH_17.sql 1/1 select sum(l.l_extendedprice) / 7.0 as avg_yearly from lineitem l, part p where p.p_partkey = l.l_partkey and p.p_brand = 'Brand#13' and p.p_container = 'JUMBO CAN' and l.l_quantity < ( select 0.2 * avg(l2.l_quantity) from lineitem l2 where l2.l_partkey = p.p_partkey ); +-+ | avg_yearly | +-+ | 1139490.7042857148 | +-+ 1 row selected (20.364 seconds) While the same query directly on the parquet tables shows the correct results: [root@ucs-node1 bugs]# /opt/mapr/drill/drill-1.6.0/bin/sqlline -u "jdbc:drill:schema=dfs.parquet" -f /tmp/17_par100.q 1/1 select sum(l.l_extendedprice) / 7.0 as avg_yearly from lineitem_par100 l, part_par100 p where p.p_partkey = l.l_partkey and p.p_brand = 'Brand#13' and p.p_container = 'JUMBO CAN' and l.l_quantity < ( select 0.2 * avg(l2.l_quantity) from lineitem_par100 l2 where l2.l_partkey = p.p_partkey ); +--+ | avg_yearly | +--+ | 3.237333813714285E7 | +--+ 1 row selected (25.266 seconds) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-4380) Fix performance regression: in creation of FileSelection in ParquetFormatPlugin to not set files if metadata cache is available.
[
https://issues.apache.org/jira/browse/DRILL-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu closed DRILL-4380.
-
Verified. LGTM.
> Fix performance regression: in creation of FileSelection in
> ParquetFormatPlugin to not set files if metadata cache is available.
>
>
> Key: DRILL-4380
> URL: https://issues.apache.org/jira/browse/DRILL-4380
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Parth Chandra
> Fix For: 1.5.0
>
>
> The regression has been caused by the changes in
> 367d74a65ce2871a1452361cbd13bbd5f4a6cc95 (DRILL-2618: handle queries over
> empty folders consistently so that they report table not found rather than
> failing.)
> In ParquetFormatPlugin, the original code created a FileSelection object in
> the following code:
> {code}
> return new FileSelection(fileNames, metaRootPath.toString(), metadata,
> selection.getFileStatusList(fs));
> {code}
> The selection.getFileStatusList call made an inexpensive call to
> FileSelection.init(). The call was inexpensive because the
> FileSelection.files member was not set and the code does not need to make an
> expensive call to get the file statuses corresponding to the files in the
> FileSelection.files member.
> In the new code, this is replaced by
> {code}
> final FileSelection newSelection = FileSelection.create(null, fileNames,
> metaRootPath.toString());
> return ParquetFileSelection.create(newSelection, metadata);
> {code}
> This sets the FileSelection.files member but not the FileSelection.statuses
> member. A subsequent call to FileSelection.getStatuses ( in
> ParquetGroupScan() ) now makes an expensive call to get all the statuses.
> It appears that there was an implicit assumption that the
> FileSelection.statuses member should be set before the FileSelection.files
> member is set. This assumption is no longer true.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-4256) Performance regression in hive planning
[
https://issues.apache.org/jira/browse/DRILL-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu closed DRILL-4256.
-
Verified by Rahul.
> Performance regression in hive planning
> ---
>
> Key: DRILL-4256
> URL: https://issues.apache.org/jira/browse/DRILL-4256
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Hive, Query Planning & Optimization
>Affects Versions: 1.5.0
>Reporter: Rahul Challapalli
>Assignee: Venki Korukanti
> Fix For: 1.5.0
>
> Attachments: jstack.tgz
>
>
> Commit # : 76f41e18207e3e3e987fef56ee7f1695dd6ddd7a
> The fix for reading hive tables backed by hbase caused a performance
> regression. The data set used in the below test has ~3700 partitions and the
> filter in the query would ensure only 1 partition get selected.
> {code}
> Commit : 76f41e18207e3e3e987fef56ee7f1695dd6ddd7a
> Query : explain plan for select count(*) from lineitem_partitioned where
> `year`=2015 and `month`=1 and `day` =1;
> Time : ~25 seconds
> {code}
> {code}
> Commit : 1ea3d6c3f144614caf460648c1c27c6d0f5b06b8
> Query : explain plan for select count(*) from lineitem_partitioned where
> `year`=2015 and `month`=1 and `day` =1;
> Time : ~6.5 seconds
> {code}
> Since the data is large, I couldn't attach it here. Reach out to me if you
> need additional information.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-4235) Hit IllegalStateException when exec.queue.enable=ture
Dechang Gu created DRILL-4235: - Summary: Hit IllegalStateException when exec.queue.enable=ture Key: DRILL-4235 URL: https://issues.apache.org/jira/browse/DRILL-4235 Project: Apache Drill Issue Type: Bug Components: Functions - Drill Affects Versions: 1.5.0 Environment: git.commit.id=6dea429949a3d6a68aefbdb3d78de41e0955239b Reporter: Dechang Gu Assignee: Hanifi Gunes Priority: Critical Fix For: 1.5.0 0: jdbc:drill:schema=dfs.parquet> select * from sys.options; Error: SYSTEM ERROR: IllegalStateException: Failure trying to change states: ENQUEUED --> RUNNING [Error Id: 6ac8167c-6fb7-4274-9e5c-bf62a195c06e on ucs-node5.perf.lab:31010] (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception during fragment initialization: Exceptions caught during event processing org.apache.drill.exec.work.foreman.Foreman.run():261 java.util.concurrent.ThreadPoolExecutor.runWorker():1145 java.util.concurrent.ThreadPoolExecutor$Worker.run():615 java.lang.Thread.run():745 Caused By (java.lang.RuntimeException) Exceptions caught during event processing org.apache.drill.common.EventProcessor.sendEvent():93 org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState():792 org.apache.drill.exec.work.foreman.Foreman.moveToState():909 org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan():420 org.apache.drill.exec.work.foreman.Foreman.runSQL():926 org.apache.drill.exec.work.foreman.Foreman.run():250 java.util.concurrent.ThreadPoolExecutor.runWorker():1145 java.util.concurrent.ThreadPoolExecutor$Worker.run():615 java.lang.Thread.run():745 Caused By (java.lang.IllegalStateException) Failure trying to change states: ENQUEUED --> RUNNING org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent():896 org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent():790 org.apache.drill.common.EventProcessor.sendEvent():73 org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState():792 org.apache.drill.exec.work.foreman.Foreman.moveToState():909 org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan():420 org.apache.drill.exec.work.foreman.Foreman.runSQL():926 org.apache.drill.exec.work.foreman.Foreman.run():250 java.util.concurrent.ThreadPoolExecutor.runWorker():1145 java.util.concurrent.ThreadPoolExecutor$Worker.run():615 java.lang.Thread.run():745 (state=,code=0) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-4205) Simple query hit IndexOutOfBoundException
[
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu closed DRILL-4205.
-
Tested and verified for the following RPM:
http://yum.qa.lab/opensource/mapr-drill-1.4.0.201512211832-1.noarch.rpm
LGTM.
> Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Parth Chandra
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
> (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet
> record reader.
> Message:
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
> optional int64 ss_sold_date_sk;
> optional int64 ss_sold_time_sk;
> optional int64 ss_item_sk;
> optional int64 ss_customer_sk;
> optional int64 ss_cdemo_sk;
> optional int64 ss_hdemo_sk;
> optional int64 ss_addr_sk;
> optional int64 ss_store_sk;
> optional int64 ss_promo_sk;
> optional int64 ss_ticket_number;
> optional int64 ss_quantity;
> optional double ss_wholesale_cost;
> optional double ss_list_price;
> optional double ss_sales_price;
> optional double ss_ext_discount_amt;
> optional double ss_ext_sales_price;
> optional double ss_ext_wholesale_cost;
> optional double ss_ext_list_price;
> optional double ss_ext_tax;
> optional double ss_coupon_amt;
> optional double ss_net_paid;
> optional double ss_net_paid_inc_tax;
> optional double ss_net_profit;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-4205) Simple query hit IndexOutOfBoundException
[
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dechang Gu closed DRILL-4205.
-
Resolution: Fixed
LGTM.
> Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Dechang Gu
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
> (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet
> record reader.
> Message:
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
> optional int64 ss_sold_date_sk;
> optional int64 ss_sold_time_sk;
> optional int64 ss_item_sk;
> optional int64 ss_customer_sk;
> optional int64 ss_cdemo_sk;
> optional int64 ss_hdemo_sk;
> optional int64 ss_addr_sk;
> optional int64 ss_store_sk;
> optional int64 ss_promo_sk;
> optional int64 ss_ticket_number;
> optional int64 ss_quantity;
> optional double ss_wholesale_cost;
> optional double ss_list_price;
> optional double ss_sales_price;
> optional double ss_ext_discount_amt;
> optional double ss_ext_sales_price;
> optional double ss_ext_wholesale_cost;
> optional double ss_ext_list_price;
> optional double ss_ext_tax;
> optional double ss_coupon_amt;
> optional double ss_net_paid;
> optional double ss_net_paid_inc_tax;
> optional double ss_net_profit;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (DRILL-4205) Simple query hit IndexOutOfBoundException
[
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067182#comment-15067182
]
Dechang Gu edited comment on DRILL-4205 at 12/21/15 10:37 PM:
--
Parth, which repo is this fix ( abdfc6a ) in?I looked in your repo
https://github.com/parthchandra/incubator-drill/commits/DRILL-4205 , but could
not find this commit (the top commit id is
aaa6bb850b9de1e86afe2e0a5afad90d753fca04)
was (Author: dechanggu):
Parth, which repo is this fix ( abdfc6a ) in?I looked in your repo
https://github.com/parthchandra/incubator-drill/commits/DRILL-4205 , but could
not find this commit (the top commit id is
aaa6bb850b9de1e86afe2e0a5afad90d753fca04
> Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Parth Chandra
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
> (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet
> record reader.
> Message:
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
> optional int64 ss_sold_date_sk;
> optional int64 ss_sold_time_sk;
> optional int64 ss_item_sk;
> optional int64 ss_customer_sk;
> optional int64 ss_cdemo_sk;
> optional int64 ss_hdemo_sk;
> optional int64 ss_addr_sk;
> optional int64 ss_store_sk;
> optional int64 ss_promo_sk;
> optional int64 ss_ticket_number;
> optional int64 ss_quantity;
> optional double ss_wholesale_cost;
> optional double ss_list_price;
> optional double ss_sales_price;
> optional double ss_ext_discount_amt;
> optional double ss_ext_sales_price;
> optional double ss_ext_wholesale_cost;
> optional double ss_ext_list_price;
> optional double ss_ext_tax;
> optional double ss_coupon_amt;
> optional double ss_net_paid;
> optional double ss_net_paid_inc_tax;
> optional double ss_net_profit;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4205) Simple query hit IndexOutOfBoundException
[
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067182#comment-15067182
]
Dechang Gu commented on DRILL-4205:
---
Parth, which repo is this fix ( abdfc6a ) in?I looked in your repo
https://github.com/parthchandra/incubator-drill/commits/DRILL-4205 , but could
not find this commit (the top commit id is
aaa6bb850b9de1e86afe2e0a5afad90d753fca04
> Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Parth Chandra
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
> (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet
> record reader.
> Message:
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
> optional int64 ss_sold_date_sk;
> optional int64 ss_sold_time_sk;
> optional int64 ss_item_sk;
> optional int64 ss_customer_sk;
> optional int64 ss_cdemo_sk;
> optional int64 ss_hdemo_sk;
> optional int64 ss_addr_sk;
> optional int64 ss_store_sk;
> optional int64 ss_promo_sk;
> optional int64 ss_ticket_number;
> optional int64 ss_quantity;
> optional double ss_wholesale_cost;
> optional double ss_list_price;
> optional double ss_sales_price;
> optional double ss_ext_discount_amt;
> optional double ss_ext_sales_price;
> optional double ss_ext_wholesale_cost;
> optional double ss_ext_list_price;
> optional double ss_ext_tax;
> optional double ss_coupon_amt;
> optional double ss_net_paid;
> optional double ss_net_paid_inc_tax;
> optional double ss_net_profit;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4205) Simple query hit IndexOutOfBoundException
[
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067358#comment-15067358
]
Dechang Gu commented on DRILL-4205:
---
Found the patch in the Apache Drill master repo. And tested it -- looks likes
the issue is resolved.
Thanks,
Dechang
> Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Parth Chandra
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
> (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet
> record reader.
> Message:
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
> optional int64 ss_sold_date_sk;
> optional int64 ss_sold_time_sk;
> optional int64 ss_item_sk;
> optional int64 ss_customer_sk;
> optional int64 ss_cdemo_sk;
> optional int64 ss_hdemo_sk;
> optional int64 ss_addr_sk;
> optional int64 ss_store_sk;
> optional int64 ss_promo_sk;
> optional int64 ss_ticket_number;
> optional int64 ss_quantity;
> optional double ss_wholesale_cost;
> optional double ss_list_price;
> optional double ss_sales_price;
> optional double ss_ext_discount_amt;
> optional double ss_ext_sales_price;
> optional double ss_ext_wholesale_cost;
> optional double ss_ext_list_price;
> optional double ss_ext_tax;
> optional double ss_coupon_amt;
> optional double ss_net_paid;
> optional double ss_net_paid_inc_tax;
> optional double ss_net_profit;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.
[ https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu closed DRILL-3765. - Verified and tested. It looks fine. alter session set `planner.enable_hep_partition_pruning` = false; explain plan for select ss_sold_date_sk, ss_sold_time_sk, ss_item_sk, ss_customer_sk from store_pb_store_sk where ss_store_sk in (11,15, 7, 202, 278, 100, 200, 300, 400, 500) and ss_customer_sk = 96479; 1 row selected (140.404 seconds) alter session set `planner.enable_hep_partition_pruning` = true; explain plan for select ss_sold_date_sk, ss_sold_time_sk, ss_item_sk, ss_customer_sk from store_pb_store_sk where ss_store_sk in (11,15, 7, 202, 278, 100, 200, 300, 400, 500) and ss_customer_sk = 96479; 1 row selected (51.001 seconds) > Partition prune rule is unnecessary fired multiple times. > -- > > Key: DRILL-3765 > URL: https://issues.apache.org/jira/browse/DRILL-3765 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > Fix For: 1.4.0 > > > It seems that the partition prune rule may be fired multiple times, even > after the first rule execution has pushed the filter into the scan operator. > Since partition prune has to build the vectors to contain the partition /file > / directory information, to invoke the partition prune rule unnecessary may > lead to big memory overhead. > Drill planner should avoid the un-necessary partition prune rule, in order to > reduce the chance of hitting OOM exception, while the partition prune rule is > executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4205) Simple query hit IndexOutOfBoundException
[
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15060854#comment-15060854
]
Dechang Gu commented on DRILL-4205:
---
The parquet file can be downloaded here:
http://mapr-114551845.s3.amazonaws.com/list.html
> Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Parth Chandra
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
> (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet
> record reader.
> Message:
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
> optional int64 ss_sold_date_sk;
> optional int64 ss_sold_time_sk;
> optional int64 ss_item_sk;
> optional int64 ss_customer_sk;
> optional int64 ss_cdemo_sk;
> optional int64 ss_hdemo_sk;
> optional int64 ss_addr_sk;
> optional int64 ss_store_sk;
> optional int64 ss_promo_sk;
> optional int64 ss_ticket_number;
> optional int64 ss_quantity;
> optional double ss_wholesale_cost;
> optional double ss_list_price;
> optional double ss_sales_price;
> optional double ss_ext_discount_amt;
> optional double ss_ext_sales_price;
> optional double ss_ext_wholesale_cost;
> optional double ss_ext_list_price;
> optional double ss_ext_tax;
> optional double ss_coupon_amt;
> optional double ss_net_paid;
> optional double ss_net_paid_inc_tax;
> optional double ss_net_profit;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
