[jira] [Created] (DRILL-5077) Memory Leak -
Khurram Faraaz created DRILL-5077:
-
Summary: Memory Leak -
Key: DRILL-5077
URL: https://issues.apache.org/jira/browse/DRILL-5077
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.9.0
Environment: 4 node cluster CentOS
Reporter: Khurram Faraaz
Priority: Blocker
terminating foreman drillbit while a query is running results in memory leak
Drill 1.9.0 git commit id: 4312d65b
Stack trace from drillbit.log
{noformat}
2016-11-28 06:12:45,338 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
27c43522-79ca-989f-3659-d7ccbc77e2e7: select count(*) from `twoKeyJsn.json`
2016-11-28 06:12:45,602 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2016-11-28 06:12:45,602 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2016-11-28 06:12:45,603 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2016-11-28 06:12:45,603 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2016-11-28 06:12:45,603 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2016-11-28 06:12:45,633 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2016-11-28 06:12:45,669 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using
1 threads. Time: 33ms total, 33.494123ms avg, 33ms max.
2016-11-28 06:12:45,669 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using
1 threads. Earliest start: 9.54 μs, Latest start: 9.54 μs, Average
start: 9.54 μs .
2016-11-28 06:12:45,913 [27c43522-79ca-989f-3659-d7ccbc77e2e7:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 27c43522-79ca-989f-3659-d7ccbc77e2e7:0:0:
State change requested AWAITING_ALLOCATION --> RUNNING
2016-11-28 06:12:45,913 [27c43522-79ca-989f-3659-d7ccbc77e2e7:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 27c43522-79ca-989f-3659-d7ccbc77e2e7:0:0:
State to report: RUNNING
Mon Nov 28 06:12:48 UTC 2016 Terminating drillbit pid 28004
2016-11-28 06:12:48,697 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Received shutdown request.
2016-11-28 06:12:55,749 [pool-6-thread-2] INFO
o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup
io.netty.channel.nio.NioEventLoopGroup@15bcfacd in 1017 ms
2016-11-28 06:12:55,750 [pool-6-thread-2] INFO
o.a.drill.exec.service.ServiceEngine - closed dataPool in 1018 ms
2016-11-28 06:12:57,749 [Drillbit-ShutdownHook#0] WARN
o.apache.drill.exec.work.WorkManager - Closing WorkManager but there are 1
running fragments.
2016-11-28 06:12:57,751 [27c43522-79ca-989f-3659-d7ccbc77e2e7:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 27c43522-79ca-989f-3659-d7ccbc77e2e7:0:0:
State change requested RUNNING --> FAILED
2016-11-28 06:12:57,751 [27c43522-79ca-989f-3659-d7ccbc77e2e7:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 27c43522-79ca-989f-3659-d7ccbc77e2e7:0:0:
State change requested FAILED --> FINISHED
2016-11-28 06:12:57,756 [27c43522-79ca-989f-3659-d7ccbc77e2e7:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
Fragment 0:0
[Error Id: 2df2f9a1-a7bf-4454-a31b-717ab4ebd815 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
NullPointerException
Fragment 0:0
[Error Id: 2df2f9a1-a7bf-4454-a31b-717ab4ebd815 on centos-01.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
~[drill-common-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.9.0.jar:1.9.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_101]
at
[jira] [Created] (DRILL-5076) Provide sources, Javadoc for Drill-private forks of Parquet, etc.
Paul Rogers created DRILL-5076:
--
Summary: Provide sources, Javadoc for Drill-private forks of
Parquet, etc.
Key: DRILL-5076
URL: https://issues.apache.org/jira/browse/DRILL-5076
Project: Apache Drill
Issue Type: Improvement
Affects Versions: 1.8.0
Reporter: Paul Rogers
Priority: Minor
Drill uses its own forks of several projects, including Parquet.
>From {{drill-root/pom.xml}}:
{code}
1.8.1-drill-r0
{code}
>From {{drill-java-exec/pom.xml}}:
{code}
org.apache.parquet
parquet-hadoop
${parquet.version}
...
{code}
These appear to be hosted at Dremio. But, the sources seem to not be available.
From {{~/.m2/repository/org/apache/parquet/parquet-common/1.8.1-drill-r0}}:
{code}
$ ls
_remote.repositories
parquet-common-1.8.1-drill-r0.jar.lastUpdated
m2e-lastUpdated.properties
parquet-common-1.8.1-drill-r0.jar.sha1
parquet-common-1.8.1-drill-r0-javadoc.jar.lastUpdated
parquet-common-1.8.1-drill-r0.pom
parquet-common-1.8.1-drill-r0-sources.jar.lastUpdated
parquet-common-1.8.1-drill-r0.pom.lastUpdated
parquet-common-1.8.1-drill-r0.jar
parquet-common-1.8.1-drill-r0.pom.sha1
$ cat _remote.repositories
...
#Mon Oct 03 15:19:00 PDT 2016
parquet-common-1.8.1-drill-r0.jar>dremio-releases=
parquet-common-1.8.1-drill-r0.pom>dremio-releases=
{code}
This shows that Maven did attempt to download sources, but none where available.
The request is to provide sources (and Javadoc) for the Private Drill build.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5075) Tests complain about Parquet metadata parse errors in Drill-created files
Paul Rogers created DRILL-5075:
--
Summary: Tests complain about Parquet metadata parse errors in
Drill-created files
Key: DRILL-5075
URL: https://issues.apache.org/jira/browse/DRILL-5075
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.8.0
Reporter: Paul Rogers
Priority: Minor
The test {{TestParquetWriter.testAllScalarTypes}} seems to create a Parquet
file, then read it using the "new" Parquet reader. However, the test throws the
following assertion (though the test still succeeds.)
Note that the exception does _not_ occur if we run the single test function by
itself. It only occurs when run as part of the entire test class, suggesting an
interaction between tests.
When run stand-alone, another behavior occurs. When the test is complete, and
the Drillbit shuts down, only then does Parquet log a bunch of
"ColumnChunkPageWriteStore: written" messages followed by:
{code}
WARNING: org.apache.parquet.CorruptStatistics: Ignoring statistics because
created_by is null or empty! See PARQUET-251 and PARQUET-297
{code}
Are we leaving a file open that is getting flushed only on shut-down?
Full error when the test runs in the entire suite:
{code}
PM WARNING: org.apache.parquet.CorruptStatistics: Ignoring statistics because
created_by could not be parsed (see PARQUET-251): parquet-mr
org.apache.parquet.VersionParser$VersionParseException: Could not parse
created_by: parquet-mr using format: (.+) version ((.*) )?\(build ?(.*)\)
at org.apache.parquet.VersionParser.parse(VersionParser.java:112)
at
org.apache.parquet.CorruptStatistics.shouldIgnoreStatistics(CorruptStatistics.java:66)
at
org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:264)
at
org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:568)
at
org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:545)
at
org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:455)
at
org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:412)
at
org.apache.drill.exec.store.parquet.Metadata.getParquetFileMetadata_v3(Metadata.java:381)
at
org.apache.drill.exec.store.parquet.Metadata.access$0(Metadata.java:379)
at
org.apache.drill.exec.store.parquet.Metadata$MetadataGatherer.runInner(Metadata.java:316)
at
org.apache.drill.exec.store.parquet.Metadata$MetadataGatherer.runInner(Metadata.java:1)
at org.apache.drill.exec.store.TimedRunnable.run(TimedRunnable.java:56)
at org.apache.drill.exec.store.TimedRunnable.run(TimedRunnable.java:122)
at
org.apache.drill.exec.store.parquet.Metadata.getParquetFileMetadata_v3(Metadata.java:278)
at
org.apache.drill.exec.store.parquet.Metadata.getParquetTableMetadata(Metadata.java:257)
at
org.apache.drill.exec.store.parquet.Metadata.getParquetTableMetadata(Metadata.java:242)
at
org.apache.drill.exec.store.parquet.Metadata.getParquetTableMetadata(Metadata.java:118)
at
org.apache.drill.exec.store.parquet.ParquetGroupScan.init(ParquetGroupScan.java:733)
at
org.apache.drill.exec.store.parquet.ParquetGroupScan.(ParquetGroupScan.java:230)
at
org.apache.drill.exec.store.parquet.ParquetGroupScan.(ParquetGroupScan.java:190)
at
org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:169)
at
org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:1)
at
org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan(FileSystemPlugin.java:145)
at
org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:103)
at
org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:85)
at
org.apache.drill.exec.planner.logical.DrillPushProjIntoScan.onMatch(DrillPushProjIntoScan.java:65)
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
at
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:404)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:343)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:240)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:290)
at
[jira] [Created] (DRILL-5074) Excessive logging for permission denied error in CTAS
Paul Rogers created DRILL-5074:
--
Summary: Excessive logging for permission denied error in CTAS
Key: DRILL-5074
URL: https://issues.apache.org/jira/browse/DRILL-5074
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.8.0
Reporter: Paul Rogers
Priority: Minor
Consider the test
TestImpersonationMetadata.testCreateTableInWSWithNoPermissionsForQueryUser.
This test attempts to do a CTAS in a workspace without proper permissions. A
perfectly fine test for perfectly normal scenario. However, the test displays
the following very long log message for what is, in essence, a very simple
condition.
Expected that Drill would expect permission problems, would log a single
message ("User fred attempted to access Table foo without proper permissions"),
and omit the gory details of the stack trace. Since this is a normal operating
problem, and not a code fault, the stack trace simply clutters the log (and
test output.)
Note: the test is being modified to suppress the logging. If you attempt to
reproduce the problem using the test after that modification, comment out the
call to {{disableWriterLogging()}}.
{code}
22:01:29.890 [27c58945-c8c1-5c3d-6ebc-9926c33ced48:frag:0:0] ERROR
o.a.d.e.p.impl.WriterRecordBatch - Failure during query
org.apache.hadoop.security.AccessControlException: Permission denied:
user=drillTestUser2, access=WRITE,
inode="/drillTestGrp0_755/table1/0_0_0.parquet":fred:drillTestGrp0:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:320)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
...
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
user=drillTestUser2, access=WRITE,
inode="/drillTestGrp0_755/table1/0_0_0.parquet":fred:drillTestGrp0:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:320)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)
...
at com.sun.proxy.$Proxy67.create(Unknown Source) ~[na:na]
at
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1623)
~[hadoop-hdfs-2.7.1.jar:na]
... 35 common frames omitted
22:01:29.987 [27c58945-c8c1-5c3d-6ebc-9926c33ced48:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: RemoteException: Permission
denied: user=drillTestUser2, access=WRITE,
inode="/drillTestGrp0_755/table1/0_0_0.parquet":fred:drillTestGrp0:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:320)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)
...
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
Fragment 0:0
[Error Id: fff615a2-006d-4a33-a87c-f0d1c25d246f on 172.30.1.66:31013]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
RemoteException: Permission denied: user=drillTestUser2, access=WRITE,
inode="/drillTestGrp0_755/table1/0_0_0.parquet":fred:drillTestGrp0:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:320)
...
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
Fragment 0:0
[Error Id: fff615a2-006d-4a33-a87c-f0d1c25d246f on 172.30.1.66:31013]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
~[drill-common-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
...
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: org.apache.hadoop.security.AccessControlException: Permission
denied: user=drillTestUser2, access=WRITE,
inode="/drillTestGrp0_755/table1/0_0_0.parquet":fred:drillTestGrp0:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:320)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)
...
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226)
[classes/:na]
... 4 common frames omitted
Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
user=drillTestUser2, access=WRITE,
inode="/drillTestGrp0_755/table1/0_0_0.parquet":fred:drillTestGrp0:drwxr-xr-x
at
[jira] [Created] (DRILL-5073) Memory allocation failure in test DrillSeparatePlanningTest.testPlanning
Paul Rogers created DRILL-5073:
--
Summary: Memory allocation failure in test
DrillSeparatePlanningTest.testPlanning
Key: DRILL-5073
URL: https://issues.apache.org/jira/browse/DRILL-5073
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.8.0
Reporter: Paul Rogers
Run the test DrillSeparatePlanningTest.testPlanning. (Or, run the whole
DrillSeparatePlanningTest) test. You will see the exception shown below.
This is an error in the server, not reported to the client and therefore not
causing the test itself to fail. It is, however, a real error that must be
fixed.
{code}
[BitClient-2] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC
communication. Connection: /172.30.1.66:54074 <--> /172.30.1.66:31015 (data
client). Closing connection.
java.lang.IllegalStateException: Attempting operation on allocator when
allocator is closed.
Allocator(ROOT) 0/0/65688912/3817865216 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
at
org.apache.drill.exec.memory.BaseAllocator.assertOpen(BaseAllocator.java:108)
~[classes/:na]
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:189)
~[classes/:na]
at
org.apache.drill.exec.memory.DrillByteBufAllocator.buffer(DrillByteBufAllocator.java:49)
~[classes/:na]
at
org.apache.drill.exec.memory.DrillByteBufAllocator.ioBuffer(DrillByteBufAllocator.java:64)
~[classes/:na]
at
io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5072) REST API Returns String for Integers and all other fields
mustafa akin created DRILL-5072:
---
Summary: REST API Returns String for Integers and all other fields
Key: DRILL-5072
URL: https://issues.apache.org/jira/browse/DRILL-5072
Project: Apache Drill
Issue Type: Bug
Components: Client - HTTP
Affects Versions: 1.8.0
Reporter: mustafa akin
The Drill REST API currently returns all the fields as string values even they
are integers or floats. For instance, even explicit casting does not help as it
can be seen from the example:
```bash
$ curl -X POST -H "Content-Type: application/json" -d '{"queryType":"SQL",
"query": "select CAST (COUNT(*) AS INT) from dfs.`/myfile`"}'
http://localhost:8047/query.json
{
"columns" : [ "EXPR$0" ],
"rows" : [ {
"EXPR$0" : "1"
} ]
}
```
I would like to contribute myself if possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
