[
https://issues.apache.org/jira/browse/DRILL-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6441:
----------------------------------
Description:
This is seen when the drillbit is shutdown using $DRILL_HOME/bin/drillbit.sh
stop, it happens when BaseAllocator tries to close.
Apache Drill 1.14.0 commit c6c5d27d91468a29656bee2acba55d3321978aab
Adding details from Tim's email here
{noformat}
This issue is not related to DRILL-5922, however it is yet another way we leak
memory. Looking at the code when you view or submit a query via the rest api as
an anonymous user, a session is created with an allocator. If you call
drillbit.sh stop during this time, the Drillbit can shutdown while the
anonymous session is active and while a child allocator is being used. In this
case you will see the error you reported below.{noformat}
{noformat}
2018-05-11 15:19:44,510 [2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:0:0:
State to report: FINISHED
Wed May 16 10:17:13 PDT 2018 Terminating drillbit pid 32076
2018-05-16 10:17:13,793 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Received shutdown request.
2018-05-16 10:17:14,876 [Drillbit-ShutdownHook#0] INFO
o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 20, cache miss
count: 6, hit rate: 70%
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] ERROR
o.a.d.exec.server.BootStrapContext - Error while closing
java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child
allocators.
Allocator(ROOT) 0/0/9577600/34359738368 (res/actual/peak/limit)
child allocators: 8
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
ledgers: 0
reservations: 0
at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496)
~[drill-memory-base-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:259)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:263)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:363)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Shutdown completed (1095 ms).
{noformat}
was:
This is seen when the drillbit is shutdown using $DRILL_HOME/bin/drillbit.sh
stop, it happens when BaseAllocator tries to close.
Apache Drill 1.14.0 commit c6c5d27d91468a29656bee2acba55d3321978aab
{noformat}
2018-05-11 15:19:44,510 [2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:0:0:
State to report: FINISHED
Wed May 16 10:17:13 PDT 2018 Terminating drillbit pid 32076
2018-05-16 10:17:13,793 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Received shutdown request.
2018-05-16 10:17:14,876 [Drillbit-ShutdownHook#0] INFO
o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 20, cache miss
count: 6, hit rate: 70%
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] ERROR
o.a.d.exec.server.BootStrapContext - Error while closing
java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child
allocators.
Allocator(ROOT) 0/0/9577600/34359738368 (res/actual/peak/limit)
child allocators: 8
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
ledgers: 0
reservations: 0
at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496)
~[drill-memory-base-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:259)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:263)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:363)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Shutdown completed (1095 ms).
{noformat}
> IllegalStateException: Allocator[ROOT] closed with outstanding child
> allocators.
> --------------------------------------------------------------------------------
>
> Key: DRILL-6441
> URL: https://issues.apache.org/jira/browse/DRILL-6441
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
> Affects Versions: 1.14.0
> Reporter: Khurram Faraaz
> Priority: Major
>
> This is seen when the drillbit is shutdown using $DRILL_HOME/bin/drillbit.sh
> stop, it happens when BaseAllocator tries to close.
> Apache Drill 1.14.0 commit c6c5d27d91468a29656bee2acba55d3321978aab
> Adding details from Tim's email here
> {noformat}
> This issue is not related to DRILL-5922, however it is yet another way we
> leak memory. Looking at the code when you view or submit a query via the rest
> api as an anonymous user, a session is created with an allocator. If you call
> drillbit.sh stop during this time, the Drillbit can shutdown while the
> anonymous session is active and while a child allocator is being used. In
> this case you will see the error you reported below.{noformat}
>
> {noformat}
>
> 2018-05-11 15:19:44,510 [2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:0:0: State to report: FINISHED
> Wed May 16 10:17:13 PDT 2018 Terminating drillbit pid 32076
> 2018-05-16 10:17:13,793 [Drillbit-ShutdownHook#0] INFO
> o.apache.drill.exec.server.Drillbit - Received shutdown request.
> 2018-05-16 10:17:14,876 [Drillbit-ShutdownHook#0] INFO
> o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 20, cache miss
> count: 6, hit rate: 70%
> 2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] ERROR
> o.a.d.exec.server.BootStrapContext - Error while closing
> java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding
> child allocators.
> Allocator(ROOT) 0/0/9577600/34359738368 (res/actual/peak/limit)
> child allocators: 8
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> ledgers: 0
> reservations: 0
> at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496)
> ~[drill-memory-base-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
> [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
> [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:259)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
> [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
> [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:263)
> [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:363)
> [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> 2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] INFO
> o.apache.drill.exec.server.Drillbit - Shutdown completed (1095 ms).
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
