[jira] [Commented] (DRILL-8480) Cleanup before finished. 0 out of 1 streams have finished
[
https://issues.apache.org/jira/browse/DRILL-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17835829#comment-17835829
]
ASF GitHub Bot commented on DRILL-8480:
---
rymarm commented on PR #2897:
URL: https://github.com/apache/drill/pull/2897#issuecomment-2048081465
@cgivre Actually, there is one more thing, that I would fix in the scope of
this PR:
https://github.com/apache/drill/blob/a726a4544dfbf1427f41fb916d3d976bd511189b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/NestedLoopJoinBatch.java#L396-L399
These few lines seem to be kludge which may not work in some very rare
cases, but I forgot what issue occurs if remove it.
I would like to take a look at this, but I can do this in a separate PR
because the current issue is completely fixed with the current changes.
> Cleanup before finished. 0 out of 1 streams have finished
> -
>
> Key: DRILL-8480
> URL: https://issues.apache.org/jira/browse/DRILL-8480
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.21.1
>Reporter: Maksym Rymar
>Assignee: Maksym Rymar
>Priority: Major
> Attachments: 1a349ff1-d1f9-62bf-ed8c-26346c548005.sys.drill,
> tableWithNumber2.parquet
>
>
> Drill fails to execute a query with the following exception:
> {code:java}
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Cleanup before finished. 0 out of 1 streams have
> finished
> Fragment: 1:0
> Please, refer to logs for more information.
> [Error Id: 270da8f4-0bb6-4985-bf4f-34853138881c on
> compute7.vmcluster.com:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:657)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:395)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:245)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:362)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.lang.IllegalStateException: Cleanup before finished. 0 out of
> 1 streams have finished
> at
> org.apache.drill.exec.work.batch.BaseRawBatchBuffer.close(BaseRawBatchBuffer.java:111)
> at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:91)
> at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:71)
> at
> org.apache.drill.exec.work.batch.AbstractDataCollector.close(AbstractDataCollector.java:121)
> at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:91)
> at
> org.apache.drill.exec.work.batch.IncomingBuffers.close(IncomingBuffers.java:144)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:567)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:417)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:240)
> ... 5 common frames omitted
> Suppressed: java.lang.IllegalStateException: Cleanup before finished.
> 0 out of 1 streams have finished
> ... 15 common frames omitted
> Suppressed: java.lang.IllegalStateException: Memory was leaked by
> query. Memory leaked: (32768)
> Allocator(op:1:0:8:UnorderedReceiver) 100/32768/32768/100
> (res/actual/peak/limit)
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519)
> at
> org.apache.drill.exec.ops.BaseOperatorContext.close(BaseOperatorContext.java:159)
> at
> org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:77)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:571)
> ... 7 common frames omitted
> Suppressed: java.lang.IllegalStateException: Memory was leaked by
> query. Memory leaked: (1016640)
> Allocator(frag:1:0) 3000/1016640/30016640/90715827882
> (res/actual/peak/limit)
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519)
>
[jira] [Updated] (DRILL-8487) HTTP Caching
[ https://issues.apache.org/jira/browse/DRILL-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sena updated DRILL-8487: Description: I am using http storage plugin and I want to activate the caching. In the documentation it says that this requires adding cacheResults. So I added this to my config. When I test using an older version(1.20.1), I can see the query result files under the tmp/http-cache directory, but when I test using a newer version(1.21.1), there are no query result files in that directory, it only contains the journal. This PR [https://github.com/apache/drill/pull/2669] may have caused the issue. Also, is it possible to implement maximum cache size? was: I am using http storage plugin and I want to activate the caching. In the documentation it says that this requires adding cacheResults. So I added this to my config. When I test using an older version(1.20.1), I can see the query result files under the tmp/http-cache directory, but when I test using a newer version(1.21.1), there are no query result files in that directory, it only contains the journal. Also, is it possible to implement maximum cache size? > HTTP Caching > > > Key: DRILL-8487 > URL: https://issues.apache.org/jira/browse/DRILL-8487 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Sena >Priority: Major > Fix For: 1.20.1 > > > I am using http storage plugin and I want to activate the caching. In the > documentation it says that this requires adding cacheResults. So I added this > to my config. When I test using an older version(1.20.1), I can see the query > result files under the tmp/http-cache directory, but when I test using a > newer version(1.21.1), there are no query result files in that directory, it > only contains the journal. This PR > [https://github.com/apache/drill/pull/2669] may have caused the issue. > Also, is it possible to implement maximum cache size? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8487) HTTP Caching
Sena created DRILL-8487: --- Summary: HTTP Caching Key: DRILL-8487 URL: https://issues.apache.org/jira/browse/DRILL-8487 Project: Apache Drill Issue Type: Bug Affects Versions: 1.21.1 Reporter: Sena Fix For: 1.20.1 I am using http storage plugin and I want to activate the caching. In the documentation it says that this requires adding cacheResults. So I added this to my config. When I test using an older version(1.20.1), I can see the query result files under the tmp/http-cache directory, but when I test using a newer version(1.21.1), there are no query result files in that directory, it only contains the journal. Also, is it possible to implement maximum cache size? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8480) Cleanup before finished. 0 out of 1 streams have finished
[
https://issues.apache.org/jira/browse/DRILL-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17835737#comment-17835737
]
ASF GitHub Bot commented on DRILL-8480:
---
cgivre commented on PR #2897:
URL: https://github.com/apache/drill/pull/2897#issuecomment-2047593214
@rymarm. What is the status of this PR? Is it ready for merging?
> Cleanup before finished. 0 out of 1 streams have finished
> -
>
> Key: DRILL-8480
> URL: https://issues.apache.org/jira/browse/DRILL-8480
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.21.1
>Reporter: Maksym Rymar
>Assignee: Maksym Rymar
>Priority: Major
> Attachments: 1a349ff1-d1f9-62bf-ed8c-26346c548005.sys.drill,
> tableWithNumber2.parquet
>
>
> Drill fails to execute a query with the following exception:
> {code:java}
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Cleanup before finished. 0 out of 1 streams have
> finished
> Fragment: 1:0
> Please, refer to logs for more information.
> [Error Id: 270da8f4-0bb6-4985-bf4f-34853138881c on
> compute7.vmcluster.com:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:657)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:395)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:245)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:362)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.lang.IllegalStateException: Cleanup before finished. 0 out of
> 1 streams have finished
> at
> org.apache.drill.exec.work.batch.BaseRawBatchBuffer.close(BaseRawBatchBuffer.java:111)
> at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:91)
> at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:71)
> at
> org.apache.drill.exec.work.batch.AbstractDataCollector.close(AbstractDataCollector.java:121)
> at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:91)
> at
> org.apache.drill.exec.work.batch.IncomingBuffers.close(IncomingBuffers.java:144)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:567)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:417)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:240)
> ... 5 common frames omitted
> Suppressed: java.lang.IllegalStateException: Cleanup before finished.
> 0 out of 1 streams have finished
> ... 15 common frames omitted
> Suppressed: java.lang.IllegalStateException: Memory was leaked by
> query. Memory leaked: (32768)
> Allocator(op:1:0:8:UnorderedReceiver) 100/32768/32768/100
> (res/actual/peak/limit)
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519)
> at
> org.apache.drill.exec.ops.BaseOperatorContext.close(BaseOperatorContext.java:159)
> at
> org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:77)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:571)
> ... 7 common frames omitted
> Suppressed: java.lang.IllegalStateException: Memory was leaked by
> query. Memory leaked: (1016640)
> Allocator(frag:1:0) 3000/1016640/30016640/90715827882
> (res/actual/peak/limit)
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:574)
> ... 7 common frames omitted {code}
> Steps to reproduce:
> 1.Enable unequal join:
> {code:java}
> alter session set `planner.enable_nljoin_for_scalar_only`=false; {code}
> 2. Disable join optimization to prevent Drill from flipping sides of
> join
[jira] [Commented] (DRILL-8486) ParquetDecodingException: could not read bytes at offset
[
https://issues.apache.org/jira/browse/DRILL-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17835735#comment-17835735
]
ASF GitHub Bot commented on DRILL-8486:
---
cgivre merged PR #2898:
URL: https://github.com/apache/drill/pull/2898
> ParquetDecodingException: could not read bytes at offset
> -
>
> Key: DRILL-8486
> URL: https://issues.apache.org/jira/browse/DRILL-8486
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.21.1
>Reporter: Maksym Rymar
>Assignee: Maksym Rymar
>Priority: Major
> Attachments: test.parquet
>
>
> Drill fails to read a parquet file with the following exception:
>
> {code:java}
> Caused by: org.apache.parquet.io.ParquetDecodingException: could not read
> bytes at offset 591804
> at
> org.apache.parquet.column.values.plain.BinaryPlainValuesReader.readBytes(BinaryPlainValuesReader.java:42)
> at
> org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput$ValuesReaderWrapper.getNextEntry(VarLenColumnBulkInput.java:754)
> ... 43 common frames omitted
> Caused by: java.io.EOFException: null
> at
> org.apache.parquet.bytes.SingleBufferInputStream.read(SingleBufferInputStream.java:52)
> at
> org.apache.parquet.bytes.BytesUtils.readIntLittleEndian(BytesUtils.java:83)
> at
> org.apache.parquet.column.values.plain.BinaryPlainValuesReader.readBytes(BinaryPlainValuesReader.java:39)
> ... 44 common frames omitted {code}
>
>
> This issue only affects queries with {{store.parquet.flat.reader.bulk}} set
> to {{{}true{}}}(by default).
> Attaching the parquet file for the reproduce: [^test.parquet].
> Query: {{select log, app_name from dfs.tmp.`test.parquet`}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
