[jira] [Created] (DRILL-8455) mergejion memory leak when exception
shihuafeng created DRILL-8455: - Summary: mergejion memory leak when exception Key: DRILL-8455 URL: https://issues.apache.org/jira/browse/DRILL-8455 Project: Apache Drill Issue Type: Bug Affects Versions: 1.20.0 Reporter: shihuafeng when mergerjoin operator spilling data to disk ,it generate following exception: mergerjoin operator can not untimely invoke release() to free memory + {code:java} org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: External Sort encountered an error while spilling to disk{code} + -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8479) mergejion memory leak when exception
shihuafeng created DRILL-8479: - Summary: mergejion memory leak when exception Key: DRILL-8479 URL: https://issues.apache.org/jira/browse/DRILL-8479 Project: Apache Drill Issue Type: Bug Components: Execution - Relational Operators Affects Versions: 1.21.1 Reporter: shihuafeng Attachments: 0001-mergejoin-leak.patch *Describe the bug* megerjoin leak when RecordIterator allocate memory exception with OutOfMemoryException{*}{*} {*}Steps to reproduce the behavior{*}: # prepare data for tpch 1s # set direct memory 5g # set planner.enable_hashjoin =false to ensure use mergejoin operator。 # set drill.memory.debug.allocator =true (Check for memory leaks ) # 20 concurrent for tpch sql8 # when it had OutOfMemoryException or null EXCEPTION , stopped all sql. # finding memory leak *Expected behavior* when all sql sop , we should find direct memory is 0 AND could not find leak log like following. {code:java} Allocator(op:2:0:11:MergeJoinPOP) 100/73728/4874240/100 (res/actual/peak/limit){code} *Error detail, log output or screenshots* {code:java} Unable to allocate buffer of size XX (rounded from XX) due to memory limit (). Current allocation: xx{code} [^0001-mergejoin-leak.patch] sql {code:java} // code placeholder select o_year, sum(case when nation = 'CHINA' then volume else 0 end) / sum(volume) as mkt_share from ( select extract(year from o_orderdate) as o_year, l_extendedprice * 1.0 as volume, n2.n_name as nation from hive.tpch1s.part, hive.tpch1s.supplier, hive.tpch1s.lineitem, hive.tpch1s.orders, hive.tpch1s.customer, hive.tpch1s.nation n1, hive.tpch1s.nation n2, hive.tpch1s.region where p_partkey = l_partkey and s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey and c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name = 'ASIA' and s_nationkey = n2.n_nationkey and o_orderdate between date '1995-01-01' and date '1996-12-31' and p_type = 'LARGE BRUSHED BRASS') as all_nations group by o_year order by o_year {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8478) mergejion memory leak when exception
shihuafeng created DRILL-8478: - Summary: mergejion memory leak when exception Key: DRILL-8478 URL: https://issues.apache.org/jira/browse/DRILL-8478 Project: Apache Drill Issue Type: Bug Components: Execution - Relational Operators Affects Versions: 1.21.1 Reporter: shihuafeng Fix For: 1.21.2 *Describe the bug* hashpartition leak when allocate memory exception with OutOfMemoryException *To Reproduce* Steps to reproduce the behavior: # prepare data for tpch 1s # 20 concurrent for tpch sql8 # set direct memory 5g # when it had OutOfMemoryException , stopped all sql. # finding memory leak *Expected behavior* (1)i set \{DRILL_MAX_DIRECT_MEMORY:-"5G"} (2) i run sql8 (sql detail as Additional context) with 20 concurrent (3) it had OutOfMemoryException when create hashPartion *Error detail, log output or screenshots* Unable to allocate buffer of size 262144 (rounded from 262140) due to memory limit (41943040). Current allocation: 20447232 sql {code:java} // code placeholder {code} select o_year, sum(case when nation = 'CHINA' then volume else 0 end) / sum(volume) as mkt_share from ( select extract(year from o_orderdate) as o_year, l_extendedprice * 1.0 as volume, n2.n_name as nation from hive.tpch1s.part, hive.tpch1s.supplier, hive.tpch1s.lineitem, hive.tpch1s.orders, hive.tpch1s.customer, hive.tpch1s.nation n1, hive.tpch1s.nation n2, hive.tpch1s.region where p_partkey = l_partkey and s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey and c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name = 'ASIA' and s_nationkey = n2.n_nationkey and o_orderdate between date '1995-01-01' and date '1996-12-31' and p_type = 'LARGE BRUSHED BRASS') as all_nations group by o_year order by o_year -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8490) Sender operator fake memory leak result to sql failed or other exception
shihuafeng created DRILL-8490: - Summary: Sender operator fake memory leak result to sql failed or other exception Key: DRILL-8490 URL: https://issues.apache.org/jira/browse/DRILL-8490 Project: Apache Drill Issue Type: Bug Components: Server Affects Versions: 1.21.1 Reporter: shihuafeng Fix For: 1.22.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8488) HashJoinPOP memory leak is caused by an oom exception
shihuafeng created DRILL-8488: - Summary: HashJoinPOP memory leak is caused by an oom exception Key: DRILL-8488 URL: https://issues.apache.org/jira/browse/DRILL-8488 Project: Apache Drill Issue Type: Bug Components: Server Affects Versions: 1.21.1 Reporter: shihuafeng Fix For: 1.22.0 [DRILL-8485|[DRILL-8485] HashJoinPOP memory leak is caused by an oom exception when read data from InputStream - ASF JIRA (apache.org)] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8489) sender memory
shihuafeng created DRILL-8489: - Summary: sender memory Key: DRILL-8489 URL: https://issues.apache.org/jira/browse/DRILL-8489 Project: Apache Drill Issue Type: Bug Components: Server Affects Versions: 1.21.1 Reporter: shihuafeng Fix For: 1.22.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8483) SpilledRecordBatch memory leak when the program threw an exception during the process of building a hash table
shihuafeng created DRILL-8483: - Summary: SpilledRecordBatch memory leak when the program threw an exception during the process of building a hash table Key: DRILL-8483 URL: https://issues.apache.org/jira/browse/DRILL-8483 Project: Apache Drill Issue Type: Bug Components: Server Affects Versions: 1.21.1 Reporter: shihuafeng Fix For: 1.21.2 During the process of reading data from disk to building hash tables in memory, if an exception is thrown, it will result in a memory SpilledRecordBatch leak exception log as following {code:java} Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer of size 8192 due to memory limit (41943040). Current allocation: 3684352 at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:241) at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216) at org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:411) at org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:270) at org.apache.drill.exec.physical.impl.common.HashPartition.allocateNewVectorContainer(HashPartition.java:215) at org.apache.drill.exec.physical.impl.common.HashPartition.allocateNewCurrentBatchAndHV(HashPartition.java:238) at org.apache.drill.exec.physical.impl.common.HashPartition.(HashPartition.java:165){code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8484) An oom exception occurred When read data from Stream with container ,resulting in hashJoinPOP memory leak
shihuafeng created DRILL-8484: - Summary: An oom exception occurred When read data from Stream with container ,resulting in hashJoinPOP memory leak Key: DRILL-8484 URL: https://issues.apache.org/jira/browse/DRILL-8484 Project: Apache Drill Issue Type: Bug Components: Server Affects Versions: 1.21.1 Reporter: shihuafeng Fix For: 1.22.0 *Describe the bug* An oom exception occurred When read data from Stream with container ,resulting in hashJoinPOP memory leak *To Reproduce* prepare data for tpch 1s # 30 concurrent for tpch sql8 # set direct memory 5g # when it had OutOfMemoryException , stopped all sql. # finding memory leak *leak info* {code:java} Allocator(frag:5:0) 500/100/31067136/40041943040 (res/actual/peak/limit) child allocators: 1 Allocator(op:5:0:1:HashJoinPOP) 100/16384/22822912/41943040 (res/actual/peak/limit) child allocators: 0 ledgers: 2 ledger[1882757] allocator: op:5:0:1:HashJoinPOP), isOwning: true, size: 8192, references: 2, life: 16936270178816167..0, allocatorManager: [1703465, life: 16936270178813617..0] holds 4 buffers. DrillBuf[2041995], udle: [1703441 0..957]{code} execption info {code:java} Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer of size 16384 (rounded from 14359) due to memory limit (41943040). Current allocation: 22583616 at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:241) at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216) at org.apache.drill.exec.cache.VectorAccessibleSerializable.readFromStreamWithContainer(VectorAccessibleSerializable.java:172) at org.apache.drill.exec.physical.impl.aggregate.SpilledRecordBatch.next(SpilledRecordBatch.java:157) ... 18 common frames omitted{code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8485) HashJoinPOP memory leak is caused by an oom exception when read data from InputStream
shihuafeng created DRILL-8485: - Summary: HashJoinPOP memory leak is caused by an oom exception when read data from InputStream Key: DRILL-8485 URL: https://issues.apache.org/jira/browse/DRILL-8485 Project: Apache Drill Issue Type: Bug Components: Server Affects Versions: 1.21.1 Reporter: shihuafeng Fix For: 1.21.1 when traversing fieldList druing read data from InputStream, if the intermediate process throw exception,we can not release previously constructed vectors. it result in memory leak。 it is similar to [DRILL-8484] HashJoinPOP memory leak is caused by an oom exception when read data from Stream with container - ASF JIRA (apache.org) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8482) Assign region throw exception when some region is deployed on affinity node and some on non-affinity node
shihuafeng created DRILL-8482: - Summary: Assign region throw exception when some region is deployed on affinity node and some on non-affinity node Key: DRILL-8482 URL: https://issues.apache.org/jira/browse/DRILL-8482 Project: Apache Drill Issue Type: Bug Components: Storage - HBase Affects Versions: 1.21.1 Reporter: shihuafeng Fix For: 1.22.0 *Describe the bug* Assign region throw exception when some region is deployed on affinity node and some on non-affinity node。 *To Reproduce* Steps to reproduce the behavior: # {code:java} NavigableMap regionsToScan = Maps.newTreeMap(); regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[0], splits[1]), SERVER_A); regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[1], splits[2]), SERVER_A); regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[2], splits[3]), SERVER_B); regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[3], splits[4]), SERVER_B); regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[6], splits[7]), SERVER_D); regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[7], splits[8]), SERVER_D); regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[8], splits[9]), SERVER_D); regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[9], splits[10]), SERVER_D); final List endpoints = Lists.newArrayList(); endpoints.add(DrillbitEndpoint.newBuilder().setAddress(HOST_A).setControlPort(1234).build()); endpoints.add(DrillbitEndpoint.newBuilder().setAddress(HOST_B).setControlPort(1234).build()); endpoints.add(DrillbitEndpoint.newBuilder().setAddress(HOST_C).setControlPort(1234).build()); HBaseGroupScan scan = new HBaseGroupScan(); scan.setRegionsToScan(regionsToScan); scan.setHBaseScanSpec(new HBaseScanSpec(TABLE_NAME_STR, splits[0], splits[0], null)); scan.applyAssignments(endpoints);{code} *Expected behavior* *Error detail, log output or screenshots* {code:java} Caused by: java.lang.NullPointerException: null at org.apache.drill.exec.store.hbase.HBaseGroupScan.applyAssignments(HBaseGroupScan.java:283){code} -- This message was sent by Atlassian Jira (v8.20.10#820010)