[ https://issues.apache.org/jira/browse/ARROW-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975929#comment-15975929 ]
Julien Le Dem edited comment on ARROW-855 at 4/25/17 10:08 PM: --------------------------------------------------------------- Our programming language is scala. 1.Through HashMap to store pillar private val entries = new util.LinkedHashMap[PillarId, Pillar](32, 0.75f, true) 2.Pillar encapsulates the data structure of arrow {code} entries.put(pillarId, newPillar(getFieldDesc(pillarId.tableId, pillarId.colIdx))) def newPillar(fieldDesc: FieldDescription): Pillar = { ArrowPillar.getPillar(fieldDesc, root) } object ArrowPillar { @NeedTest def getPillar(field: FieldDescription, allocator: BufferAllocator): Pillar = { field.desc.getType match { case MemoType.Boolean => new ArrowNullableBoolPillar(field, allocator) case MemoType.Short => new ArrowNullableShortPillar(field, allocator) ...... } } } trait Pillar{ def append(value:Option[Any]):Unit ...... } {code} 3.The following is our free memory function {code} def remove(pillarId: PillarId): Boolean = { entries.synchronized { var pillar = entries.remove(pillarId) if (pillar != null) { pillar.close() pillar = null true } else { false } } } {code} was (Author: fushengxu): Our programming language is scala. 1.Through HashMap to store pillar private val entries = new util.LinkedHashMap[PillarId, Pillar](32, 0.75f, true) 2.Pillar encapsulates the data structure of arrow entries.put(pillarId, newPillar(getFieldDesc(pillarId.tableId, pillarId.colIdx))) def newPillar(fieldDesc: FieldDescription): Pillar = { ArrowPillar.getPillar(fieldDesc, root) } object ArrowPillar { @NeedTest def getPillar(field: FieldDescription, allocator: BufferAllocator): Pillar = { field.desc.getType match { case MemoType.Boolean => new ArrowNullableBoolPillar(field, allocator) case MemoType.Short => new ArrowNullableShortPillar(field, allocator) ...... } } } trait Pillar{ def append(value:Option[Any]):Unit ...... } 3.The following is our free memory function def remove(pillarId: PillarId): Boolean = { entries.synchronized { var pillar = entries.remove(pillarId) if (pillar != null) { pillar.close() pillar = null true } else { false } } } > Arrow Memory Leak > ----------------- > > Key: ARROW-855 > URL: https://issues.apache.org/jira/browse/ARROW-855 > Project: Apache Arrow > Issue Type: Bug > Components: Java - Memory, Java - Vectors > Affects Versions: 0.1.0 > Environment: CentOS release 6.7+Indellij IDEA > Reporter: xufusheng > Priority: Critical > Labels: test > > we create a memory table by arrow and the source data come from HBase. > Create a memory table and then drop the table,there will be a memory leak. > Hundreds of times,There will be OutOfMemoryError. > anyone encounter similar problems? -- This message was sent by Atlassian JIRA (v6.3.15#6346)