[jira] [Comment Edited] (ARROW-855) Arrow Memory Leak
[
https://issues.apache.org/jira/browse/ARROW-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002278#comment-16002278
]
Wes McKinney edited comment on ARROW-855 at 5/9/17 1:10 PM:
Hello,[~wesmckinn] [~julienledem],when elementSize = 12800,memory cannot be
release.
{color:red}
environment：
scala：scala-sdk-2.10.4
jdk:1.7.0_79
arrow:arrow-0.1.0
the following is the test code:
{color}
{code:language=scala}
import java.util.concurrent.ConcurrentHashMap
import org.apache.arrow.memory.{RootAllocator, BufferAllocator}
import org.apache.arrow.vector.NullableVarBinaryVector
import org.scalatest.{BeforeAndAfter, FunSuite}
import scala.util.Random
class ArrowLeakPureSuite extends FunSuite with BeforeAndAfter {
val root: BufferAllocator = new RootAllocator(1024 * 1024 * 1024 * 8L)
val elementCount = 1024
val elementSize = 12800 // when 25600, all is ok.
val binaGenerator = new BinaryGenerator(elementSize)
def pushVector(allocator: BufferAllocator, name: String, length: Int):
NullableVarBinaryVector = {
val vector = new NullableVarBinaryVector(name, allocator)
vector.allocateNew()
val mutator = vector.getMutator
(0 to length -1).foreach(i =>{
val value = binaGenerator.next()
mutator.setSafe(i, value, 0, value.length)
})
vector
}
test("leak") {
val vectorCount = 100
val entries = new ConcurrentHashMap[String, NullableVarBinaryVector]
(0 to vectorCount).foreach(i => {
val vector = pushVector(root,s"vector-$i", elementCount)
entries.put(s"vector-$i", vector)
})
// Push Finished.
println("Push Finished.")
Thread.sleep( 10 * 1000)
(0 to vectorCount).foreach(i => {
val vector = entries.get(s"vector-$i")
vector.close()
})
// Close Finished, now the data in Memory should be released, However it
didn't when elementSize = 12800.
println("Close Finished.")
Thread.sleep( 20 * 1000)
println("Test Finished.")
}
trait Generator[T] {
def next(): T
def nextStr(): String
}
abstract class GeneratorWithRandom[T] extends Generator[T] {
val random = new Random()
def nextStr() = next().toString
}
class BinaryGenerator(val length: Int) extends
GeneratorWithRandom[Array[Byte]] {
val binary = new Array[Byte](length)
def next() = {
random.nextBytes(binary)
binary
}
override def nextStr() = {
new String(binary)
}
}
}
{code}
was (Author: fushengxu):
Hello,[~wesmckinn] [~julienledem],when elementSize = 12800,memory cannot be
release.
{color:red}
environment：
scala：scala-sdk-2.10.4
jdk:1.7.0_79
arrow:arrow-0.1.0
the following is the test code:
{color}
import java.util.concurrent.ConcurrentHashMap
import org.apache.arrow.memory.{RootAllocator, BufferAllocator}
import org.apache.arrow.vector.NullableVarBinaryVector
import org.scalatest.{BeforeAndAfter, FunSuite}
import scala.util.Random
class ArrowLeakPureSuite extends FunSuite with BeforeAndAfter {
val root: BufferAllocator = new RootAllocator(1024 * 1024 * 1024 * 8L)
val elementCount = 1024
val elementSize = 12800 // when 25600, all is ok.
val binaGenerator = new BinaryGenerator(elementSize)
def pushVector(allocator: BufferAllocator, name: String, length: Int):
NullableVarBinaryVector = {
val vector = new NullableVarBinaryVector(name, allocator)
vector.allocateNew()
val mutator = vector.getMutator
(0 to length -1).foreach(i =>{
val value = binaGenerator.next()
mutator.setSafe(i, value, 0, value.length)
})
vector
}
test("leak") {
val vectorCount = 100
val entries = new ConcurrentHashMap[String, NullableVarBinaryVector]
(0 to vectorCount).foreach(i => {
val vector = pushVector(root,s"vector-$i", elementCount)
entries.put(s"vector-$i", vector)
})
// Push Finished.
println("Push Finished.")
Thread.sleep( 10 * 1000)
(0 to vectorCount).foreach(i => {
val vector = entries.get(s"vector-$i")
vector.close()
})
{color:red}
// Close Finished, now the data in Memory should be released, However it
didn't when elementSize = 12800.
{color}
println("Close Finished.")
Thread.sleep( 20 * 1000)
println("Test Finished.")
}
trait Generator[T] {
def next(): T
def nextStr(): String
}
abstract class GeneratorWithRandom[T] extends Generator[T] {
val random = new Random()
def nextStr() = next().toString
}
class BinaryGenerator(val length: Int) extends
GeneratorWithRandom[Array[Byte]] {
val binary = new Array[Byte](length)
def next() = {
random.nextBytes(binary)
binary
}
override def nextStr() = {
new String(binary)
}
}
}
> Arrow Memory Leak
> -
>
> Key: ARROW-855
> URL: https://issues.apache.org/jira/browse/ARROW-85
[jira] [Comment Edited] (ARROW-855) Arrow Memory Leak
[
https://issues.apache.org/jira/browse/ARROW-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002278#comment-16002278
]
xufusheng edited comment on ARROW-855 at 5/9/17 8:41 AM:
-
Hello,[~wesmckinn] [~julienledem],when elementSize = 12800,memory cannot be
release.
{color:red}
environment：
scala：scala-sdk-2.10.4
jdk:1.7.0_79
arrow:arrow-0.1.0
the following is the test code:
{color}
import java.util.concurrent.ConcurrentHashMap
import org.apache.arrow.memory.{RootAllocator, BufferAllocator}
import org.apache.arrow.vector.NullableVarBinaryVector
import org.scalatest.{BeforeAndAfter, FunSuite}
import scala.util.Random
class ArrowLeakPureSuite extends FunSuite with BeforeAndAfter {
val root: BufferAllocator = new RootAllocator(1024 * 1024 * 1024 * 8L)
val elementCount = 1024
val elementSize = 12800 // when 25600, all is ok.
val binaGenerator = new BinaryGenerator(elementSize)
def pushVector(allocator: BufferAllocator, name: String, length: Int):
NullableVarBinaryVector = {
val vector = new NullableVarBinaryVector(name, allocator)
vector.allocateNew()
val mutator = vector.getMutator
(0 to length -1).foreach(i =>{
val value = binaGenerator.next()
mutator.setSafe(i, value, 0, value.length)
})
vector
}
test("leak") {
val vectorCount = 100
val entries = new ConcurrentHashMap[String, NullableVarBinaryVector]
(0 to vectorCount).foreach(i => {
val vector = pushVector(root,s"vector-$i", elementCount)
entries.put(s"vector-$i", vector)
})
// Push Finished.
println("Push Finished.")
Thread.sleep( 10 * 1000)
(0 to vectorCount).foreach(i => {
val vector = entries.get(s"vector-$i")
vector.close()
})
{color:red}
// Close Finished, now the data in Memory should be released, However it
didn't when elementSize = 12800.
{color}
println("Close Finished.")
Thread.sleep( 20 * 1000)
println("Test Finished.")
}
trait Generator[T] {
def next(): T
def nextStr(): String
}
abstract class GeneratorWithRandom[T] extends Generator[T] {
val random = new Random()
def nextStr() = next().toString
}
class BinaryGenerator(val length: Int) extends
GeneratorWithRandom[Array[Byte]] {
val binary = new Array[Byte](length)
def next() = {
random.nextBytes(binary)
binary
}
override def nextStr() = {
new String(binary)
}
}
}
was (Author: fushengxu):
[~wesmckinn] [~julienledem]
{color:red}
environment：
scala：scala-sdk-2.10.4
jdk:1.7.0_79
arrow:arrow-0.1.0
the following is the test code:
{color}
import java.util.concurrent.ConcurrentHashMap
import org.apache.arrow.memory.{RootAllocator, BufferAllocator}
import org.apache.arrow.vector.NullableVarBinaryVector
import org.scalatest.{BeforeAndAfter, FunSuite}
import scala.util.Random
class ArrowLeakPureSuite extends FunSuite with BeforeAndAfter {
val root: BufferAllocator = new RootAllocator(1024 * 1024 * 1024 * 8L)
val elementCount = 1024
val elementSize = 12800 // when 25600, all is ok.
val binaGenerator = new BinaryGenerator(elementSize)
def pushVector(allocator: BufferAllocator, name: String, length: Int):
NullableVarBinaryVector = {
val vector = new NullableVarBinaryVector(name, allocator)
vector.allocateNew()
val mutator = vector.getMutator
(0 to length -1).foreach(i =>{
val value = binaGenerator.next()
mutator.setSafe(i, value, 0, value.length)
})
vector
}
test("leak") {
val vectorCount = 100
val entries = new ConcurrentHashMap[String, NullableVarBinaryVector]
(0 to vectorCount).foreach(i => {
val vector = pushVector(root,s"vector-$i", elementCount)
entries.put(s"vector-$i", vector)
})
// Push Finished.
println("Push Finished.")
Thread.sleep( 10 * 1000)
(0 to vectorCount).foreach(i => {
val vector = entries.get(s"vector-$i")
vector.close()
})
{color:red}
// Close Finished, now the data in Memory should be released, However it
didn't when elementSize = 12800.
{color}
println("Close Finished.")
Thread.sleep( 20 * 1000)
println("Test Finished.")
}
trait Generator[T] {
def next(): T
def nextStr(): String
}
abstract class GeneratorWithRandom[T] extends Generator[T] {
val random = new Random()
def nextStr() = next().toString
}
class BinaryGenerator(val length: Int) extends
GeneratorWithRandom[Array[Byte]] {
val binary = new Array[Byte](length)
def next() = {
random.nextBytes(binary)
binary
}
override def nextStr() = {
new String(binary)
}
}
}
> Arrow Memory Leak
> -
>
> Key: ARROW-855
> URL: https://issues.apache.org/jira/browse/ARROW-855
> Project: Apache Arrow
> Issue Type: Bug
>
[jira] [Comment Edited] (ARROW-855) Arrow Memory Leak
[
https://issues.apache.org/jira/browse/ARROW-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002278#comment-16002278
]
xufusheng edited comment on ARROW-855 at 5/9/17 8:30 AM:
-
[~wesmckinn] [~julienledem]
{color:red}
environment：
scala：scala-sdk-2.10.4
jdk:1.7.0_79
arrow:arrow-0.1.0
the following is the test code:
{color}
import java.util.concurrent.ConcurrentHashMap
import org.apache.arrow.memory.{RootAllocator, BufferAllocator}
import org.apache.arrow.vector.NullableVarBinaryVector
import org.scalatest.{BeforeAndAfter, FunSuite}
import scala.util.Random
class ArrowLeakPureSuite extends FunSuite with BeforeAndAfter {
val root: BufferAllocator = new RootAllocator(1024 * 1024 * 1024 * 8L)
val elementCount = 1024
val elementSize = 12800 // when 25600, all is ok.
val binaGenerator = new BinaryGenerator(elementSize)
def pushVector(allocator: BufferAllocator, name: String, length: Int):
NullableVarBinaryVector = {
val vector = new NullableVarBinaryVector(name, allocator)
vector.allocateNew()
val mutator = vector.getMutator
(0 to length -1).foreach(i =>{
val value = binaGenerator.next()
mutator.setSafe(i, value, 0, value.length)
})
vector
}
test("leak") {
val vectorCount = 100
val entries = new ConcurrentHashMap[String, NullableVarBinaryVector]
(0 to vectorCount).foreach(i => {
val vector = pushVector(root,s"vector-$i", elementCount)
entries.put(s"vector-$i", vector)
})
// Push Finished.
println("Push Finished.")
Thread.sleep( 10 * 1000)
(0 to vectorCount).foreach(i => {
val vector = entries.get(s"vector-$i")
vector.close()
})
{color:red}
// Close Finished, now the data in Memory should be released, However it
didn't when elementSize = 12800.
{color}
println("Close Finished.")
Thread.sleep( 20 * 1000)
println("Test Finished.")
}
trait Generator[T] {
def next(): T
def nextStr(): String
}
abstract class GeneratorWithRandom[T] extends Generator[T] {
val random = new Random()
def nextStr() = next().toString
}
class BinaryGenerator(val length: Int) extends
GeneratorWithRandom[Array[Byte]] {
val binary = new Array[Byte](length)
def next() = {
random.nextBytes(binary)
binary
}
override def nextStr() = {
new String(binary)
}
}
}
was (Author: fushengxu):
{color:red}
environment：
scala：scala-sdk-2.10.4
jdk:1.7.0_79
arrow:arrow-0.1.0
the following is the test code:
{color}
import java.util.concurrent.ConcurrentHashMap
import org.apache.arrow.memory.{RootAllocator, BufferAllocator}
import org.apache.arrow.vector.NullableVarBinaryVector
import org.scalatest.{BeforeAndAfter, FunSuite}
import scala.util.Random
class ArrowLeakPureSuite extends FunSuite with BeforeAndAfter {
val root: BufferAllocator = new RootAllocator(1024 * 1024 * 1024 * 8L)
val elementCount = 1024
val elementSize = 12800 // when 25600, all is ok.
val binaGenerator = new BinaryGenerator(elementSize)
def pushVector(allocator: BufferAllocator, name: String, length: Int):
NullableVarBinaryVector = {
val vector = new NullableVarBinaryVector(name, allocator)
vector.allocateNew()
val mutator = vector.getMutator
(0 to length -1).foreach(i =>{
val value = binaGenerator.next()
mutator.setSafe(i, value, 0, value.length)
})
vector
}
test("leak") {
val vectorCount = 100
val entries = new ConcurrentHashMap[String, NullableVarBinaryVector]
(0 to vectorCount).foreach(i => {
val vector = pushVector(root,s"vector-$i", elementCount)
entries.put(s"vector-$i", vector)
})
// Push Finished.
println("Push Finished.")
Thread.sleep( 10 * 1000)
(0 to vectorCount).foreach(i => {
val vector = entries.get(s"vector-$i")
vector.close()
})
{color:red}
// Close Finished, now the data in Memory should be released, However it
didn't when elementSize = 12800.
{color}
println("Close Finished.")
Thread.sleep( 20 * 1000)
println("Test Finished.")
}
trait Generator[T] {
def next(): T
def nextStr(): String
}
abstract class GeneratorWithRandom[T] extends Generator[T] {
val random = new Random()
def nextStr() = next().toString
}
class BinaryGenerator(val length: Int) extends
GeneratorWithRandom[Array[Byte]] {
val binary = new Array[Byte](length)
def next() = {
random.nextBytes(binary)
binary
}
override def nextStr() = {
new String(binary)
}
}
}
> Arrow Memory Leak
> -
>
> Key: ARROW-855
> URL: https://issues.apache.org/jira/browse/ARROW-855
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java - Memory, Java - Vectors
>Affects Versions: 0.1.0
> Envi
[jira] [Comment Edited] (ARROW-855) Arrow Memory Leak
[
https://issues.apache.org/jira/browse/ARROW-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002278#comment-16002278
]
xufusheng edited comment on ARROW-855 at 5/9/17 8:28 AM:
-
{color:red}
environment：
scala：scala-sdk-2.10.4
jdk:1.7.0_79
arrow:arrow-0.1.0
the following is the test code:
{color}
import java.util.concurrent.ConcurrentHashMap
import org.apache.arrow.memory.{RootAllocator, BufferAllocator}
import org.apache.arrow.vector.NullableVarBinaryVector
import org.scalatest.{BeforeAndAfter, FunSuite}
import scala.util.Random
class ArrowLeakPureSuite extends FunSuite with BeforeAndAfter {
val root: BufferAllocator = new RootAllocator(1024 * 1024 * 1024 * 8L)
val elementCount = 1024
val elementSize = 12800 // when 25600, all is ok.
val binaGenerator = new BinaryGenerator(elementSize)
def pushVector(allocator: BufferAllocator, name: String, length: Int):
NullableVarBinaryVector = {
val vector = new NullableVarBinaryVector(name, allocator)
vector.allocateNew()
val mutator = vector.getMutator
(0 to length -1).foreach(i =>{
val value = binaGenerator.next()
mutator.setSafe(i, value, 0, value.length)
})
vector
}
test("leak") {
val vectorCount = 100
val entries = new ConcurrentHashMap[String, NullableVarBinaryVector]
(0 to vectorCount).foreach(i => {
val vector = pushVector(root,s"vector-$i", elementCount)
entries.put(s"vector-$i", vector)
})
// Push Finished.
println("Push Finished.")
Thread.sleep( 10 * 1000)
(0 to vectorCount).foreach(i => {
val vector = entries.get(s"vector-$i")
vector.close()
})
{color:red}
// Close Finished, now the data in Memory should be released, However it
didn't when elementSize = 12800.
{color}
println("Close Finished.")
Thread.sleep( 20 * 1000)
println("Test Finished.")
}
trait Generator[T] {
def next(): T
def nextStr(): String
}
abstract class GeneratorWithRandom[T] extends Generator[T] {
val random = new Random()
def nextStr() = next().toString
}
class BinaryGenerator(val length: Int) extends
GeneratorWithRandom[Array[Byte]] {
val binary = new Array[Byte](length)
def next() = {
random.nextBytes(binary)
binary
}
override def nextStr() = {
new String(binary)
}
}
}
was (Author: fushengxu):
{color:red}
environment：
scala：scala-sdk-2.10.4
jdk:1.7.0_79
arrow:arrow-0.1.0
{color}
_ the following is the test code: _
import java.util.concurrent.ConcurrentHashMap
import org.apache.arrow.memory.{RootAllocator, BufferAllocator}
import org.apache.arrow.vector.NullableVarBinaryVector
import org.scalatest.{BeforeAndAfter, FunSuite}
import scala.util.Random
class ArrowLeakPureSuite extends FunSuite with BeforeAndAfter {
val root: BufferAllocator = new RootAllocator(1024 * 1024 * 1024 * 8L)
val elementCount = 1024
val elementSize = 12800 // when 25600, all is ok.
val binaGenerator = new BinaryGenerator(elementSize)
def pushVector(allocator: BufferAllocator, name: String, length: Int):
NullableVarBinaryVector = {
val vector = new NullableVarBinaryVector(name, allocator)
vector.allocateNew()
val mutator = vector.getMutator
(0 to length -1).foreach(i =>{
val value = binaGenerator.next()
mutator.setSafe(i, value, 0, value.length)
})
vector
}
test("leak") {
val vectorCount = 100
val entries = new ConcurrentHashMap[String, NullableVarBinaryVector]
(0 to vectorCount).foreach(i => {
val vector = pushVector(root,s"vector-$i", elementCount)
entries.put(s"vector-$i", vector)
})
// Push Finished.
println("Push Finished.")
Thread.sleep( 10 * 1000)
(0 to vectorCount).foreach(i => {
val vector = entries.get(s"vector-$i")
vector.close()
})
_ // Close Finished, now the data in Memory should be released, However it
didn't when elementSize = 12800._
println("Close Finished.")
Thread.sleep( 20 * 1000)
println("Test Finished.")
}
trait Generator[T] {
def next(): T
def nextStr(): String
}
abstract class GeneratorWithRandom[T] extends Generator[T] {
val random = new Random()
def nextStr() = next().toString
}
class BinaryGenerator(val length: Int) extends
GeneratorWithRandom[Array[Byte]] {
val binary = new Array[Byte](length)
def next() = {
random.nextBytes(binary)
binary
}
override def nextStr() = {
new String(binary)
}
}
}
> Arrow Memory Leak
> -
>
> Key: ARROW-855
> URL: https://issues.apache.org/jira/browse/ARROW-855
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java - Memory, Java - Vectors
>Affects Versions: 0.1.0
> Environment: CentOS release 6.7+Indellij IDEA
>
[jira] [Comment Edited] (ARROW-855) Arrow Memory Leak
[
https://issues.apache.org/jira/browse/ARROW-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975954#comment-15975954
]
Julien Le Dem edited comment on ARROW-855 at 4/25/17 10:09 PM:
---
1.abstract class
{code}
abstract class ArrowPillar[V <: ValueVector](val field: FieldDescription, val
allocator: BufferAllocator)
extends CountablePillar {
val vector: V = ArrowPillar.getNewVector(mField.getName, allocator,
mField.getType()).asInstanceOf[V]
initialSize match {
case Some(v) => vector.isInstanceOf[FixedWidthVector] match {
case true =>
vector.asInstanceOf[FixedWidthVector].allocateNew(v)
vector.asInstanceOf[V]
case false => vector.allocateNew()
}
case None => vector.allocateNew()
}
..
def mutator():ValueVector.Mutator = vector.getMutator
def accessor():ValueVector.Accessor = vector.getAccessor
override def pureDel(index: Int): Unit = {
throw new MemoException("Don't support this operation now.")
}
override def close(): Unit = vector.close()
override def clear(): Unit = vector.clear()
}
{code}
2.one implementation
{code}
class ArrowNullableStringPillar(override val field: FieldDescription, override
val allocator: BufferAllocator)
extends ArrowPillar[NullableVarCharVector](field, allocator) {
override def pureSet(index: Int, value: Option[Any]): Unit = {
value match {
case None =>
mutator.asInstanceOf[NullableVarCharVector#Mutator].setNull(index)
case Some(v) =>
val value = v.asInstanceOf[String].getBytes
val length = v.asInstanceOf[String].getBytes.length
mutator.asInstanceOf[NullableVarCharVector#Mutator].setSafe(index,
value, 0, length)
}
}
override def pureGet(index: Int): Option[Any] = {
if (accessor.isNull(index)) None
else
Some(accessor.asInstanceOf[NullableVarCharVector#Accessor].get(index))
}
}
{code}
was (Author: fushengxu):
1.abstract class
abstract class ArrowPillar[V <: ValueVector](val field: FieldDescription, val
allocator: BufferAllocator)
extends CountablePillar {
val vector: V = ArrowPillar.getNewVector(mField.getName, allocator,
mField.getType()).asInstanceOf[V]
initialSize match {
case Some(v) => vector.isInstanceOf[FixedWidthVector] match {
case true =>
vector.asInstanceOf[FixedWidthVector].allocateNew(v)
vector.asInstanceOf[V]
case false => vector.allocateNew()
}
case None => vector.allocateNew()
}
..
def mutator():ValueVector.Mutator = vector.getMutator
def accessor():ValueVector.Accessor = vector.getAccessor
override def pureDel(index: Int): Unit = {
throw new MemoException("Don't support this operation now.")
}
override def close(): Unit = vector.close()
override def clear(): Unit = vector.clear()
}
2.one implementation
class ArrowNullableStringPillar(override val field: FieldDescription, override
val allocator: BufferAllocator)
extends ArrowPillar[NullableVarCharVector](field, allocator) {
override def pureSet(index: Int, value: Option[Any]): Unit = {
value match {
case None =>
mutator.asInstanceOf[NullableVarCharVector#Mutator].setNull(index)
case Some(v) =>
val value = v.asInstanceOf[String].getBytes
val length = v.asInstanceOf[String].getBytes.length
mutator.asInstanceOf[NullableVarCharVector#Mutator].setSafe(index,
value, 0, length)
}
}
override def pureGet(index: Int): Option[Any] = {
if (accessor.isNull(index)) None
else
Some(accessor.asInstanceOf[NullableVarCharVector#Accessor].get(index))
}
}
> Arrow Memory Leak
> -
>
> Key: ARROW-855
> URL: https://issues.apache.org/jira/browse/ARROW-855
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java - Memory, Java - Vectors
>Affects Versions: 0.1.0
> Environment: CentOS release 6.7+Indellij IDEA
>Reporter: xufusheng
>Priority: Critical
> Labels: test
>
> we create a memory table by arrow and the source data come from HBase.
> Create a memory table and then drop the table,there will be a memory leak.
> Hundreds of times,There will be OutOfMemoryError.
> anyone encounter similar problems?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (ARROW-855) Arrow Memory Leak
[
https://issues.apache.org/jira/browse/ARROW-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975929#comment-15975929
]
Julien Le Dem edited comment on ARROW-855 at 4/25/17 10:08 PM:
---
Our programming language is scala.
1.Through HashMap to store pillar
private val entries = new util.LinkedHashMap[PillarId, Pillar](32, 0.75f, true)
2.Pillar encapsulates the data structure of arrow
{code}
entries.put(pillarId, newPillar(getFieldDesc(pillarId.tableId,
pillarId.colIdx)))
def newPillar(fieldDesc: FieldDescription): Pillar = {
ArrowPillar.getPillar(fieldDesc, root)
}
object ArrowPillar {
@NeedTest
def getPillar(field: FieldDescription, allocator: BufferAllocator): Pillar = {
field.desc.getType match {
case MemoType.Boolean => new ArrowNullableBoolPillar(field, allocator)
case MemoType.Short => new ArrowNullableShortPillar(field, allocator)
..
}
}
}
trait Pillar{
def append(value:Option[Any]):Unit
..
}
{code}
3.The following is our free memory function
{code}
def remove(pillarId: PillarId): Boolean = {
entries.synchronized {
var pillar = entries.remove(pillarId)
if (pillar != null) {
pillar.close()
pillar = null
true
} else {
false
}
}
}
{code}
was (Author: fushengxu):
Our programming language is scala.
1.Through HashMap to store pillar
private val entries = new util.LinkedHashMap[PillarId, Pillar](32, 0.75f, true)
2.Pillar encapsulates the data structure of arrow
entries.put(pillarId, newPillar(getFieldDesc(pillarId.tableId,
pillarId.colIdx)))
def newPillar(fieldDesc: FieldDescription): Pillar = {
ArrowPillar.getPillar(fieldDesc, root)
}
object ArrowPillar {
@NeedTest
def getPillar(field: FieldDescription, allocator: BufferAllocator): Pillar = {
field.desc.getType match {
case MemoType.Boolean => new ArrowNullableBoolPillar(field, allocator)
case MemoType.Short => new ArrowNullableShortPillar(field, allocator)
..
}
}
}
trait Pillar{
def append(value:Option[Any]):Unit
..
}
3.The following is our free memory function
def remove(pillarId: PillarId): Boolean = {
entries.synchronized {
var pillar = entries.remove(pillarId)
if (pillar != null) {
pillar.close()
pillar = null
true
} else {
false
}
}
}
> Arrow Memory Leak
> -
>
> Key: ARROW-855
> URL: https://issues.apache.org/jira/browse/ARROW-855
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java - Memory, Java - Vectors
>Affects Versions: 0.1.0
> Environment: CentOS release 6.7+Indellij IDEA
>Reporter: xufusheng
>Priority: Critical
> Labels: test
>
> we create a memory table by arrow and the source data come from HBase.
> Create a memory table and then drop the table,there will be a memory leak.
> Hundreds of times,There will be OutOfMemoryError.
> anyone encounter similar problems?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (ARROW-855) Arrow Memory Leak
[
https://issues.apache.org/jira/browse/ARROW-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975929#comment-15975929
]
Julien Le Dem edited comment on ARROW-855 at 4/25/17 10:08 PM:
---
Our programming language is scala.
1.Through HashMap to store pillar
{code}
private val entries = new util.LinkedHashMap[PillarId, Pillar](32, 0.75f, true)
{code}
2.Pillar encapsulates the data structure of arrow
{code}
entries.put(pillarId, newPillar(getFieldDesc(pillarId.tableId,
pillarId.colIdx)))
def newPillar(fieldDesc: FieldDescription): Pillar = {
ArrowPillar.getPillar(fieldDesc, root)
}
object ArrowPillar {
@NeedTest
def getPillar(field: FieldDescription, allocator: BufferAllocator): Pillar = {
field.desc.getType match {
case MemoType.Boolean => new ArrowNullableBoolPillar(field, allocator)
case MemoType.Short => new ArrowNullableShortPillar(field, allocator)
..
}
}
}
trait Pillar{
def append(value:Option[Any]):Unit
..
}
{code}
3.The following is our free memory function
{code}
def remove(pillarId: PillarId): Boolean = {
entries.synchronized {
var pillar = entries.remove(pillarId)
if (pillar != null) {
pillar.close()
pillar = null
true
} else {
false
}
}
}
{code}
was (Author: fushengxu):
Our programming language is scala.
1.Through HashMap to store pillar
private val entries = new util.LinkedHashMap[PillarId, Pillar](32, 0.75f, true)
2.Pillar encapsulates the data structure of arrow
{code}
entries.put(pillarId, newPillar(getFieldDesc(pillarId.tableId,
pillarId.colIdx)))
def newPillar(fieldDesc: FieldDescription): Pillar = {
ArrowPillar.getPillar(fieldDesc, root)
}
object ArrowPillar {
@NeedTest
def getPillar(field: FieldDescription, allocator: BufferAllocator): Pillar = {
field.desc.getType match {
case MemoType.Boolean => new ArrowNullableBoolPillar(field, allocator)
case MemoType.Short => new ArrowNullableShortPillar(field, allocator)
..
}
}
}
trait Pillar{
def append(value:Option[Any]):Unit
..
}
{code}
3.The following is our free memory function
{code}
def remove(pillarId: PillarId): Boolean = {
entries.synchronized {
var pillar = entries.remove(pillarId)
if (pillar != null) {
pillar.close()
pillar = null
true
} else {
false
}
}
}
{code}
> Arrow Memory Leak
> -
>
> Key: ARROW-855
> URL: https://issues.apache.org/jira/browse/ARROW-855
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java - Memory, Java - Vectors
>Affects Versions: 0.1.0
> Environment: CentOS release 6.7+Indellij IDEA
>Reporter: xufusheng
>Priority: Critical
> Labels: test
>
> we create a memory table by arrow and the source data come from HBase.
> Create a memory table and then drop the table,there will be a memory leak.
> Hundreds of times,There will be OutOfMemoryError.
> anyone encounter similar problems?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (ARROW-855) Arrow Memory Leak
[
https://issues.apache.org/jira/browse/ARROW-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975929#comment-15975929
]
xufusheng edited comment on ARROW-855 at 4/20/17 2:28 AM:
--
Our programming language is scala.
1.Through HashMap to store pillar
private val entries = new util.LinkedHashMap[PillarId, Pillar](32, 0.75f, true)
2.Pillar encapsulates the data structure of arrow
entries.put(pillarId, newPillar(getFieldDesc(pillarId.tableId,
pillarId.colIdx)))
def newPillar(fieldDesc: FieldDescription): Pillar = {
ArrowPillar.getPillar(fieldDesc, root)
}
object ArrowPillar {
@NeedTest
def getPillar(field: FieldDescription, allocator: BufferAllocator): Pillar = {
field.desc.getType match {
case MemoType.Boolean => new ArrowNullableBoolPillar(field, allocator)
case MemoType.Short => new ArrowNullableShortPillar(field, allocator)
..
}
}
}
trait Pillar{
def append(value:Option[Any]):Unit
..
}
3.The following is our free memory function
def remove(pillarId: PillarId): Boolean = {
entries.synchronized {
var pillar = entries.remove(pillarId)
if (pillar != null) {
pillar.close()
pillar = null
true
} else {
false
}
}
}
was (Author: fushengxu):
Our programming language is scala.
1.Through HashMap to store pillar
private val entries = new util.LinkedHashMap[PillarId, Pillar](32, 0.75f, true)
2.Pillar encapsulates the data structure of arrow
entries.put(pillarId, newPillar(getFieldDesc(pillarId.tableId,
pillarId.colIdx)))
def newPillar(fieldDesc: FieldDescription): Pillar = {
ArrowPillar.getPillar(fieldDesc, root)
}
object ArrowPillar {
@NeedTest
def getPillar(field: FieldDescription, allocator: BufferAllocator): Pillar = {
field.desc.getType match {
case MemoType.Boolean => new ArrowNullableBoolPillar(field, allocator)
case MemoType.Short => new ArrowNullableShortPillar(field, allocator)
..
}
}
trait Pillar{
def append(value:Option[Any]):Unit
..
}
3.The following is our release function
def remove(pillarId: PillarId): Boolean = {
entries.synchronized {
var pillar = entries.remove(pillarId)
if (pillar != null) {
pillar.close()
pillar = null
true
} else {
false
}
}
}
> Arrow Memory Leak
> -
>
> Key: ARROW-855
> URL: https://issues.apache.org/jira/browse/ARROW-855
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java - Memory, Java - Vectors
>Affects Versions: 0.1.0
> Environment: CentOS release 6.7+Indellij IDEA
>Reporter: xufusheng
>Priority: Critical
> Labels: test
>
> we create a memory table by arrow and the source data come from HBase.
> Create a memory table and then drop the table,there will be a memory leak.
> Hundreds of times,There will be OutOfMemoryError.
> anyone encounter similar problems?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
