[jira] [Updated] (SPARK-20095) A code bug in CodegenContext.withSubExprEliminationExprs
[ https://issues.apache.org/jira/browse/SPARK-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] song fengfei updated SPARK-20095: - Priority: Major (was: Minor) > A code bug in CodegenContext.withSubExprEliminationExprs > > > Key: SPARK-20095 > URL: https://issues.apache.org/jira/browse/SPARK-20095 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: song fengfei > > In the function:CodegenContext.withSubExprEliminationExprs...{ > val oldsubExprEliminationExprs = subExprEliminationExprs >subExprEliminationExprs.clear > ... > // Restore previous subExprEliminationExprs > subExprEliminationExprs.clear > oldsubExprEliminationExprs.foreach(subExprEliminationExprs += _) > } > it seems that the oldsubExprEliminationExprs and subExprEliminationExprs are > the same instance,after the second subExprEliminationExprs.clear ,the > oldsubExprEliminationExprs is also cleared,so,the previous > subExprEliminationExprs in CodegenContext will not be Restored. > Is it a bug? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-20095) A code bug in CodegenContext.withSubExprEliminationExprs
song fengfei created SPARK-20095: Summary: A code bug in CodegenContext.withSubExprEliminationExprs Key: SPARK-20095 URL: https://issues.apache.org/jira/browse/SPARK-20095 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.1.0 Reporter: song fengfei Priority: Minor In the function:CodegenContext.withSubExprEliminationExprs...{ val oldsubExprEliminationExprs = subExprEliminationExprs subExprEliminationExprs.clear ... // Restore previous subExprEliminationExprs subExprEliminationExprs.clear oldsubExprEliminationExprs.foreach(subExprEliminationExprs += _) } it seems that the oldsubExprEliminationExprs and subExprEliminationExprs are the same instance,after the second subExprEliminationExprs.clear ,the oldsubExprEliminationExprs is also cleared,so,the previous subExprEliminationExprs in CodegenContext will not be Restored. Is it a bug? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17291) The shuffle data fetched based on netty were directly stored in off-memoryr?
[ https://issues.apache.org/jira/browse/SPARK-17291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15446142#comment-15446142 ] song fengfei commented on SPARK-17291: -- Thanks very much,this is the first time to create issue,and can't understand “user@ is the place for questions” before!Thanks again. > The shuffle data fetched based on netty were directly stored in off-memoryr? > > > Key: SPARK-17291 > URL: https://issues.apache.org/jira/browse/SPARK-17291 > Project: Spark > Issue Type: IT Help > Components: Shuffle >Reporter: song fengfei > > If the shuffle data were stored in off-memory,and isn't it easily lead to OOM > when a map output was too big?Additionly,it was also out of spark memory > management? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17291) The shuffle data fetched based on netty were directly stored in off-memoryr?
[ https://issues.apache.org/jira/browse/SPARK-17291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15446110#comment-15446110 ] song fengfei commented on SPARK-17291: -- yeah, they are same,but neither was resolved and instead, they were all changed to resolved? Is it not allowed to ask question here? > The shuffle data fetched based on netty were directly stored in off-memoryr? > > > Key: SPARK-17291 > URL: https://issues.apache.org/jira/browse/SPARK-17291 > Project: Spark > Issue Type: IT Help > Components: Shuffle >Reporter: song fengfei > > If the shuffle data were stored in off-memory,and isn't it easily lead to OOM > when a map output was too big?Additionly,it was also out of spark memory > management? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-17286) The fetched data of Shuffle read stored in the ManagedBuffer, so its underlying data stored in off-memory or file?
[ https://issues.apache.org/jira/browse/SPARK-17286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] song fengfei closed SPARK-17286. > The fetched data of Shuffle read stored in the ManagedBuffer, so its > underlying data stored in off-memory or file? > --- > > Key: SPARK-17286 > URL: https://issues.apache.org/jira/browse/SPARK-17286 > Project: Spark > Issue Type: Question > Components: Block Manager, Input/Output, Shuffle >Reporter: song fengfei > > The fetched data of Shuffle read stored in the ManagedBuffer, so its > underlying data stored in off-memory or file? if stored in off-memory, > Whether it is also managed by spark memorymanager ? and if a map output is > too big, placed it directly in the memory isn't very easily lead to oom? I > did not understand this piece, hoping to get the answer here,thanks! -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-17291) The shuffle data fetched based on netty were directly stored in off-memoryr?
song fengfei created SPARK-17291: Summary: The shuffle data fetched based on netty were directly stored in off-memoryr? Key: SPARK-17291 URL: https://issues.apache.org/jira/browse/SPARK-17291 Project: Spark Issue Type: IT Help Components: Shuffle Reporter: song fengfei If the shuffle data were stored in off-memory,and isn't it easily lead to OOM when a map output was too big?Additionly,it was also out of spark memory management? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13510) Shuffle may throw FetchFailedException: Direct buffer memory
[ https://issues.apache.org/jira/browse/SPARK-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15445571#comment-15445571 ] song fengfei commented on SPARK-13510: -- Hi, Hong Shen You mean that the shuffle data fetched based on Netty were stored in off-memory?and, didn't write them to file even though the block data too big? > Shuffle may throw FetchFailedException: Direct buffer memory > > > Key: SPARK-13510 > URL: https://issues.apache.org/jira/browse/SPARK-13510 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.6.0 >Reporter: Hong Shen > Attachments: spark-13510.diff > > > In our cluster, when I test spark-1.6.0 with a sql, it throw exception and > failed. > {code} > 16/02/17 15:36:03 INFO storage.ShuffleBlockFetcherIterator: Sending request > for 1 blocks (915.4 MB) from 10.196.134.220:7337 > 16/02/17 15:36:03 INFO shuffle.ExternalShuffleClient: External shuffle fetch > from 10.196.134.220:7337 (executor id 122) > 16/02/17 15:36:03 INFO client.TransportClient: Sending fetch chunk request 0 > to /10.196.134.220:7337 > 16/02/17 15:36:36 WARN server.TransportChannelHandler: Exception in > connection from /10.196.134.220:7337 > java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:645) > at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:228) > at io.netty.buffer.PoolArena.allocate(PoolArena.java:212) > at io.netty.buffer.PoolArena.allocate(PoolArena.java:132) > at > io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:271) > at > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) > at > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146) > at > io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:107) > at > io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117) > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:744) > 16/02/17 15:36:36 ERROR client.TransportResponseHandler: Still have 1 > requests outstanding when connection from /10.196.134.220:7337 is closed > 16/02/17 15:36:36 ERROR shuffle.RetryingBlockFetcher: Failed to fetch block > shuffle_3_81_2, and will not retry (0 retries) > {code} > The reason is that when shuffle a big block(like 1G), task will allocate > the same memory, it will easily throw "FetchFailedException: Direct buffer > memory". > If I add -Dio.netty.noUnsafe=true spark.executor.extraJavaOptions, it will > throw > {code} > java.lang.OutOfMemoryError: Java heap space > at > io.netty.buffer.PoolArena$HeapArena.newUnpooledChunk(PoolArena.java:607) > at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:237) > at io.netty.buffer.PoolArena.allocate(PoolArena.java:215) > at io.netty.buffer.PoolArena.allocate(PoolArena.java:132) > {code} > > In mapreduce shuffle, it will firstly judge whether the block can cache in > memery, but spark doesn't. > If the block is more than we can cache in memory, we should write to disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-17286) The fetched data of Shuffle read stored in the ManagedBuffer, so its underlying data stored in off-memory or file?
song fengfei created SPARK-17286: Summary: The fetched data of Shuffle read stored in the ManagedBuffer, so its underlying data stored in off-memory or file? Key: SPARK-17286 URL: https://issues.apache.org/jira/browse/SPARK-17286 Project: Spark Issue Type: Question Components: Block Manager, Input/Output, Shuffle Reporter: song fengfei The fetched data of Shuffle read stored in the ManagedBuffer, so its underlying data stored in off-memory or file? if stored in off-memory, Whether it is also managed by spark memorymanager ? and if a map output is too big, placed it directly in the memory isn't very easily lead to oom? I did not understand this piece, hoping to get the answer here,thanks! -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org