[jira] [Created] (SPARK-42834) Divided by zero occurs in PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse
Li Ying created SPARK-42834: --- Summary: Divided by zero occurs in PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse Key: SPARK-42834 URL: https://issues.apache.org/jira/browse/SPARK-42834 Project: Spark Issue Type: Bug Components: Shuffle Affects Versions: 3.2.0 Reporter: Li Ying {color:#22}Sometimes when run a SQL job with push based shuffle, exception occurs as below. It seems that there’s no element in the bitmaps which stores merge chunk meta. See org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse.{color} {color:#22} {color} {color:#22}Is it a bug that we should not createChunkBlockInfos when bitmaps is empty or the bitmaps should never be empty here ?{color} {code:java} Caused by: java.lang.ArithmeticException: / by zero at org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse(PushBasedFetchHelper.scala:117) at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:980) at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:84) {code} related code: {code:java} def createChunkBlockInfosFromMetaResponse( shuffleId: Int, shuffleMergeId: Int, reduceId: Int, blockSize: Long, bitmaps: Array[RoaringBitmap]): ArrayBuffer[(BlockId, Long, Int)] = { val approxChunkSize = blockSize / bitmaps.length val blocksToFetch = new ArrayBuffer[(BlockId, Long, Int)]() for (i <- bitmaps.indices) { val blockChunkId = ShuffleBlockChunkId(shuffleId, shuffleMergeId, reduceId, i) chunksMetaMap.put(blockChunkId, bitmaps(i)) logDebug(s"adding block chunk $blockChunkId of size $approxChunkSize") blocksToFetch += ((blockChunkId, approxChunkSize, SHUFFLE_PUSH_MAP_ID)) } blocksToFetch } {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-42834) Divided by zero occurs in PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse
[ https://issues.apache.org/jira/browse/SPARK-42834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Ying updated SPARK-42834: Description: {color:#22}Sometimes when run a SQL job with push based shuffle, exception occurs as below. It seems that there’s no element in the bitmaps which stores merge chunk meta. {color} {color:#22}Is it a bug that we should not createChunkBlockInfos when bitmaps is empty or the bitmaps should never be empty here ?{color} {code:java} Caused by: java.lang.ArithmeticException: / by zero at org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse(PushBasedFetchHelper.scala:117) at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:980) at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:84) {code} related code: {code:java} def createChunkBlockInfosFromMetaResponse( shuffleId: Int, shuffleMergeId: Int, reduceId: Int, blockSize: Long, bitmaps: Array[RoaringBitmap]): ArrayBuffer[(BlockId, Long, Int)] = { val approxChunkSize = blockSize / bitmaps.length val blocksToFetch = new ArrayBuffer[(BlockId, Long, Int)]() for (i <- bitmaps.indices) { val blockChunkId = ShuffleBlockChunkId(shuffleId, shuffleMergeId, reduceId, i) chunksMetaMap.put(blockChunkId, bitmaps(i)) logDebug(s"adding block chunk $blockChunkId of size $approxChunkSize") blocksToFetch += ((blockChunkId, approxChunkSize, SHUFFLE_PUSH_MAP_ID)) } blocksToFetch } {code} was: {color:#22}Sometimes when run a SQL job with push based shuffle, exception occurs as below. It seems that there’s no element in the bitmaps which stores merge chunk meta. See org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse.{color} {color:#22} {color} {color:#22}Is it a bug that we should not createChunkBlockInfos when bitmaps is empty or the bitmaps should never be empty here ?{color} {code:java} Caused by: java.lang.ArithmeticException: / by zero at org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse(PushBasedFetchHelper.scala:117) at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:980) at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:84) {code} related code: {code:java} def createChunkBlockInfosFromMetaResponse( shuffleId: Int, shuffleMergeId: Int, reduceId: Int, blockSize: Long, bitmaps: Array[RoaringBitmap]): ArrayBuffer[(BlockId, Long, Int)] = { val approxChunkSize = blockSize / bitmaps.length val blocksToFetch = new ArrayBuffer[(BlockId, Long, Int)]() for (i <- bitmaps.indices) { val blockChunkId = ShuffleBlockChunkId(shuffleId, shuffleMergeId, reduceId, i) chunksMetaMap.put(blockChunkId, bitmaps(i)) logDebug(s"adding block chunk $blockChunkId of size $approxChunkSize") blocksToFetch += ((blockChunkId, approxChunkSize, SHUFFLE_PUSH_MAP_ID)) } blocksToFetch } {code} > Divided by zero occurs in > PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse > > > Key: SPARK-42834 > URL: https://issues.apache.org/jira/browse/SPARK-42834 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 3.2.0 >Reporter: Li Ying >Priority: Major > > {color:#22}Sometimes when run a SQL job with push based shuffle, > exception occurs as below. It seems that there’s no element in the bitmaps > which stores merge chunk meta. {color} > {color:#22}Is it a bug that we should not createChunkBlockInfos when > bitmaps is empty or the bitmaps should never be empty here ?{color} > > {code:java} > Caused by: java.lang.ArithmeticException: / by zero > at > org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse(PushBasedFetchHelper.scala:117) > at > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:980) > at > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:84) > {code} > related code: > {code:java} > def createChunkBlockInfosFromMetaResponse( > shuffleId: Int, > shuffleMergeId: Int, > reduceId: Int, > blockSize: Long, > bitmaps: Array[RoaringBitmap]): ArrayBuffer[(BlockId, Long, Int)] = { > val approxChunkSize = blockSize / bitmaps.length > val blocksToFetch = new ArrayBuffer[(BlockId, Long, Int)]() > for (i <- bitmaps.indices) { > val blockChunkId = ShuffleBlockChunkId(shuffleId, shuffleMergeId, > reduceId, i) > chunksMetaMap.put(blockChunkId, bitmaps(i)) > logDebug(s"adding block chunk $blockChunkId of size $approxChunkSize") > blocksToFetch += ((bl
[jira] [Commented] (SPARK-42834) Divided by zero occurs in PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse
[ https://issues.apache.org/jira/browse/SPARK-42834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701610#comment-17701610 ] Li Ying commented on SPARK-42834: - [~csingh] Could you please help confirm this? > Divided by zero occurs in > PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse > > > Key: SPARK-42834 > URL: https://issues.apache.org/jira/browse/SPARK-42834 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 3.2.0 >Reporter: Li Ying >Priority: Major > > {color:#22}Sometimes when run a SQL job with push based shuffle, > exception occurs as below. It seems that there’s no element in the bitmaps > which stores merge chunk meta. {color} > {color:#22}Is it a bug that we should not createChunkBlockInfos when > bitmaps is empty or the bitmaps should never be empty here ?{color} > > {code:java} > Caused by: java.lang.ArithmeticException: / by zero > at > org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse(PushBasedFetchHelper.scala:117) > at > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:980) > at > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:84) > {code} > related code: > {code:java} > def createChunkBlockInfosFromMetaResponse( > shuffleId: Int, > shuffleMergeId: Int, > reduceId: Int, > blockSize: Long, > bitmaps: Array[RoaringBitmap]): ArrayBuffer[(BlockId, Long, Int)] = { > val approxChunkSize = blockSize / bitmaps.length > val blocksToFetch = new ArrayBuffer[(BlockId, Long, Int)]() > for (i <- bitmaps.indices) { > val blockChunkId = ShuffleBlockChunkId(shuffleId, shuffleMergeId, > reduceId, i) > chunksMetaMap.put(blockChunkId, bitmaps(i)) > logDebug(s"adding block chunk $blockChunkId of size $approxChunkSize") > blocksToFetch += ((blockChunkId, approxChunkSize, SHUFFLE_PUSH_MAP_ID)) > } > blocksToFetch > } {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-42834) Divided by zero occurs in PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse
[ https://issues.apache.org/jira/browse/SPARK-42834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17702050#comment-17702050 ] Li Ying commented on SPARK-42834: - [~csingh] Thanks for help. I would take this fix :) > Divided by zero occurs in > PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse > > > Key: SPARK-42834 > URL: https://issues.apache.org/jira/browse/SPARK-42834 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 3.2.0 >Reporter: Li Ying >Priority: Major > > {color:#22}Sometimes when run a SQL job with push based shuffle, > exception occurs as below. It seems that there’s no element in the bitmaps > which stores merge chunk meta. {color} > {color:#22}Is it a bug that we should not createChunkBlockInfos when > bitmaps is empty or the bitmaps should never be empty here ?{color} > > {code:java} > Caused by: java.lang.ArithmeticException: / by zero > at > org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse(PushBasedFetchHelper.scala:117) > at > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:980) > at > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:84) > {code} > related code: > {code:java} > def createChunkBlockInfosFromMetaResponse( > shuffleId: Int, > shuffleMergeId: Int, > reduceId: Int, > blockSize: Long, > bitmaps: Array[RoaringBitmap]): ArrayBuffer[(BlockId, Long, Int)] = { > val approxChunkSize = blockSize / bitmaps.length > val blocksToFetch = new ArrayBuffer[(BlockId, Long, Int)]() > for (i <- bitmaps.indices) { > val blockChunkId = ShuffleBlockChunkId(shuffleId, shuffleMergeId, > reduceId, i) > chunksMetaMap.put(blockChunkId, bitmaps(i)) > logDebug(s"adding block chunk $blockChunkId of size $approxChunkSize") > blocksToFetch += ((blockChunkId, approxChunkSize, SHUFFLE_PUSH_MAP_ID)) > } > blocksToFetch > } {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-42834) Divided by zero occurs in PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse
[ https://issues.apache.org/jira/browse/SPARK-42834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Ying closed SPARK-42834. --- > Divided by zero occurs in > PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse > > > Key: SPARK-42834 > URL: https://issues.apache.org/jira/browse/SPARK-42834 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 3.2.0 >Reporter: Li Ying >Priority: Major > > {color:#22}Sometimes when run a SQL job with push based shuffle, > exception occurs as below. It seems that there’s no element in the bitmaps > which stores merge chunk meta. {color} > {color:#22}Is it a bug that we should not createChunkBlockInfos when > bitmaps is empty or the bitmaps should never be empty here ?{color} > > {code:java} > Caused by: java.lang.ArithmeticException: / by zero > at > org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse(PushBasedFetchHelper.scala:117) > at > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:980) > at > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:84) > {code} > related code: > {code:java} > def createChunkBlockInfosFromMetaResponse( > shuffleId: Int, > shuffleMergeId: Int, > reduceId: Int, > blockSize: Long, > bitmaps: Array[RoaringBitmap]): ArrayBuffer[(BlockId, Long, Int)] = { > val approxChunkSize = blockSize / bitmaps.length > val blocksToFetch = new ArrayBuffer[(BlockId, Long, Int)]() > for (i <- bitmaps.indices) { > val blockChunkId = ShuffleBlockChunkId(shuffleId, shuffleMergeId, > reduceId, i) > chunksMetaMap.put(blockChunkId, bitmaps(i)) > logDebug(s"adding block chunk $blockChunkId of size $approxChunkSize") > blocksToFetch += ((blockChunkId, approxChunkSize, SHUFFLE_PUSH_MAP_ID)) > } > blocksToFetch > } {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-42834) Divided by zero occurs in PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse
[ https://issues.apache.org/jira/browse/SPARK-42834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Ying resolved SPARK-42834. - Resolution: Won't Do > Divided by zero occurs in > PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse > > > Key: SPARK-42834 > URL: https://issues.apache.org/jira/browse/SPARK-42834 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 3.2.0 >Reporter: Li Ying >Priority: Major > > {color:#22}Sometimes when run a SQL job with push based shuffle, > exception occurs as below. It seems that there’s no element in the bitmaps > which stores merge chunk meta. {color} > {color:#22}Is it a bug that we should not createChunkBlockInfos when > bitmaps is empty or the bitmaps should never be empty here ?{color} > > {code:java} > Caused by: java.lang.ArithmeticException: / by zero > at > org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse(PushBasedFetchHelper.scala:117) > at > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:980) > at > org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:84) > {code} > related code: > {code:java} > def createChunkBlockInfosFromMetaResponse( > shuffleId: Int, > shuffleMergeId: Int, > reduceId: Int, > blockSize: Long, > bitmaps: Array[RoaringBitmap]): ArrayBuffer[(BlockId, Long, Int)] = { > val approxChunkSize = blockSize / bitmaps.length > val blocksToFetch = new ArrayBuffer[(BlockId, Long, Int)]() > for (i <- bitmaps.indices) { > val blockChunkId = ShuffleBlockChunkId(shuffleId, shuffleMergeId, > reduceId, i) > chunksMetaMap.put(blockChunkId, bitmaps(i)) > logDebug(s"adding block chunk $blockChunkId of size $approxChunkSize") > blocksToFetch += ((blockChunkId, approxChunkSize, SHUFFLE_PUSH_MAP_ID)) > } > blocksToFetch > } {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-38973) When push-based shuffle is enabled, a stage may not complete when retried
[ https://issues.apache.org/jira/browse/SPARK-38973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17702556#comment-17702556 ] Li Ying commented on SPARK-38973: - [~csingh] Should this bugfix be merged into 3.2.x branches? > When push-based shuffle is enabled, a stage may not complete when retried > - > > Key: SPARK-38973 > URL: https://issues.apache.org/jira/browse/SPARK-38973 > Project: Spark > Issue Type: Bug > Components: Shuffle, Spark Core >Affects Versions: 3.2.0 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Fix For: 3.3.0 > > > With push-based shuffle enabled and adaptive merge finalization, there are > scenarios where a re-attempt of ShuffleMapStage may not complete. > With Adaptive Merge Finalization, a stage may be triggered for finalization > when it is in the below state: > # The stage is *not* running ({*}not{*} in the _running_ set of the > DAGScheduler) - had failed or canceled or waiting, and > # The stage has no pending partitions (all the tasks completed at-least > once). > For such a stage when the finalization completes, the stage will still not be > marked as {_}mergeFinalized{_}. > The stage of the stage will be: > * _stage.shuffleDependency.mergeFinalized = false_ > * _stage.shuffleDependency.getFinalizeTask = finalizeTask_ > * Merged statuses of the state are unregistered > > When the stage is resubmitted, the newer attempt of the stage will never > complete even though its tasks may be completed. This is because the newer > attempt of the stage will have {_}shuffleMergeEnabled = true{_}, since with > the previous attempt the stage was never marked as {_}mergedFinalized{_}, and > the _finalizeTask_ is present (from finalization attempt for previous stage > attempt). > > So, when all the tasks of the newer attempt complete, then these conditions > will be true: > * stage will be running > * There will be no pending partitions since all the tasks completed > * _stage.shuffleDependency.shuffleMergeEnabled = true_ > * _stage.shuffleDependency.shuffleMergeFinalized = false_ > * _stage.shuffleDependency.getFinalizeTask_ is not empty > This leads the DAGScheduler to try scheduling finalization and not trigger > the completion of the Stage. However because of the last condition it never > even schedules the finalization and the stage never completes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org