xkrogen commented on a change in pull request #32389: URL: https://github.com/apache/spark/pull/32389#discussion_r633729337
########## File path: core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala ########## @@ -123,6 +131,42 @@ class ShuffleBlockFetcherIteratorSuite extends SparkFunSuite with PrivateMethodT verify(wrappedInputStream.invokePrivate(delegateAccess()), times(1)).close() } + // scalastyle:off argcount + private def createShuffleBlockIteratorWithDefaults( + blocksByAddress: Map[BlockManagerId, (Traversable[BlockId], Long, Int)], + taskContext: Option[TaskContext] = None, + streamWrapperLimitSize: Option[Long] = None, + blockManager: Option[BlockManager] = None, + maxBytesInFlight: Long = Long.MaxValue, + maxReqsInFlight: Int = Int.MaxValue, + maxBlocksInFlightPerAddress: Int = Int.MaxValue, + maxReqSizeShuffleToMem: Int = Int.MaxValue, + detectCorrupt: Boolean = true, + detectCorruptUseExtraMemory: Boolean = true, + shuffleMetrics: Option[ShuffleReadMetricsReporter] = None, + doBatchFetch: Boolean = false): ShuffleBlockFetcherIterator = { + val tContext = taskContext.getOrElse(TaskContext.empty()) + new ShuffleBlockFetcherIterator( + tContext, + transfer, + blockManager.getOrElse(createMockBlockManager()), + blocksByAddress.map { case (blockManagerId, (blocks, blockSize, blockMapIndex)) => + (blockManagerId, blocks.map(blockId => (blockId, blockSize, blockMapIndex)).toSeq) + }.toIterator, Review comment: I think this is closely related to your [other comment](https://github.com/apache/spark/pull/32389#discussion_r632823782), so I'll respond to both here. I agree that this is tailored to the current set of tests and thus current usage, as opposed to potential future usage. If we were designing a public API, or even a private API in the production code, I would agree with you. But in this case for a class-private method in a test file, I'm not convinced that designing for future possibilities is the right move. If someone later adds tests which do need blocks of different sizes, they can make the modification you've described, right? For now I would preference simplicity, and we can introduce the additional complexity if necessary. WDYT? This is not a strong conviction on my side so if you are not convinced I can make changes as you proposed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org