Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/14617#discussion_r107939634
--- Diff: core/src/main/scala/org/apache/spark/storage/StorageUtils.scala
---
@@ -47,24 +50,30 @@ class StorageStatus(val blockManagerId: BlockManagerId,
val maxMem: Long) {
private val _nonRddBlocks = new mutable.HashMap[BlockId, BlockStatus]
/**
- * Storage information of the blocks that entails memory, disk, and
off-heap memory usage.
+ * Storage information of the blocks that entails on-heap memory,
off-heap memory and disk usage.
*
* As with the block maps, we store the storage information separately
for RDD blocks and
* non-RDD blocks for the same reason. In particular, RDD storage
information is stored
- * in a map indexed by the RDD ID to the following 4-tuple:
+ * in a map indexed by the RDD ID to the following 3-tuple case class:
*
* (memory size, disk size, storage level)
*
* We assume that all the blocks that belong to the same RDD have the
same storage level.
- * This field is not relevant to non-RDD blocks, however, so the storage
information for
- * non-RDD blocks contains only the first 3 fields (in the same order).
*/
- private val _rddStorageInfo = new mutable.HashMap[Int, (Long, Long,
StorageLevel)]
- private var _nonRddStorageInfo: (Long, Long) = (0L, 0L)
+ case class RddStorageInfo(memoryUsage: Long, diskUsage: Long, level:
StorageLevel)
+ private val _rddStorageInfo = new mutable.HashMap[Int, RddStorageInfo]
+
+ // On-heap memory, off-heap memory and disk usage of non rdd storage
+ case class NonRddStorageInfo(var onHeapUsage: Long, var offHeapUsage:
Long, var diskUsage: Long)
--- End diff --
`private`
and one super-nit: I find comments that just repeat field or method names
to be pretty pointless, so I'd delete that comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
