[ https://issues.apache.org/jira/browse/SPARK-27468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
shahid updated SPARK-27468: --------------------------- Attachment: Screenshot from 2019-04-17 10-42-55.png > "Storage Level" in "RDD Storage Page" is not correct > ---------------------------------------------------- > > Key: SPARK-27468 > URL: https://issues.apache.org/jira/browse/SPARK-27468 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.4.1 > Reporter: Shixiong Zhu > Priority: Major > Attachments: Screenshot from 2019-04-17 10-42-55.png > > > I ran the following unit test and checked the UI. > {code} > val conf = new SparkConf() > .setAppName("test") > .setMaster("local-cluster[2,1,1024]") > .set("spark.ui.enabled", "true") > sc = new SparkContext(conf) > val rdd = sc.makeRDD(1 to 10, 1).persist(StorageLevel.MEMORY_ONLY_2) > rdd.count() > Thread.sleep(3600000) > {code} > The storage level is "Memory Deserialized 1x Replicated" in the RDD storage > page. > I tried to debug and found this is because Spark emitted the following two > events: > {code} > event: SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(1, > 10.8.132.160, 65473, None),rdd_0_0,StorageLevel(memory, deserialized, 2 > replicas),56,0)) > event: SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(0, > 10.8.132.160, 65474, None),rdd_0_0,StorageLevel(memory, deserialized, 1 > replicas),56,0)) > {code} > The storage level in the second event will overwrite the first one. "1 > replicas" comes from this line: > https://github.com/apache/spark/blob/3ab96d7acf870e53c9016b0b63d0b328eec23bed/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1457 > Maybe AppStatusListener should calculate the replicas from events? > Another fact we may need to think about is when replicas is 2, will two Spark > events arrive in the same order? Currently, two RPCs from different executors > can arrive in any order. > Credit goes to [~srfnmnk] who reported this issue originally. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org