[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281700#comment-14281700 ]
Reynold Xin edited comment on SPARK-3694 at 1/18/15 9:00 AM: ------------------------------------------------------------- Sorry I didn't see this ticket before I submitted the patch. I thought about the approach proposed here more - while it is doable with reflection, I think it'd be fairly complicated to do it. It would basically require us to implement most of the stuff in ObjectOutputStream. The following needs to be taken into account: 1. Externalizable: writeExternal can still call writeObject, so we would need to create a mocked ObjectOutput 2. Cyclic detection to make sure we don't run into a loop 3. Arrays 4. Primitive types 5. readResolve was (Author: rxin): Sorry I didn't see this ticket before I submitted the patch. I thought about the approach proposed here more - while it is doable with reflection, I think it'd be fairly complicated to do it. It would basically require us to implement most of the stuff in ObjectOutputStream. The following needs to be taken into account: 1. Externalizable: writeExternal can still call writeObject, so we would need to create a mocked ObjectOutput 2. Cyclic detection to make sure we don't run into a loop 3. Arrays 4. Primitive types > Allow printing object graph of tasks/RDD's with a debug flag > ------------------------------------------------------------ > > Key: SPARK-3694 > URL: https://issues.apache.org/jira/browse/SPARK-3694 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Reporter: Patrick Wendell > Assignee: Ilya Ganelin > Labels: starter > > This would be useful for debugging extra references inside of RDD's > Here is an example for inspiration: > http://ehcache.org/xref/net/sf/ehcache/pool/sizeof/ObjectGraphWalker.html > We'd want to print this trace for both the RDD serialization inside of the > DAGScheduler and the task serialization in the TaskSetManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org