[ https://issues.apache.org/jira/browse/SPARK-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138242#comment-14138242 ]
Josh Rosen commented on SPARK-2321: ----------------------------------- I agree that this should be a pull API. A pull-based API will be easier to expose in Python and Java and might be sufficient for many of the common use-cases. With a push-based API, we might have to worry about things like callback processing speed, rate-limiting, etc. (the Rx frameworks have interesting ways of addressing these concerns, but I'm not sure whether we want to add those dependencies); the pull approach won't face these issues. > Design a proper progress reporting & event listener API > ------------------------------------------------------- > > Key: SPARK-2321 > URL: https://issues.apache.org/jira/browse/SPARK-2321 > Project: Spark > Issue Type: Improvement > Components: Java API, Spark Core > Affects Versions: 1.0.0 > Reporter: Reynold Xin > Assignee: Josh Rosen > Priority: Critical > > This is a ticket to track progress on redesigning the SparkListener and > JobProgressListener API. > There are multiple problems with the current design, including: > 0. I'm not sure if the API is usable in Java (there are at least some enums > we used in Scala and a bunch of case classes that might complicate things). > 1. The whole API is marked as DeveloperApi, because we haven't paid a lot of > attention to it yet. Something as important as progress reporting deserves a > more stable API. > 2. There is no easy way to connect jobs with stages. Similarly, there is no > easy way to connect job groups with jobs / stages. > 3. JobProgressListener itself has no encapsulation at all. States can be > arbitrarily mutated by external programs. Variable names are sort of randomly > decided and inconsistent. > We should just revisit these and propose a new, concrete design. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org