Patrick Wendell created SPARK-3874:
--------------------------------------
Summary: Provide stable TaskContext API
Key: SPARK-3874
URL: https://issues.apache.org/jira/browse/SPARK-3874
Project: Spark
Issue Type: Improvement
Components: Spark Core
Reporter: Patrick Wendell
Assignee: Prashant Sharma
We made some improvements in SPARK-3543 but for Spark 1.2 we should convert
TaskContext into a fully stable API. To do this I’d suggest the following
changes - note that some of this reverses parts of SPARK-3543. The goal is to
provide a class that users can’t easily construct and exposes only the public
functionality.
1. Separate TaskContext into a public abstract class (TaskContext) and a
private implementation called TaskContextImpl. The former should be a Java
abstract class - the latter should be a private[spark] Scala class to reduce
visibility (or maybe we can keep it as Java and tell people not to use it?).
2. TaskContext abstract class will have (NOTE: this changes getXX() to XX()
intentionally)
public isCompleted()
public isInterrupted()
public addTaskCompletionListener(...)
public addTaskCompletionCallback(...) (deprecated)
public stageId()
public partitionId()
public attemptId()
pubic isRunningLocally()
STATIC
public get()
set() and unset() at default visibility
3. A new private[spark] static object TaskContextHelper in the same package as
TaskContext will exist to expose set() and unset() from within Spark using
forwarder methods that just call TaskContext.set(). If someone within Spark
wants to set this they call TaskContextHelper.set() and it forwards it.
4. TaskContextImpl will be used whenever we construct a TaskContext internally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
