ParkGyeongTae commented on code in PR #4936:
URL: https://github.com/apache/zeppelin/pull/4936#discussion_r2121174830
##########
spark-submit/src/main/java/org/apache/zeppelin/spark/submit/SparkSubmitInterpreter.java:
##########
@@ -37,32 +36,72 @@
/**
- * Support %spark-submit which run spark-submit command. Internally,
- * it would run shell command via ShellInterpreter.
- *
+ * Interpreter that supports the `%spark-submit` command in Apache Zeppelin.
+ * <p>
+ * This interpreter allows users to submit Spark jobs using the standard
`spark-submit` CLI
+ * interface.
+ * Internally, it delegates execution to the ShellInterpreter to run
`spark-submit` as a shell
+ * command.
+ * <p>
+ * Key features:
+ * - Automatically builds and executes the `spark-submit` command using the
configured SPARK_HOME
+ * path.
+ * - Extracts the Spark UI URL from logs and publishes it to the Zeppelin
frontend.
+ * - Tracks the YARN Application ID from the logs, allowing the job to be
cancelled via `yarn
+ * application -kill`.
+ * - Handles both YARN and local Spark modes.
+ * <p>
+ * Required configuration:
+ * - SPARK_HOME must be set in the interpreter setting or environment
variables. It should point
+ * to the root
+ * directory of a valid Spark installation.
+ * <p>
+ * Example usage in a Zeppelin notebook:
+ * %spark-submit --class org.apache.spark.examples.SparkPi /path/to/jar
spark-args
*/
public class SparkSubmitInterpreter extends ShellInterpreter {
private static final Logger LOGGER =
LoggerFactory.getLogger(SparkSubmitInterpreter.class);
- private String sparkHome;
-
- // paragraphId --> yarnAppId
+ private final String sparkHome;
private ConcurrentMap<String, String> yarnAppIdMap = new
ConcurrentHashMap<>();
public SparkSubmitInterpreter(Properties property) {
super(property);
- // Set time to be max integer so that the shell process won't timeout.
- setProperty("shell.command.timeout.millisecs", Integer.MAX_VALUE + "");
- this.sparkHome = properties.getProperty("SPARK_HOME");
+ setProperty("shell.command.timeout.millisecs",
String.valueOf(Integer.MAX_VALUE));
+ this.sparkHome = property.getProperty("SPARK_HOME");
Review Comment:
I reverted the variable name from `property` back to `properties`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
