[jira] [Commented] (SPARK-19829) The log about driver should support rolling like executor
[
https://issues.apache.org/jira/browse/SPARK-19829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902982#comment-15902982
]
hustfxj commented on SPARK-19829:
-
ok.
> The log about driver should support rolling like executor
> -
>
> Key: SPARK-19829
> URL: https://issues.apache.org/jira/browse/SPARK-19829
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: hustfxj
>Priority: Minor
>
> We should rollback the log of the driver , or the log maybe large！！！
> {code:title=DriverRunner.java|borderStyle=solid}
> // modify the runDriver
> private def runDriver(builder: ProcessBuilder, baseDir: File, supervise:
> Boolean): Int = {
> builder.directory(baseDir)
> def initialize(process: Process): Unit = {
> // Redirect stdout and stderr to files-- the old code
> // val stdout = new File(baseDir, "stdout")
> // CommandUtils.redirectStream(process.getInputStream, stdout)
> //
> // val stderr = new File(baseDir, "stderr")
> // val formattedCommand = builder.command.asScala.mkString("\"", "\"
> \"", "\"")
> // val header = "Launch Command: %s\n%s\n\n".format(formattedCommand,
> "=" * 40)
> // Files.append(header, stderr, StandardCharsets.UTF_8)
> // CommandUtils.redirectStream(process.getErrorStream, stderr)
> // Redirect its stdout and stderr to files-support rolling
> val stdout = new File(baseDir, "stdout")
> stdoutAppender = FileAppender(process.getInputStream, stdout, conf)
> val stderr = new File(baseDir, "stderr")
> val formattedCommand = builder.command.asScala.mkString("\"", "\" \"",
> "\"")
> val header = "Launch Command: %s\n%s\n\n".format(formattedCommand, "="
> * 40)
> Files.append(header, stderr, StandardCharsets.UTF_8)
> stderrAppender = FileAppender(process.getErrorStream, stderr, conf)
> }
> runCommandWithRetry(ProcessBuilderLike(builder), initialize, supervise)
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19829) The log about driver should support rolling like executor
[
https://issues.apache.org/jira/browse/SPARK-19829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899228#comment-15899228
]
Sean Owen commented on SPARK-19829:
---
Why wouldn't log4j be a good solution here? its purpose is managing logs and
can define rolling appenders.
> The log about driver should support rolling like executor
> -
>
> Key: SPARK-19829
> URL: https://issues.apache.org/jira/browse/SPARK-19829
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: hustfxj
>Priority: Minor
>
> We should rollback the log of the driver , or the log maybe large！！！
> {code:title=DriverRunner.java|borderStyle=solid}
> // modify the runDriver
> private def runDriver(builder: ProcessBuilder, baseDir: File, supervise:
> Boolean): Int = {
> builder.directory(baseDir)
> def initialize(process: Process): Unit = {
> // Redirect stdout and stderr to files-- the old code
> // val stdout = new File(baseDir, "stdout")
> // CommandUtils.redirectStream(process.getInputStream, stdout)
> //
> // val stderr = new File(baseDir, "stderr")
> // val formattedCommand = builder.command.asScala.mkString("\"", "\"
> \"", "\"")
> // val header = "Launch Command: %s\n%s\n\n".format(formattedCommand,
> "=" * 40)
> // Files.append(header, stderr, StandardCharsets.UTF_8)
> // CommandUtils.redirectStream(process.getErrorStream, stderr)
> // Redirect its stdout and stderr to files-support rolling
> val stdout = new File(baseDir, "stdout")
> stdoutAppender = FileAppender(process.getInputStream, stdout, conf)
> val stderr = new File(baseDir, "stderr")
> val formattedCommand = builder.command.asScala.mkString("\"", "\" \"",
> "\"")
> val header = "Launch Command: %s\n%s\n\n".format(formattedCommand, "="
> * 40)
> Files.append(header, stderr, StandardCharsets.UTF_8)
> stderrAppender = FileAppender(process.getErrorStream, stderr, conf)
> }
> runCommandWithRetry(ProcessBuilderLike(builder), initialize, supervise)
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19829) The log about driver should support rolling like executor
[
https://issues.apache.org/jira/browse/SPARK-19829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898713#comment-15898713
]
hustfxj commented on SPARK-19829:
-
[~sowen] Yes, this is handled by systems like YARN. But my spark cluster is
standalone. A standalone cluster should build its own rolling about driver' log
in default other than defined log4j configuration.
> The log about driver should support rolling like executor
> -
>
> Key: SPARK-19829
> URL: https://issues.apache.org/jira/browse/SPARK-19829
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: hustfxj
>Priority: Minor
>
> We should rollback the log of the driver , or the log maybe large！！！
> {code:title=DriverRunner.java|borderStyle=solid}
> // modify the runDriver
> private def runDriver(builder: ProcessBuilder, baseDir: File, supervise:
> Boolean): Int = {
> builder.directory(baseDir)
> def initialize(process: Process): Unit = {
> // Redirect stdout and stderr to files-- the old code
> // val stdout = new File(baseDir, "stdout")
> // CommandUtils.redirectStream(process.getInputStream, stdout)
> //
> // val stderr = new File(baseDir, "stderr")
> // val formattedCommand = builder.command.asScala.mkString("\"", "\"
> \"", "\"")
> // val header = "Launch Command: %s\n%s\n\n".format(formattedCommand,
> "=" * 40)
> // Files.append(header, stderr, StandardCharsets.UTF_8)
> // CommandUtils.redirectStream(process.getErrorStream, stderr)
> // Redirect its stdout and stderr to files-support rolling
> val stdout = new File(baseDir, "stdout")
> stdoutAppender = FileAppender(process.getInputStream, stdout, conf)
> val stderr = new File(baseDir, "stderr")
> val formattedCommand = builder.command.asScala.mkString("\"", "\" \"",
> "\"")
> val header = "Launch Command: %s\n%s\n\n".format(formattedCommand, "="
> * 40)
> Files.append(header, stderr, StandardCharsets.UTF_8)
> stderrAppender = FileAppender(process.getErrorStream, stderr, conf)
> }
> runCommandWithRetry(ProcessBuilderLike(builder), initialize, supervise)
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
