[jira] (SPARK-19404) Command line arguments warnings are written to stdout when using spark-submit and custom code
Title: Message Title Reekunj Parmar closed an issue as Works for Me Spark / SPARK-19404 Command line arguments warnings are written to stdout when using spark-submit and custom code Change By: Reekunj Parmar Resolution: Works for Me Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (SPARK-19404) Command line arguments warnings are written to stdout when using spark-submit and custom code
Title: Message Title
Reekunj Parmar updated an issue
Spark / SPARK-19404
Command line arguments warnings are written to stdout when using spark-submit and custom code
Change By:
Reekunj Parmar
Expecting a clean stdout for piping results of an aggregation operation using spark-submit and custom code. Spark-defaults.conf has this value present in it ("spark.yarn.driver.memoryOverhead 384") which when present causes line #122 in this file https://github.com/apache/spark/blob/v1.6.1/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala#L122 to write directly to stdout , polluting our stream to the downstream process .All of these warnings are required to be sent to the logger basically to file output instead of standard output. Even writing to Console.err or System.err will work.
Add Comment
This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (SPARK-19404) Command line arguments warnings are written to stdout when using spark-submit and custom code
Title: Message Title
Reekunj Parmar updated an issue
Spark / SPARK-19404
Command line arguments warnings are written to stdout when using spark-submit and custom code
Change By:
Reekunj Parmar
Customer expecting Expecting a clean stdout for piping results of an aggregation operation using spark-submit and custom code. Our spark Spark -defaults.conf has this value present in it ("spark.yarn.driver.memoryOverhead 384") which when present causes line #122 in this file https://github.com/apache/spark/blob/v1.6.1/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala#L122 to write directly to stdout, polluting our stream to the downstream process. Customer requires that all All of these warnings are to be sent to the logger so they can be co-located with the other log data basically to file output instead of standard output . Even writing to Console.err or System.err would will work.
Add Comment
This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (SPARK-19404) Command line arguments warnings are written to stdout when using spark-submit and custom code
Title: Message Title Reekunj Parmar updated an issue Spark / SPARK-19404 Command line arguments warnings are written to stdout when using spark-submit and custom code Change By: Reekunj Parmar External issue URL: https://hortonworks.jira.com/browse/BUG-73156 Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (SPARK-19404) Command line arguments warnings are written to stdout when using spark-submit and custom code
Title: Message Title Sean Owen commented on SPARK-19404 Re: Command line arguments warnings are written to stdout when using spark-submit and custom code Nobody here is going to do work for your customer, of course. Generally it's on you to propose the change if you want it done, as JIRA is for developers. In this case, it's such an easy change that it's within reach of any developer. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (SPARK-19404) Command line arguments warnings are written to stdout when using spark-submit and custom code
Title: Message Title Reekunj Parmar commented on SPARK-19404 Re: Command line arguments warnings are written to stdout when using spark-submit and custom code Sean Owen - I did not understand your question. I do not know how to fix this as I am not developer , require fix from the community for one of my existing customer. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (SPARK-19404) Command line arguments warnings are written to stdout when using spark-submit and custom code
Title: Message Title Sean Owen commented on SPARK-19404 Re: Command line arguments warnings are written to stdout when using spark-submit and custom code Agree this should use a logger. Can you fix it in a PR? Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (SPARK-19404) Command line arguments warnings are written to stdout when using spark-submit and custom code
Title: Message Title
Reekunj Parmar created an issue
Spark / SPARK-19404
Command line arguments warnings are written to stdout when using spark-submit and custom code
Issue Type:
Bug
Affects Versions:
1.6.1
Assignee:
Unassigned
Components:
Java API
Created:
30/Jan/17 16:24
Priority:
Minor
Reporter:
Reekunj Parmar
Customer expecting a clean stdout for piping results of an aggregation operation using spark-submit and custom code. Our spark-defaults.conf has this value present in it ("spark.yarn.driver.memoryOverhead 384") which when present causes line #122 in this file https://github.com/apache/spark/blob/v1.6.1/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala#L122 to write directly to stdout, polluting our stream to the downstream process. Customer requires that all of these warnings be sent to the logger so they can be co-located with the other log data. Even writing to Console.err or System.err would work.
