Dear All,
When using spark 1.3.0 spark-submit with directing out and err to a log file, I
saw some strange lines inside that looks like this:
[Stage 0:> (0 + 2) / 120]
[Stage 0:> (2 + 2) / 120]
[Stage 0:==> (6 + 2) / 120]
[Stage 0:=====> (12 + 2) / 120]
[Stage 0:=========> (20 + 2) / 120]
[Stage 0:===========> (24 + 2) / 120]
[Stage 0:==============> (32 + 2) / 120]
[Stage 0:===================> (42 + 2) / 120]
[Stage 0:========================> (52 + 2) / 120]
[Stage 0:===========================> (59 + 2) / 120]
[Stage 0:===============================> (68 + 2) / 120]
[Stage 0:====================================> (78 + 3) / 120]
[Stage 0:=========================================> (88 + 4) / 120]
[Stage 0:=============================================> (100 + 2) / 120]
[Stage 0:==================================================> (110 + 2) / 120]
Here is my log4j property:
# Set everything to be logged to the console
log4j.rootCategory=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}:
%m%n
# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
I want to know how to disable this kind of stage progress message?
Best regards,
Henry
________________________________
The privileged confidential information contained in this email is intended for
use only by the addressees as indicated by the original sender of this email.
If you are not the addressee indicated in this email or are not responsible for
delivery of the email to such a person, please kindly reply to the sender
indicating this fact and delete all copies of it from your computer and network
server immediately. Your cooperation is highly appreciated. It is advised that
any unauthorized use of confidential information of Winbond is strictly
prohibited; and any information in this email irrelevant to the official
business of Winbond shall be deemed as neither given nor endorsed by Winbond.
