[
https://issues.apache.org/jira/browse/SPARK-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14392992#comment-14392992
]
Platon Potapov commented on SPARK-6569:
---------------------------------------
If we talk about log level reduction only, then no, I don't feel strongly about
the change.
But I would suggest not logging anything at all. Cody's rationale behind the
warning is for empty input batches to not go unnoticed. I feel that is a
responsibility of a monitoring agent external to the spark job - an agent
knowing the specifics of the use case the job is being run in, and hence having
the capacity to decide whether an empty batch is an abnormal situation (in my
case, it isn't).
If that's not an option - well, please close the ticket then.
> Kafka directInputStream logs what appear to be incorrect warnings
> -----------------------------------------------------------------
>
> Key: SPARK-6569
> URL: https://issues.apache.org/jira/browse/SPARK-6569
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.3.0
> Environment: Spark 1.3.0
> Reporter: Platon Potapov
> Priority: Minor
>
> During what appears to be normal operation of streaming from a Kafka topic,
> the following log records are observed, logged periodically:
> {code}
> [Stage 391:==========================================> (3 + 0) /
> 4]
> 2015-03-27 12:49:54 WARN KafkaRDD: Beginning offset ${part.fromOffset} is the
> same as ending offset skipping raw 0
> 2015-03-27 12:49:54 WARN KafkaRDD: Beginning offset ${part.fromOffset} is the
> same as ending offset skipping raw 0
> 2015-03-27 12:49:54 WARN KafkaRDD: Beginning offset ${part.fromOffset} is the
> same as ending offset skipping raw 0
> {code}
> * the part.fromOffset placeholder is not correctly substituted to a value
> * is the condition really mandates a warning being logged?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
