[
https://issues.apache.org/jira/browse/SPARK-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313020#comment-14313020
]
Patrick Wendell commented on SPARK-5647:
----------------------------------------
Isn't it just possible to get the file path in the case of file output format,
and then read the size of that file? The main challenge I see is how quickly
that size becomes visible to the HDFS client. In general I think it's worth
doing because a lot of people still use older versions of the Spark HDFS
client, for instance people based on AWS who primarily read from S3 and don't
keep up to date with the newest Hadoop API's.
> Output metrics do not show up for older hadoop versions (< 2.5)
> ---------------------------------------------------------------
>
> Key: SPARK-5647
> URL: https://issues.apache.org/jira/browse/SPARK-5647
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Reporter: Kostas Sakellis
>
> Need to add output metrics for hadoop < 2.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
