[
https://issues.apache.org/jira/browse/SPARK-10912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370089#comment-16370089
]
Gil Vernik commented on SPARK-10912:
------------------------------------
This patch need to be generic and doesn't include "s3a" in the code, but rather
take the value from configuration. This way other connectors, will benefit from
this patch as well. [~srowen] how is this sounds?
> Improve Spark metrics executor.filesystem
> -----------------------------------------
>
> Key: SPARK-10912
> URL: https://issues.apache.org/jira/browse/SPARK-10912
> Project: Spark
> Issue Type: Improvement
> Components: Deploy
> Affects Versions: 1.5.0
> Reporter: Yongjia Wang
> Priority: Minor
> Attachments: s3a_metrics.patch
>
>
> In org.apache.spark.executor.ExecutorSource it has 2 filesystem metrics:
> "hdfs" and "file". I started using s3 as the persistent storage with Spark
> standalone cluster in EC2, and s3 read/write metrics do not appear anywhere.
> The 'file' metric appears to be only for driver reading local file, it would
> be nice to also report shuffle read/write metrics, so it can help with
> optimization.
> I think these 2 things (s3 and shuffle) are very useful and cover all the
> missing information about Spark IO especially for s3 setup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
