subject:"How to measure IO time in Spark over S3"

Re: How to measure IO time in Spark over S3

2017-02-13 Thread Steve Loughran

esn't do renames. The 1.6 DirectOutputCommitter did, but it couldn't handle failure & retry. Future ones will. -Steve From: Gili Nachum <gilinac...@gmail.com> Sent: 13 February 2017 06:55 To: user@spark.apache.org Subject: How to measure IO time in Spark ove

How to measure IO time in Spark over S3

2017-02-12 Thread Gili Nachum

Hi! How can I tell IO duration for a Spark application doing R/W from S3 (using S3 as a filesystem sc.textFile("s3a://...")? I would like to know the % of time doing IO of the overall app execution time. Gili.