It's not actually that tough. We already use a custom Hadoop FileSystem for S3 because when we started using Spark with S3 the native FileSystem was very unreliable. Our's is based on the code from Presto. (see https://github.com/prestodb/presto/blob/master/presto-hive/src/main/java/com/facebook/presto/hive/s3/PrestoS3FileSystem.java ).
I already have a version that introduces a hash to the filename for the file that's actually written to the S3 to see if it makes a difference per https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html#get-workload-considerations . FWIW, it doesn't. I'm going to modify that experiment to override the key name like before except actually mode the file, keep track of the state, and override the rename method. The problems with this approach are: 1) it's brittle because it depends on the internal directory and file naming conventions in Hadoop and Parquet. 2) It will assume (as seems to be currently the case) that the 'rename' call is done for all files from the driver. But it should do until there's a better solution in the Hadoop committer. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
