You need to put randomness into the beginning of the key, if you put it other than into the beginning, it's not guaranteed that you're going to have good performance.
The way we achieved this is by writing to HDFS first, and then having a custom DistCp implemented using Spark that copies parquet files using random keys, and then saves the list of resulting keys to S3, and when we want to use those parquet files, we just need to load the listing file, and then take keys from it and pass them into the loader. You only need to do this when you have way too many files, if the number of keys you operate is reasonably small (let's say, in thousands), you won't get any benefits. Also the S3 buckets have internal optimizations, and overtime it adjusts to the workload, i.e. some additional underlying partitions are getting added, some splits happen, etc. If you want to have good performance from start, you would need to use randomization, yes. Or alternatively, you can contact AWS and tell them about the naming schema that you're going to have (but it must be set in stone), and then they can try to pre-optimize the bucket for you. On Thu, Mar 8, 2018 at 11:42 AM, Subhash Sriram <subhash.sri...@gmail.com> wrote: > Hey Spark user community, > > I am writing Parquet files from Spark to S3 using S3a. I was reading this > article about improving S3 bucket performance, specifically about how it > can help to introduce randomness to your key names so that data is written > to different partitions. > > https://aws.amazon.com/premiumsupport/knowledge- > center/s3-bucket-performance-improve/ > > Is there a straight forward way to accomplish this randomness in Spark via > the DataSet API? The only thing that I could think of would be to actually > split the large set into multiple sets (based on row boundaries), and then > write each one with the random key name. > > Is there an easier way that I am missing? > > Thanks in advance! > Subhash > > >
