You can store the list of keys (I believe you use them in source file path, right?) in a file, one key per line. Then you can read the file using sc.textFile (So you will get a RDD of file paths) and then apply your function as a map.
r = sc.textFile(list_file).map(your_function) HTH On Sun, Feb 12, 2017 at 10:04 PM, Sam Elamin <hussam.ela...@gmail.com> wrote: > Hey folks > > Really simple question here. I currently have an etl pipeline that reads > from s3 and saves the data to an endstore > > > I have to read from a list of keys in s3 but I am doing a raw extract then > saving. Only some of the extracts have a simple transformation but overall > the code looks the same > > > I abstracted away this logic into a method that takes in an s3 path does > the common transformations and saves to source > > > But the job takes about 10 mins or so because I'm iteratively going down a > list of keys > > Is it possible to asynchronously do this? > > FYI I'm using spark.read.json to read from s3 because it infers my schema > > Regards > Sam > -- Best Regards, Ayan Guha
