That sounds like allot of work and if I understand you correctly it sounds like a piece of middleware that already exists (I could be wrong). Alluxio?
Good luck and let us know how it goes! Gary On 20 November 2017 at 14:10, Jim Carroll <jimfcarr...@gmail.com> wrote: > Thanks. In the meantime I might just write a custom file system that maps > writes to parquet file parts to their final locations and then skips the > move. > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
