Thanks Sean. I want to load each batch into Redshift. What's the best/most efficient way to do that?
Vadim > On Apr 16, 2015, at 1:35 PM, Sean Owen <so...@cloudera.com> wrote: > > You can't, since that's how it's designed to work. Batches are saved > in different "files", which are really directories containing > partitions, as is common in Hadoop. You can move them later, or just > read them where they are. > > On Thu, Apr 16, 2015 at 6:32 PM, Vadim Bichutskiy > <vadim.bichuts...@gmail.com> wrote: >> I am using Spark Streaming where during each micro-batch I output data to S3 >> using >> saveAsTextFile. Right now each batch of data is put into its own directory >> containing >> 2 objects, "_SUCCESS" and "part-00000." >> >> How do I output each batch into a common directory? >> >> Thanks, >> Vadim >> ᐧ --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
