الجنرالOn 2015/03/18 17:20:54 Ayoub wrote:> In case it would interest other
peoples, here is what I come up with and it> seems to work fine:> > case
class RDDAsInputStream(private val rdd: RDD[String]) extends>
java.io.InputStream {> var bytes =
rdd.flatMap(_.getBytes("UTF-8")).toLocalIterator> > def read(): Int = {>
if(bytes.hasNext) bytes.next.toInt> else -1> }> override def
markSupported(): Boolean = false> }> > > 2015-03-13 13:56 GMT+01:00 Sean Owen
<so...@cloudera.com>:> > > OK, then you do not want to collect() the RDD. You
can get an iterator,> > yes.> > There is no such thing as making an Iterator
into an InputStream. An> > Iterator is a sequence of arbitrary objects; an
InputStream is a> > channel to a stream of bytes.> > I think you can employ
similar Guava / Commons utilities to make an> > Iterator of Streams in a stream
of Readers, join the Readers, and> > encode the result as bytes in an
InputStream.> >> > On Fri, Mar 13, 2015 at 10:33 AM, Ayoub <be...@gmail.com>> >
wrote:> > > Thanks Sean,> > >> > > I forgot to mention that the data is too big
to be collected on the> > driver.> > >> > > So yes your proposition would work
in theory but in my case I cannot hold> > > all the data in the driver memory,
therefore it wouldn't work.> > >> > > I guess the crucial point to to do the
collect in a lazy way and in that> > > subject I noticed that we can get a
local iterator from an RDD but that> > > rises two questions:> > >> > > - does
that involves an mediate collect just like with "collect()" or is> > it> > >
lazy process ?> > > - how to go from an iterator to an InputStream ?> > >> > >>
> > 2015-03-13 11:17 GMT+01:00 Sean Owen <[hidden email]>:> > >>> > >> These
are quite different creatures. You have a distributed set of> > >> Strings, but
want a local stream of bytes, which involves three> > >> conversions:> > >>> >
>> - collect data to driver> > >> - concatenate strings in some way> > >> -
encode strings as bytes according to an encoding> > >>> > >> Your approach is
OK but might be faster to avoid disk, if you have> > >> enough memory:> > >>> >
>> - collect() to a Array[String] locally> > >> - use Guava utilities to turn a
bunch of Strings into a Reader> > >> - Use the Apache Commons ReaderInputStream
to read it as encoded bytes> > >>> > >> I might wonder if that's all really
what you want to do though.> > >>> > >>> > >> On Fri, Mar 13, 2015 at 9:54 AM,
Ayoub <[hidden email]> wrote:> > >> > Hello,> > >> >> > >> > I need to convert
an RDD[String] to a java.io.InputStream but I didn't> > >> > find> > >> > an
east way to do it.> > >> > Currently I am saving the RDD as temporary file and
then opening an> > >> > inputstream on the file but that is not really
optimal.> > >> >> > >> > Does anybody know a better way to do that ?> > >> >> >
>> > Thanks,> > >> > Ayoub.> > >> >> > >> >> > >> >> > >> > --> > >> > View
this message in context:> > >> >> >
http://apache-spark-user-list.1001560.n3.nabble.com/RDD-to-InputStream-tp22031.html>
> >> > Sent from the Apache Spark User List mailing list archive at> >
Nabble.com.> > >> >> > >> >
---------------------------------------------------------------------> > >> >
To unsubscribe, e-mail: [hidden email]> > >> > For additional commands, e-mail:
[hidden email]> > >> >> > >> > >> > >> > > ________________________________> >
> View this message in context: Re: RDD to InputStream> > >> > > Sent from the
Apache Spark User List mailing list archive at Nabble.com.> >> > > > > --> View
this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Re-RDD-to-InputStream-tp22121.html>
Sent from the Apache Spark User List mailing list archive at Nabble.com.مرسل
من هاتف Samsung Galaxy الذكي.
