On Wed, Oct 29, 2014 at 3:29 PM, spr <s...@yarcdata.com> wrote:
> I am processing a log file, from each line of which I want to extract the
> zeroth and 4th elements (and an integer 1 for counting) into a tuple. I
> had
> hoped to be able to index the Array for elements 0 and 4, but Arrays appear
> not to support vector indexing. I'm not finding a way to extract and
> combine the elements properly, perhaps due to being a SparkStreaming/Scala
> newbie.
>
> My code so far looks like:
>
> 1] var lines = ssc.textFileStream(dirArg)
> 2] var linesArray = lines.map( line => (line.split("\t")))
> 3] var respH = linesArray.map( lineArray => lineArray(4) )
> 4a] var time = linesArray.map( lineArray => lineArray(0) )
> 4b] var time = linesArray.map( lineArray => (lineArray(0), 1))
> 5] var newState = respH.union(time)
>
> If I use line 4a and not 4b, it compiles properly. (I still have issues
> getting my update function to updateStateByKey working, so don't know if it
> _works_ properly.)
>
> If I use line 4b and not 4a, it fails at compile time with
>
> [error] foo.scala:82: type mismatch;
> [error] found : org.apache.spark.streaming.dstream.DStream[(String,
> Int)]
> [error] required: org.apache.spark.streaming.dstream.DStream[String]
> [error] var newState = respH.union(time)
>
> This implies that the DStreams being union()ed have to be of identical
> per-element type. Can anyone confirm that's true?
>
Yes. As shown in the scaladoc/javadoc they have to be the same type. (
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.dstream.DStream
)
>
> If so, is there a way to extract the needed elements and build the new
> DStream?
>
Maybe you can say what you want your new DStream to look like? If you just
want to extract the zero and fourth elements and have them together I'd do
the extraction in a single map
e.g. something like
val iLikeCoffeeDStream = linesArrayDStream.map(lineArray => (lineArray(0),
lineArray(4))
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-extract-combine-elements-of-an-Array-in-DStream-element-tp17676.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
--
Cell : 425-233-8271
