Hi all, im currently reading a csv file shich has the following format:
(String, Double, Double,Double, Double, Double)
and can map this no problems using:
val dataRDD = sc.textFile("file.csv").
map(_.split (",")).
map(a=> (Array(a(0)), Array(a(1).toDouble, a(2).toDouble), a(3),
Array(a(4).toDouble, a(5).toDouble)))
What i would like to do is because the input file may have a different
number of fields ie it might have an extra double which needs to go in the
first array of doubles ie:
(String, Double, Double,Double, Double,Double, Double)
which would see my map as
val dataRDD = sc.textFile("file.csv").
map(_.split (",")).
map(a=> (Array(a(0)), Array(a(1).toDouble, a(2).toDouble,
a(3).toDouble), a(4), Array(a(5).toDouble, a(6).toDouble)))
Is there a way i can make this map more dynamic ie if i create vals:
val Array_1 = 3
val Array_2 = 2
Then use these to pick up the values for array 1 which we know should
contain 3 values and say okay give me a(1) through to a(3)
thanks in advance
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/reading-a-csv-dynamically-tp21304.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
