Hi,
I am not sure this is the correct approach
Read a text file in
val f = sc.textFile("/tmp/ASE15UpgradeGuide.txt")
Now I want to get rid of empty lines and filter only the lines that contain
"ASE15"
f.filter(_ > "").filter(_ contains("ASE15")).
The above works but I am not sure whether I need two filter transformation
above? Can it be done in one?
Now I want to map the above filter to lines with carriage return ans split
them by ","
f.filter(_ > "").filter(_ contains("ASE15")).map(line => (line.split("\t")))
res88: org.apache.spark.rdd.RDD[Array[String]] = MapPartitionsRDD[131] at
map at <console>:30
Now I want to split the output by ","
scala> f.filter(_ > "").filter(_ contains("ASE15")).map(line =>
(line.split("\t").split(",")))
<console>:30: error: value split is not a member of Array[String]
f.filter(_ > "").filter(_ contains("ASE15")).map(line =>
(line.split("\t").split(",")))
^
Any advice will be appreciated
Thanks
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
