You can drop header in csv by
rddData.mapPartitionsWithIndex((partitionIdx: Int, lines: Iterator[String])
=> {
if (partitionIdx == 0) {
lines.drop(1)
}
lines
}
On May 2, 2014 6:02 PM, "SK" <skrishna...@gmail.com> wrote:
> 1) I have a csv file where one of the field has integer data but it appears
> as strings: "1", "3" etc. I tried using toInt to implcitly convert the
> strings to int after reading (field(3).toInt). But I got a
> NumberFormatException. So I defined my own conversion
> as follows, but I still get a NumberFormatException - the toInt function on
> StringOps is failing. Any idea, how I can convert strings to int?
>
> implicit def string2Int(s: String): Int = augmentString(s).toInt
>
> def getInt(k: Int) = k
>
> def main(args: Array[String]) {
> val n: Int = f("6")
> }
>
> 2) How do we drop the headers when reading csv files? Is there a csv file
> format with skipHeaders option (as in scalding) in spark?
>
> thanks
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/string-to-int-conversion-tp5261.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
