Hi,
I’m running a job whose simple task it is to find files that cannot be read
(sometimes our gz files are corrupted).
With 1.0.x, this worked perfectly. Since 1.1.0 however, I’m getting an
exception:
scala.collection.mutable.ArrayOps$ofRef$.length$extension(ArrayOps.scala:114)
sc.wholeTextFiles(input)
.foreach { case (fileName, _) =>
try {
println(s"Scanning $fileName")
sc.textFile(fileName).take(1)
println(s"Successfully scanned $fileName")
} catch {
case t: Throwable => println(s"Failed to process $fileName, reason
${t.getStackTrace.head}")
}
}
Also since 1.1.0, the printlns are no longer visible on the console, only in
the Spark UI worker output.
Thanks for any help
- Marius
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]