Hi, For hdfs files written with below code:
rdd.saveAsTextFile(getHdfsPath(...), classOf [org.apache.hadoop.io.compress.GzipCodec]) I can see the hdfs files been generated: 0 /lz/streaming/am/1441734600000/_SUCCESS 1.6 M /lz/streaming/am/1441734600000/part-00000.gz 1.6 M /lz/streaming/am/1441734600000/part-00001.gz 1.6 M /lz/streaming/am/1441734600000/part-00002.gz ... How do I read it using SparkContext? My naive attempt: val t1 = sc.textFile("/lz/streaming/am/1441734600000") t1.take(1).head did not work: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/lz/streaming/am/1441734600000 at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:285) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:304) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:207) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) Thanks, Shenyan