[jira] [Created] (ZEPPELIN-5914) spark.sparkContext.textFile(...) is not working

[Amandeep (Jira)]

Amandeep created ZEPPELIN-5914:
----------------------------------

             Summary: spark.sparkContext.textFile(...) is not working 
                 Key: ZEPPELIN-5914
                 URL: https://issues.apache.org/jira/browse/ZEPPELIN-5914
             Project: Zeppelin
          Issue Type: Bug
            Reporter: Amandeep


When using spark-3.1.2 with hadoop 3.2.0 sparkContext.textFile(..) throws a NPE:

 
java.lang.NullPointerException
  at 
org.apache.hadoop.mapred.TextInputFormat.isSplitable(TextInputFormat.java:49)
  at 
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:363)
  at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:205)
  at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:300)
  at scala.Option.getOrElse(Option.scala:189)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:296)
  at 
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
  at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:300)
  at scala.Option.getOrElse(Option.scala:189)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:296)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2261)
  at org.apache.spark.rdd.RDD.count(RDD.scala:1253)
  ... 114 elided
 

Built zepplein from master branch via
mvn clean package -DskipTests -Dcheckstyle.skip -Pbuild-distr -Pspark-3.1 
-Phadoop3 -Pspark-scala-2.12 -Pscala-2.12 -Pinclude-hadoop -Pweb-dist
 

This works on spark-shell but not on zeppelins spark interpreter. 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)