Sorry forgot to paste by code:
import org.apache.hadoop.io.LongWritable
import org.apache.hadoop.io.Text
import org.apache.hadoop.mapred.{FileSplit, TextInputFormat}
import org.apache.spark.rdd.HadoopRDD
val poolAllPath:String = "file:///Users/wulei3/Coding/data/car-data/2015"
val poolAllFile = sc.hadoopFile(poolAllPath, classOf[TextInputFormat],
classOf[LongWritable], classOf[Text], sc.defaultMinPartitions)
val poolAllRDD = poolAllFile.asInstanceOf[HadoopRDD[LongWritable, Text]]
val poolAll = poolAllRDD.mapPartitionsWithInputSplit {
case (inputSplit, iterator) =>
val file = inputSplit.asInstanceOf[FileSplit]
iterator.filter(_._2.toString.split(",").length.equals(2)).map {
case tpl =>
(file.getPath.toString.substring(70,76),
tpl._2.toString.split(",")(1))
}
}
val table1 = poolAll.map{
case (year_month:String, num:String) =>
(year_month, 1)
}.reduceByKey(_+_).collect.toList.toDF("year_month", "num_lotters")
table1.registerTempTable("tb")
%sql select * from tb
------------------ Original ------------------
From: "MichaelYoung";<[email protected]>;
Date: Sun, Aug 7, 2016 12:59 PM
To: "[email protected]"<[email protected]>;
Subject: Re: RE: Zeppelin JSON files - Table not found
Setting zeppelin.spark.useHiveContext to false did not solve this problem in my
notebook, did I miss something ?
Zeppelin version : 0.6.0
Spark version: 1.6.2
Settings:
"properties": {
"spark.executor.memory": "1G",
"zeppelin.spark.printREPLOutput": "true",
"spark.cores.max": "2",
"zeppelin.spark.sql.stacktrace": "true",
"zeppelin.spark.importImplicit": "true",
"zeppelin.spark.concurrentSQL": "true",
"zeppelin.spark.useHiveContext": "false",
"zeppelin.pyspark.python": "python",
"zeppelin.dep.localrepo": "local-repo",
"zeppelin.R.knitr": "true",
"zeppelin.spark.maxResult": "1000",
"master": "local[*]",
"spark.app.name": "Zeppelin-over-Spark",
"zeppelin.R.image.width": "100%",
------------------ Original ------------------
From: "Joaquin Alzola";<[email protected]>;
Date: Fri, Jul 29, 2016 00:00 AM
To: "[email protected]"<[email protected]>;
Subject: RE: Zeppelin JSON files - Table not found
>ERROR:
>org.apache.spark.sql.AnalysisException: Table not found: facebook; line 2 pos
>5
Changing the zeppelin.spark.useHiveContext to false solved the issue.
What format does Zeppelin need the JSON to be?
Playing aroung everything on the same line only picks up the last , while in
different line it give corrupt JSON.
This email is confidential and may be subject to privilege. If you are not the
intended recipient, please do not copy or disclose its content but contact the
sender immediately upon receipt.
