Re: XML Data Source for Spark
here is an example with code.
http://stackoverflow.com/questions/33078221/xml-processing-in-spark
I haven't tried.
On Monday, April 25, 2016 1:06 PM, Jinan Alhajjaj
wrote:
Hi All,I am
trying to use XML data source that is used for parsing and querying XML data
with Apache Spark, for Spark SQL and data frames.I am using Apache spark
version 1.6.1 and I am using Java as a programming language. I wrote this
sample code :SparkConf conf = new
SparkConf().setAppName("parsing").setMaster("local");
JavaSparkContext sc = new JavaSparkContext(conf);
SQLContext sqlContext = new SQLContext(sc);
DataFrame df =
sqlContext.read().format("com.databricks.spark.xml").option("rowtag",
"page").load("file.xml");
When I run this code I faced a problem which is Exception in thread "main"
java.lang.NoSuchMethodError:
scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; at
com.databricks.spark.xml.XmlOptions.(XmlOptions.scala:26) at
com.databricks.spark.xml.XmlOptions$.apply(XmlOptions.scala:48) at
com.databricks.spark.xml.DefaultSource.createRelation(DefaultSource.scala:58)
at
com.databricks.spark.xml.DefaultSource.createRelation(DefaultSource.scala:44)
at
org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119) at
org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109) at
datbricxml.parsing.main(parsing.java:16).Please, I need to solve this error for
my senior project ASAP.
NumberFormatException: For input string: "0.00000"
Hi all
I am trying to read:
sc.textFile(DataFile).mapPartitions(lines => {
val parser = new CSVParser(",")
lines.map(line=>parseLineToTuple(line, parser))
})
Data looks like:
android
phone,0,0,0,,0,0,0,0,0,0,0,5,0,0,0,5,0,0.0,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0.0,0,0,0
ios
phone,0,-1,0,,0,0,0,0,0,0,1,0,0,0,0,1,0,0.0,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0.0,0,0,0
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in
stage 23055.0 failed 4 times, most recent failure: Lost task 1.3 in stage
23055.0 (TID 191607, ):
java.lang.NumberFormatException: For input string: "0.0"
Has anyone faced such issues. Is there a solution?
Thanks,
Mohamed
