I am trying to create a new JavaPairRDD from data in an HDFS file. My code is:
sparkContext = new JavaSparkContext("yarn-client", "SumFramesPerTimeUnit",
sparkConf);
JavaPairRDD inputRDD =
sparkContext.newAPIHadoopFile(fileFilter, FixedLengthInputFormat.class,
LongWritable.class, BytesWritable.
ncher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
I’ve checked and the class does live in the spark assembly. Any thoughts as
what might be wrong?
Best Regards,
David R Robison
Senior Systems Engineer
[cid:image004.png@01D19182.F24CA3E0]
From: Dav
the JavaSparkContext. I have a logging call
right after creating the SparkContext and it is never executied.
Any idea what I’m doing wrong? David
Best Regards,
David R Robison
Senior Systems Engineer
[cid:image004.png@01D19182.F24CA3E0]
From: Rohit Verma [mailto:rohit.ve...@rokittech.com
I am trying to submit a spark job through the yarn-client master setting. The
job gets created and submitted to the clients but immediately errors out. Here
is the relevant portion of the log:
15:39:37,385 INFO [org.apache.spark.deploy.yarn.Client] (default task-1)
Requesting a new application
I am trying to create a Spark javaRDD using the newAPIHadoopFile and the
FixedLengthInputFormat. Here is my code snippit,
Configuration config = new Configuration();
config.setInt(FixedLengthInputFormat.FIXED_RECORD_LENGTH, JPEG_INDEX_SIZE);
config.set("fs.hdfs.impl", DistributedFileSystem.class.