Error in run multiple unit test that extends DataFrameSuiteBase

2016-09-23 Thread Jinyuan Zhou
After I created two test case that FlatSpec with DataFrameSuiteBase. But I got errors when do sbt test. I was able to run each of them separately. My test cases does use sqlContext to read files. Here is the exception stack. Judging from the exception, I may need to unregister RpcEndpoint after

NPE in using AvroKeyValueInputFormat for newAPIHadoopFile

2015-12-15 Thread Jinyuan Zhou
Hi, I tried to load avro files in hdfs but keep getting NPE. I am using AvroKeyValueInputFormat inside newAPIHadoopFile method. Anyone have any clue? Here is stack trace Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 0.0 failed 4