I am trying to create a new JavaPairRDD from data in an HDFS file. My code is:
sparkContext = new JavaSparkContext("yarn-client", "SumFramesPerTimeUnit", sparkConf); JavaPairRDD<LongWritable, BytesWritable> inputRDD = sparkContext.newAPIHadoopFile(fileFilter, FixedLengthInputFormat.class, LongWritable.class, BytesWritable.class, config); However, when I run the job I get the following error: com.fasterxml.jackson.databind.JsonMappingException: Infinite recursion (StackOverflowError) (through reference chain: scala.collection.convert.IterableWrapper[0]->org.apache.spark.rdd.RDDOperationScope["allScopes"]->scala.collection.convert.IterableWrapper[0]->org.apache.spark.rdd.RDDOperationScope["allScopes"]->...) at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:680) at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:156) at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:132) at com.fasterxml.jackson.module.scala.ser.IterableSerializer.serializeContents(IterableSerializerModule.scala:30) at com.fasterxml.jackson.module.scala.ser.IterableSerializer.serializeContents(IterableSerializerModule.scala:16) at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:185) at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:575) at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:666) at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:156) Any thoughts as to what may be going wrong? David David R Robison Senior Systems Engineer O. +1 512 247 3700 M. +1 757 286 0022 david.robi...@psgglobal.net<mailto:david.robi...@psgglobal.net> www.psgglobal.net<http://www.psgglobal.net/> Prometheus Security Group Global, Inc. 3019 Alvin Devane Boulevard Building 4, Suite 450 Austin, TX 78741