Let me crash this thread to suggest this *might* be related to this problem I'm trying to solve: https://issues.apache.org/jira/browse/SPARK-4196
Basically the question there is: this blank Configuration object gets made on the driver in the saveAsNewAPIHadoopFiles call, and seems to need to be serialized to use it in foreachRDD. This fails for me and at least 2 other users I know. But I feel like I am missing something. If you're investigating handling of Configuration when enabling checkpointing with the getOrCreate method, have a look and see if you have any comments vis-a-vis this JIRA. On Tue, Nov 4, 2014 at 5:48 PM, Marcelo Vanzin <[email protected]> wrote: > On Tue, Nov 4, 2014 at 9:34 AM, Cody Koeninger <[email protected]> wrote: >> 2. Is there a reason StreamingContext.getOrCreate defaults to a blank >> hadoop configuration rather than >> org.apache.spark.deploy.SparkHadoopUtil.get.conf, >> which would pull values from spark config? > > This is probably something I overlooked when I changed the rest of the > code to use SparkHadoopUtil.get.conf. Feel free to send a PR for it, > we should get it into 1.2 so that all code creates configuration > objects in a consistent way. > > -- > Marcelo > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
