Can you show us the code that you are using? This might help. This is the updated streaming programming guide for 1.3, soon to be up, this is a quick preview. http://people.apache.org/~tdas/spark-1.3.0-temp-docs/streaming-programming-guide.html#dataframe-and-sql-operations
TD On Wed, Mar 11, 2015 at 12:20 PM, Marius Soutier <[email protected]> wrote: > Forgot to mention, it works when using .foreachRDD(_.saveAsTextFile(“”)). > > > On 11.03.2015, at 18:35, Marius Soutier <[email protected]> wrote: > > > > Hi, > > > > I’ve written a Spark Streaming Job that inserts into a Parquet, using > stream.foreachRDD(_insertInto(“table”, overwrite = true). Now I’ve added > checkpointing; everything works fine when starting from scratch. When > starting from a checkpoint however, the job doesn’t work and produces the > following exception in the foreachRDD: > > > > ERROR org.apache.spark.streaming.scheduler.JobScheduler: Error running > job streaming job 1426093830000 ms.2 > > org.apache.spark.SparkException: RDD transformations and actions can > only be invoked by the driver, not inside of other transformations; for > example, rdd1.map(x => rdd2.values.count() * x) is invalid because the > values transformation and count action cannot be performed inside of the > rdd1.map transformation. For more information, see SPARK-5063. > > at org.apache.spark.rdd.RDD.sc(RDD.scala:90) > > at org.apache.spark.rdd.RDD.<init>(RDD.scala:143) > > at org.apache.spark.sql.SchemaRDD.<init>(SchemaRDD.scala:108) > > at > org.apache.spark.sql.SQLContext.createSchemaRDD(SQLContext.scala:114) > > at > MyStreamingJob$$anonfun$createComputation$3.apply(MyStreamingJob:167) > > at > MyStreamingJob$$anonfun$createComputation$3.apply(MyStreamingJob:167) > > at > org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1.apply(DStream.scala:535) > > at > org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1.apply(DStream.scala:535) > > at > org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:42) > > at > org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40) > > at > org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40) > > > > > > > > > > Cheers > > - Marius > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
