Accessing SparkConfig from mapWithState function

Hi, Is there a way to get access to SparkConfig from mapWithState function? I am looking to implement logic using the config property in mapWithState function. Thanks, Raj

Re: save RDD of Avro GenericRecord as parquet throws UnsupportedOperationException

generic.GenericRecord,org.apache.avro.generic.GenericRecord,((String, String, String), org.apache.avro.generic.GenericRecord)], org.apache.spark.sql.types.StructType) [INFO] val profileDataFrame = sqlContext.createDataFrame(mergedProfiles, schema) Thanks, Raj ___

save RDD of Avro GenericRecord as parquet throws UnsupportedOperationException

Hi, I am trying to save RDD of Avro GenericRecord as parquet. I am using Spark 1.6.1. DStreamOfAvroGenericRecord.foreachRDD(rdd => rdd.toDF().write.parquet("s3://bucket/data.parquet")) Getting the following exception. Is there a way to save Avro GenericRecord as Parquet or ORC file?