Writing Parquet from Avro objects - cannot write null value for numeric fields

[Sunita Arvind]

Hello Experts,

I am trying to allow null values in numeric fields. Here are the details of
the issue I have:
http://stackoverflow.com/questions/41492344/spark-avro-to-parquet-writing-null-values-in-number-fields

I also tried making all columns nullable by using the below function (from
one of the suggestions on web)

def setNullableStateForAllColumns( df: DataFrame, nullable: Boolean) :
DataFrame = {
  df.sqlContext.createDataFrame(df.rdd,
StructType(df.schema.map(_.copy(nullable = nullable))))
}

The printSchema shows that the columns are now nullable, but still I
cannot persist

as parquet with null in the numeric fields.

Is there a workaround to it? I need to be able to allow null values
for numeric fields

Thanks in advance.

regards

Sunita