Hello, I'm interested in reading/writing parquet SchemaRDDs that support the Parquet Decimal converted type. The first thing I did was update the Spark parquet dependency to version 1.5.0, as this version introduced support for decimals in parquet. However, conversion between the catalyst decimal type and the parquet decimal type is complicated by the fact that the catalyst type does not specify a decimal precision and scale but the parquet type requires them.
I'm wondering if perhaps we could add an optional precision and scale to the catalyst decimal type? The catalyst decimal type would have unspecified precision and scale by default for backwards compatibility, but users who want to serialize a SchemaRDD with decimal(s) to parquet would have to narrow their decimal type(s) by specifying a precision and scale. Thoughts? Michael --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
