Michael, Actually, Adrian Wang already created pull requests for these issues.
https://github.com/apache/spark/pull/3820 https://github.com/apache/spark/pull/3822 What do you think? Alex On Mon, Dec 29, 2014 at 3:07 PM, Michael Armbrust <mich...@databricks.com> wrote: > I'd love to get both of these in. There is some trickiness that I talk > about on the JIRA for timestamps since the SQL timestamp class can support > nano seconds and I don't think parquet has a type for this. Other systems > (impala) seem to use INT96. It would be great to maybe ask on the parquet > mailing list what the plan is there to make sure that whatever we do is > going to be compatible long term. > > Michael > > On Mon, Dec 29, 2014 at 8:13 AM, Alessandro Baretta <alexbare...@gmail.com > > wrote: > >> Daoyuan, >> >> Thanks for creating the jiras. I need these features by... last week, so >> I'd be happy to take care of this myself, if only you or someone more >> experienced than me in the SparkSQL codebase could provide some guidance. >> >> Alex >> On Dec 29, 2014 12:06 AM, "Wang, Daoyuan" <daoyuan.w...@intel.com> wrote: >> >>> Hi Alex, >>> >>> I'll create JIRA SPARK-4985 for date type support in parquet, and >>> SPARK-4987 for timestamp type support. For decimal type, I think we only >>> support decimals that fits in a long. >>> >>> Thanks, >>> Daoyuan >>> >>> -----Original Message----- >>> From: Alessandro Baretta [mailto:alexbare...@gmail.com] >>> Sent: Saturday, December 27, 2014 2:47 PM >>> To: dev@spark.apache.org; Michael Armbrust >>> Subject: Unsupported Catalyst types in Parquet >>> >>> Michael, >>> >>> I'm having trouble storing my SchemaRDDs in Parquet format with >>> SparkSQL, due to my RDDs having having DateType and DecimalType fields. >>> What would it take to add Parquet support for these Catalyst? Are there any >>> other Catalyst types for which there is no Catalyst support? >>> >>> Alex >>> >> >
