Gents,
I tried #3820. It doesn't work. I'm still getting the following exceptions:
Exception in thread "Thread-45" java.lang.RuntimeException: Unsupported
datatype DateType
at scala.sys.package$.error(package.scala:27)
at
org.apache.spark.sql.parquet.ParquetTypesConverter$anonfun$fromDataType$2.apply(ParquetTypes.scala:343)
at
org.apache.spark.sql.parquet.ParquetTypesConverter$anonfun$fromDataType$2.apply(ParquetTypes.scala:292)
at scala.Option.getOrElse(Option.scala:120)
at
org.apache.spark.sql.parquet.ParquetTypesConverter$.fromDataType(ParquetTypes.scala:291)
at
org.apache.spark.sql.parquet.ParquetTypesConverter$anonfun$4.apply(ParquetTypes.scala:363)
at
org.apache.spark.sql.parquet.ParquetTypesConverter$anonfun$4.apply(ParquetTypes.scala:362)
I would more than happy to fix this myself, but I would need some help
wading through the code. Could anyone explain to me what exactly is needed
to support a new data type in SparkSQL's Parquet storage engine?
Thanks.
Alex
On Mon, Dec 29, 2014 at 10:20 PM, Wang, Daoyuan <daoyuan.w...@intel.com>
wrote:
> By adding a flag in SQLContext, I have modified #3822 to include
> nanoseconds now. Since passing too many flags is ugly, now I need the whole
> SQLContext, so that we can put more flags there.
>
>
>
> Thanks,
>
> Daoyuan
>
>
>
> *From:* Michael Armbrust [mailto:mich...@databricks.com]
> *Sent:* Tuesday, December 30, 2014 10:43 AM
> *To:* Alessandro Baretta
> *Cc:* Wang, Daoyuan; dev@spark.apache.org
> *Subject:* Re: Unsupported Catalyst types in Parquet
>
>
>
> Yeah, I saw those. The problem is that #3822 truncates timestamps that
> include nanoseconds.
>
>
>
> On Mon, Dec 29, 2014 at 5:14 PM, Alessandro Baretta <alexbare...@gmail.com>
> wrote:
>
> Michael,
>
>
>
> Actually, Adrian Wang already created pull requests for these issues.
>
>
>
> https://github.com/apache/spark/pull/3820
>
> https://github.com/apache/spark/pull/3822
>
>
>
> What do you think?
>
>
>
> Alex
>
>
>
> On Mon, Dec 29, 2014 at 3:07 PM, Michael Armbrust <mich...@databricks.com>
> wrote:
>
> I'd love to get both of these in. There is some trickiness that I talk
> about on the JIRA for timestamps since the SQL timestamp class can support
> nano seconds and I don't think parquet has a type for this. Other systems
> (impala) seem to use INT96. It would be great to maybe ask on the parquet
> mailing list what the plan is there to make sure that whatever we do is
> going to be compatible long term.
>
>
>
> Michael
>
>
>
> On Mon, Dec 29, 2014 at 8:13 AM, Alessandro Baretta <alexbare...@gmail.com>
> wrote:
>
> Daoyuan,
>
> Thanks for creating the jiras. I need these features by... last week, so
> I'd be happy to take care of this myself, if only you or someone more
> experienced than me in the SparkSQL codebase could provide some guidance.
>
> Alex
>
> On Dec 29, 2014 12:06 AM, "Wang, Daoyuan" <daoyuan.w...@intel.com> wrote:
>
> Hi Alex,
>
> I'll create JIRA SPARK-4985 for date type support in parquet, and
> SPARK-4987 for timestamp type support. For decimal type, I think we only
> support decimals that fits in a long.
>
> Thanks,
> Daoyuan
>
> -----Original Message-----
> From: Alessandro Baretta [mailto:alexbare...@gmail.com]
> Sent: Saturday, December 27, 2014 2:47 PM
> To: dev@spark.apache.org; Michael Armbrust
> Subject: Unsupported Catalyst types in Parquet
>
> Michael,
>
> I'm having trouble storing my SchemaRDDs in Parquet format with SparkSQL,
> due to my RDDs having having DateType and DecimalType fields. What would it
> take to add Parquet support for these Catalyst? Are there any other
> Catalyst types for which there is no Catalyst support?
>
> Alex
>
>
>
>
>
>
>
