Dongjoon Hyun created SPARK-18860:
-------------------------------------
Summary: Update Parquet to 1.9.0
Key: SPARK-18860
URL: https://issues.apache.org/jira/browse/SPARK-18860
Project: Spark
Issue Type: Bug
Components: SQL
Reporter: Dongjoon Hyun
This issue aims to update Parquet to 1.9.0 and remove the hacks due to Parquet
1.8.1 limitation.
{code}
- // !! HACK ALERT !!
- //
- // PARQUET-363 & PARQUET-278: parquet-mr 1.8.1 doesn't allow constructing
empty GroupType,
- // which prevents us to avoid selecting any columns for queries like `SELECT
COUNT(*) FROM t`.
- // This issue has been fixed in parquet-mr 1.8.2-SNAPSHOT.
- //
- // To workaround this problem, here we first construct a `MessageType` with
a single dummy
- // field, and then remove the field to obtain an empty `MessageType`.
- //
- // TODO Reverts this change after upgrading parquet-mr to 1.8.2+
val EMPTY_MESSAGE = Types
.buildMessage()
- .required(PrimitiveType.PrimitiveTypeName.INT32).named("dummy")
.named(ParquetSchemaConverter.SPARK_PARQUET_SCHEMA_NAME)
- EMPTY_MESSAGE.getFields.clear()
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
