[ https://issues.apache.org/jira/browse/SPARK-16926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Davies Liu resolved SPARK-16926. -------------------------------- Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 Issue resolved by pull request 14515 [https://github.com/apache/spark/pull/14515] > Partition columns are present in columns metadata for partition but not table > ----------------------------------------------------------------------------- > > Key: SPARK-16926 > URL: https://issues.apache.org/jira/browse/SPARK-16926 > Project: Spark > Issue Type: Bug > Components: SQL > Reporter: Brian Cho > Fix For: 2.0.1, 2.1.0 > > > A change introduced in SPARK-14388 removes partition columns from the column > metadata of tables, but not for partitions. This causes TableReader to > believe that the schema is different and create an unnecessary conversion > object inspector, taking the else codepath in TableReader below: > {code} > val soi = if > (rawDeser.getObjectInspector.equals(tableDeser.getObjectInspector)) { > rawDeser.getObjectInspector.asInstanceOf[StructObjectInspector] > } else { > ObjectInspectorConverters.getConvertedOI( > rawDeser.getObjectInspector, > tableDeser.getObjectInspector).asInstanceOf[StructObjectInspector] > } > {code} > Printing the properties as debug output confirms the difference for the Hive > table. > Table properties (tableDesc.getProperties): > {code} > 16/08/04 20:36:58 DEBUG HadoopTableReader: columns.types, > string:bigint:string:bigint:bigint:array<string> > {code} > Partition properties (partProps): > {code} > 16/08/04 20:36:58 DEBUG HadoopTableReader: columns.types, > string:bigint:string:bigint:bigint:array<string>:string:string:string > {code} > Where the final three string columns are partition columns -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org