Cheng Lian created SPARK-11007: ---------------------------------- Summary: Add dictionary support for CatalystDecimalConverter Key: SPARK-11007 URL: https://issues.apache.org/jira/browse/SPARK-11007 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.5.1, 1.5.0, 1.4.1, 1.4.0 Reporter: Cheng Lian Assignee: Cheng Lian
Currently {{CatalystDecimalConverter}} doesn't explicitly support dictionary encoding. The consequence is that, the underlying Parquet {{ColumnReader}} always sends raw {{Int}}/{{Long}}/{{Binary}} values decoded from the dictionary to {{CatalystDecimalConverter}} even if the column is encoded using a dictionary. By adding explicit dictionary support (similar to what {{CatalystStringConverter}} does), we can avoid constructing decimals repeatedly. This should be especially effective for binary backed decimals. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org