[
https://issues.apache.org/jira/browse/IMPALA-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098514#comment-18098514
]
ASF subversion and git services commented on IMPALA-12675:
----------------------------------------------------------
Commit 60a0e9a03e382c63d8be479c83bf46a74920e284 in impala's branch
refs/heads/master from Daniel Vanko
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=60a0e9a03 ]
IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
The TPC-DS data generator (dsdgen) emits some string columns in
ISO-8859-1 (Latin-1) rather than UTF-8. For example,
customer.c_birth_country holds country names whose accented letters
are single Latin-1 bytes (e.g. 0xD4 for O-circumflex in "COTE
D'IVOIRE") that are not valid UTF-8, which will cause data-load failure
if PARQUET_ANNOTATE_STRINGS_UTF8 is turned on (IMPALA-12675).
Tag the TPC-DS text tables with the serde property
'serialization.encoding'='ISO-8859-1' so Impala decodes the raw data
files to UTF-8 on read -- including when the derived Parquet (and
other format) tables are populated from the text tables via
INSERT ... SELECT. The raw dsdgen output is kept as-is on disk, and
this exercises the serialization.encoding feature (IMPALA-10319) in
the standard data load.
The property is applied via ALTER TABLE ... SET SERDEPROPERTIES in the
LOAD section rather than at CREATE: Impala's CREATE does not accept a
custom SERDE, and 'serialization.encoding' is honored only from serde
properties (not TBLPROPERTIES). The LOAD section is the text-only load
phase, so the ALTER runs before the derived tables read the data.
Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Assisted-by: Claude Opus 4.8 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24585
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Set PARQUET_ANNOTATE_STRINGS_UTF8 to true by default
> ----------------------------------------------------
>
> Key: IMPALA-12675
> URL: https://issues.apache.org/jira/browse/IMPALA-12675
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Zoltán Borók-Nagy
> Assignee: Dániel Gábor Vankó
> Priority: Blocker
>
> Now that we have the BINARY type for binary data, and also have UTF-8
> support, it makes sense to set PARQUET_ANNOTATE_STRINGS_UTF8 to true by
> default in a new major release
> *We should only resolve this issue if we are ready for a new major release.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
