[
https://issues.apache.org/jira/browse/IMPALA-5961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098513#comment-18098513
]
ASF subversion and git services commented on IMPALA-5961:
---------------------------------------------------------
Commit 60a0e9a03e382c63d8be479c83bf46a74920e284 in impala's branch
refs/heads/master from Daniel Vanko
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=60a0e9a03 ]
IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
The TPC-DS data generator (dsdgen) emits some string columns in
ISO-8859-1 (Latin-1) rather than UTF-8. For example,
customer.c_birth_country holds country names whose accented letters
are single Latin-1 bytes (e.g. 0xD4 for O-circumflex in "COTE
D'IVOIRE") that are not valid UTF-8, which will cause data-load failure
if PARQUET_ANNOTATE_STRINGS_UTF8 is turned on (IMPALA-12675).
Tag the TPC-DS text tables with the serde property
'serialization.encoding'='ISO-8859-1' so Impala decodes the raw data
files to UTF-8 on read -- including when the derived Parquet (and
other format) tables are populated from the text tables via
INSERT ... SELECT. The raw dsdgen output is kept as-is on disk, and
this exercises the serialization.encoding feature (IMPALA-10319) in
the standard data load.
The property is applied via ALTER TABLE ... SET SERDEPROPERTIES in the
LOAD section rather than at CREATE: Impala's CREATE does not accept a
custom SERDE, and 'serialization.encoding' is honored only from serde
properties (not TBLPROPERTIES). The LOAD section is the text-only load
phase, so the ALTER runs before the derived tables read the data.
Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Assisted-by: Claude Opus 4.8 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24585
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Test data for TPC-DS schema contains a non-Unicode character
> ------------------------------------------------------------
>
> Key: IMPALA-5961
> URL: https://issues.apache.org/jira/browse/IMPALA-5961
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Affects Versions: Impala 2.10.0
> Reporter: Tim Wood
> Assignee: Dániel Gábor Vankó
> Priority: Major
> Labels: TPCDS
> Fix For: Impala 5.0.0
>
> Attachments: ttq-50.out
>
>
> The customer table contains rows whose c_birth_country values contain
> character 0xd4 (o-circumflex) in an illegal position for Unicode. This causes
> tpcds-q30 to fail. Either tests need to change to accommodate the different
> character set, or the test data should change to contain the proper Unicode
> character.
> To reproduce, build a mini-cluster and load with test data (./buildall.sh
> -testdata ...), then run the query from the attached file. Find the affected
> rows with:
> SELECT * FROM customer WHERE c_birth_country LIKE '%IVOIRE%';
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
