[
https://issues.apache.org/jira/browse/HIVE-29805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-29805:
----------------------------------
Labels: pull-request-available (was: )
> HiveIgnoreKeyTextOutputFormat does not write header/footer lines for
> skip.header/footer.line.count, causing data loss on append
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29805
> URL: https://issues.apache.org/jira/browse/HIVE-29805
> Project: Hive
> Issue Type: Bug
> Components: File Formats
> Reporter: Deng Ziming
> Priority: Major
> Labels: pull-request-available
>
> For text tables created with TBLPROPERTIES('skip.header.line.count'='N')
> and/or
> ('skip.footer.line.count'='M'), INSERT and INSERT OVERWRITE produce data files
> that do NOT contain those N header lines or M footer lines.
> On read, Hive's SkippingTextInputFormat skips the first N lines and the last M
> lines of every file. When a new file is appended without a header/footer, the
> reader still skips the first/last N/M lines of that file, so real data rows
> are
> lost. This is especially visible on INSERT INTO / append operations.
> Reproduce:
> CREATE TABLE t (a string, b string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS TEXTFILE
> TBLPROPERTIES ('skip.header.line.count'='1');
> INSERT OVERWRITE TABLE t VALUES ('x','y'),('a','b'),('c','d');
> SELECT * FROM t;
> Expected:
> x y
> a b
> c d
> Actual (before fix):
> a b
> c d
> (the first data row 'x y' is skipped as if it were the header)
> Root cause:
> HiveIgnoreKeyTextOutputFormat only serializes data rows. It ignores the table
> properties skip.header.line.count and skip.footer.line.count, so the writer
> never emits the header/footer lines that the reader is configured to skip.
> Proposed fix:
> Make HiveIgnoreKeyTextOutputFormat emit a header line built from the column
> names (using the table's delimiter/quote properties) and emit M footer lines
> before closing the stream. This mirrors how Spark native CSV handles
> header=true: the writer emits a header line and the reader skips it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
