Deng Ziming created HIVE-29805:
----------------------------------
Summary: HiveIgnoreKeyTextOutputFormat does not write
header/footer lines for skip.header/footer.line.count, causing data loss on
append
Key: HIVE-29805
URL: https://issues.apache.org/jira/browse/HIVE-29805
Project: Hive
Issue Type: Bug
Components: File Formats
Reporter: Deng Ziming
For text tables created with TBLPROPERTIES('skip.header.line.count'='N') and/or
('skip.footer.line.count'='M'), INSERT and INSERT OVERWRITE produce data files
that do NOT contain those N header lines or M footer lines.
On read, Hive's SkippingTextInputFormat skips the first N lines and the last M
lines of every file. When a new file is appended without a header/footer, the
reader still skips the first/last N/M lines of that file, so real data rows are
lost. This is especially visible on INSERT INTO / append operations.
Reproduce:
CREATE TABLE t (a string, b string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE
TBLPROPERTIES ('skip.header.line.count'='1');
INSERT OVERWRITE TABLE t VALUES ('x','y'),('a','b'),('c','d');
SELECT * FROM t;
Expected:
x y
a b
c d
Actual (before fix):
a b
c d
(the first data row 'x y' is skipped as if it were the header)
Root cause:
HiveIgnoreKeyTextOutputFormat only serializes data rows. It ignores the table
properties skip.header.line.count and skip.footer.line.count, so the writer
never emits the header/footer lines that the reader is configured to skip.
Proposed fix:
Make HiveIgnoreKeyTextOutputFormat emit a header line built from the column
names (using the table's delimiter/quote properties) and emit M footer lines
before closing the stream. This mirrors how Spark native CSV handles
header=true: the writer emits a header line and the reader skips it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
