hemanthumashankar0511 commented on PR #6684:
URL: https://github.com/apache/hive/pull/6684#issuecomment-6118967346
> @humashankar26 Hello, can you help to check if this is worth fixing?
Hi! @dengziming, sorry I couldn't notice this among all the traffic. I’m not
sure about the best design, but I think it’s worth fixing, since inserted rows
were silently dropped on master and are returned with the patch.
The problem is silent data loss: `INSERT` writes files with no header, but
the reader skips the first N lines of every file, so inserted rows disappear on
`SELECT` with no error.
I tested out the patch on a cluster. All results below are from my own runs.
| Test | `SELECT *` result | Raw file contents |
|---|---|---|
| header=1, 3 rows inserted | all 3 rows | column names, then the 3 data
rows |
| header=2, 2 rows inserted | exactly 2 rows | column-name line twice, then
2 data rows |
| header=1 + footer=1, 2 rows inserted | exactly 2 rows | header, 2 data
rows, one blank line |
<details>
<summary>header=1</summary>
```sql
CREATE TABLE t (a string, b string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE
TBLPROPERTIES ('skip.header.line.count'='1');
INSERT INTO t VALUES ('x','y'),('a','b'),('c','d');
SELECT * FROM t;
```
```
+------+------+
| t.a | t.b |
+------+------+
| x | y |
| a | b |
| c | d |
+------+------+
```
Raw file (`dfs -cat .../t/000000_0`). The first line is the header the
writer added; the `a,b` further down is one of the inserted rows:
```
a,b
x,y
a,b
c,d
```
</details>
<details>
<summary>header=2</summary>
```sql
CREATE TABLE t_n2 (colx string, coly string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE
TBLPROPERTIES ('skip.header.line.count'='2');
INSERT INTO t_n2 VALUES ('r1a','r1b'),('r2a','r2b');
SELECT * FROM t_n2;
```
```
+------------+------------+
| t_n2.colx | t_n2.coly |
+------------+------------+
| r1a | r1b |
| r2a | r2b |
+------------+------------+
```
Raw file:
```
colx,coly
colx,coly
r1a,r1b
r2a,r2b
```
</details>
<details>
<summary>header=1 + footer=1</summary>
```sql
CREATE TABLE t_hf (colx string, coly string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE
TBLPROPERTIES ('skip.header.line.count'='1','skip.footer.line.count'='1');
INSERT INTO t_hf VALUES ('r1a','r1b'),('r2a','r2b');
SELECT * FROM t_hf;
```
```
+------------+------------+
| t_hf.colx | t_hf.coly |
+------------+------------+
| r1a | r1b |
| r2a | r2b |
+------------+------------+
```
Raw file (last line is blank):
```
colx,coly
r1a,r1b
r2a,r2b
```
</details>
I'm new to Hive, so this is a user-level check and not a review. Committers
or PMC members who know this area better would have a better idea, so it would
be worth getting their eyes on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
