Smruti Biswal created HIVE-29785:
------------------------------------
Summary: SkippingTextInputFormat: ClassCastException on
skip.header.line.count files with lone-CR (\r) line endings
Key: HIVE-29785
URL: https://issues.apache.org/jira/browse/HIVE-29785
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 4.2.1
Reporter: Smruti Biswal
A SELECT count(*) on an uncompressed TEXTFILE table that has
'skip.header.line.count'='1' fails during Tez split generation with:
{code:java}
java.lang.ClassCastException: class java.io.PushbackInputStream cannot be cast
to class org.apache.hadoop.fs.Seekable (java.io.PushbackInputStream is in
module java.base of loader 'bootstrap'; org.apache.hadoop.fs.Seekable is in
unnamed module of loader 'app')
at
org.apache.hadoop.fs.FSDataInputStream.getPos(FSDataInputStream.java:80)
at
org.apache.hadoop.hive.ql.io.SkippingTextInputFormat.getCachedStartIndex(SkippingTextInputFormat.java:127)
at
org.apache.hadoop.hive.ql.io.SkippingTextInputFormat.makeSplitInternal(SkippingTextInputFormat.java:76)
at
org.apache.hadoop.hive.ql.io.SkippingTextInputFormat.makeSplit(SkippingTextInputFormat.java:69)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:382)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:543)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:851)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:289){code}
The issue appears only on lone-CR (\r) line endings. LF (\n) and CRLF (\r\n)
are unaffected.
Even if bare-CR isn't valid CSV, the ClassCastException is misleading. It
should fail with a
clear error message relating to line terminator or be handled.
*Repro :*
{code:java}
# printf "id;name;place;\n1;smruti;ctc;\n2;biswal;bbsr;\n3;'';NULL;\n" >
data1.csv
# tr '\n' '\r' < data1.csv > data1_cr_only.csv
# od -c data1_cr_only.csv
0000000 i d ; n a m e ; p l a c e ; \r 1
0000020 ; s m r u t i ; c t c ; \r 2 ; b
0000040 i s w a l ; b b s r ; \r 3 ; ' '
0000060 ; N U L L ; \r
0000067
{code}
Use the data1_cr_only.csv in a table with _skip.header.line.count_ or
_skip.footer.line.count_ set.
{code:java}
CREATE TABLE text_split_test (id INT, name STRING, place STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ';'
STORED AS TEXTFILE
TBLPROPERTIES ('skip.header.line.count' = '1'); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
