[
https://issues.apache.org/jira/browse/HIVE-29785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-29785:
----------------------------------
Labels: pull-request-available (was: )
> SkippingTextInputFormat: ClassCastException on skip.header.line.count files
> with lone-CR (\r) line endings
> ----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29785
> URL: https://issues.apache.org/jira/browse/HIVE-29785
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 4.2.1
> Reporter: Smruti Biswal
> Assignee: Smruti Biswal
> Priority: Major
> Labels: pull-request-available
>
> A SELECT count(*) on an uncompressed TEXTFILE table that has
> 'skip.header.line.count'='1' fails during Tez split generation with:
> {code:java}
> java.lang.ClassCastException: class java.io.PushbackInputStream cannot be
> cast to class org.apache.hadoop.fs.Seekable (java.io.PushbackInputStream is
> in module java.base of loader 'bootstrap'; org.apache.hadoop.fs.Seekable is
> in unnamed module of loader 'app')
> at
> org.apache.hadoop.fs.FSDataInputStream.getPos(FSDataInputStream.java:80)
> at
> org.apache.hadoop.hive.ql.io.SkippingTextInputFormat.getCachedStartIndex(SkippingTextInputFormat.java:127)
> at
> org.apache.hadoop.hive.ql.io.SkippingTextInputFormat.makeSplitInternal(SkippingTextInputFormat.java:76)
> at
> org.apache.hadoop.hive.ql.io.SkippingTextInputFormat.makeSplit(SkippingTextInputFormat.java:69)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:382)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:543)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:851)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:289){code}
> The issue appears only on lone-CR (\r) line endings. LF (\n) and CRLF (\r\n)
> are unaffected.
> Even if bare-CR isn't valid CSV, the ClassCastException is misleading. It
> should fail with a
> clear error message relating to line terminator or be handled.
> *Repro :*
>
> {code:java}
> # printf "id;name;place;\n1;smruti;ctc;\n2;biswal;bbsr;\n3;'';NULL;\n" >
> data1.csv
> # tr '\n' '\r' < data1.csv > data1_cr_only.csv
> # od -c data1_cr_only.csv
> 0000000 i d ; n a m e ; p l a c e ; \r 1
> 0000020 ; s m r u t i ; c t c ; \r 2 ; b
> 0000040 i s w a l ; b b s r ; \r 3 ; ' '
> 0000060 ; N U L L ; \r
> 0000067
> {code}
> Use the data1_cr_only.csv in a table with _skip.header.line.count_ or
> _skip.footer.line.count_ set.
> {code:java}
> CREATE TABLE text_split_test (id INT, name STRING, place STRING)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY ';'
> STORED AS TEXTFILE
> TBLPROPERTIES ('skip.header.line.count' = '1');
> SELECT COUNT(*) from text_split_test;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
