----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15663/#review29253 -----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java <https://reviews.apache.org/r/15663/#comment56427> I think this code to separate header reading and initializing footer buffer a separate function to improve readability. ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java <https://reviews.apache.org/r/15663/#comment56424> can you make this a constant string and referce the constant variable name ? Unfortunately (for some historic reason that I don't understand) the place where all the similar constants are being stored is in serde/if/serde.thrift ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java <https://reviews.apache.org/r/15663/#comment56425> I think it is better fail here and let the user know, rather than use a smaller footer skip lines than what user specified. ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java <https://reviews.apache.org/r/15663/#comment56426> how about using hive's ObjectPair instead ? ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java <https://reviews.apache.org/r/15663/#comment56401> Sorry, I had failed to notice that this is a table property. But we need to document this table property. ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java <https://reviews.apache.org/r/15663/#comment56404> This does not handle the case where multiple splits are combined together. We need to disable combination of splits in CombineHiveInputFormat getSplits() call. ql/src/test/org/apache/hadoop/hive/ql/io/TestHiveBinarySearchRecordReader.java <https://reviews.apache.org/r/15663/#comment56422> Is this change for avoiding some NPE ? ql/src/test/queries/clientpositive/file_with_header_footer.q <https://reviews.apache.org/r/15663/#comment56423> can you add some negative tests where skip.footer.number exceeds hive.file.max.footer ? - Thejas Nair On Nov. 19, 2013, 1:31 a.m., Eric Hanson wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/15663/ > ----------------------------------------------------------- > > (Updated Nov. 19, 2013, 1:31 a.m.) > > > Review request for hive and Thejas Nair. > > > Bugs: HIVE-5795 > https://issues.apache.org/jira/browse/HIVE-5795 > > > Repository: hive-git > > > Description > ------- > > Hive should be able to skip header and footer rows when reading data file for > a table > > (I am uploading this on behalf of Shuaishuai Nie since he's not in the office) > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 32ab3d8 > data/files/header_footer_table_1/0001.txt PRE-CREATION > data/files/header_footer_table_1/0002.txt PRE-CREATION > data/files/header_footer_table_1/0003.txt PRE-CREATION > data/files/header_footer_table_2/2012/01/01/0001.txt PRE-CREATION > data/files/header_footer_table_2/2012/01/02/0002.txt PRE-CREATION > data/files/header_footer_table_2/2012/01/03/0003.txt PRE-CREATION > itests/qtest/pom.xml a453d8a > ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 5abcfc1 > ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java > dd5cb6b > ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 0ec6e63 > > ql/src/test/org/apache/hadoop/hive/ql/io/TestHiveBinarySearchRecordReader.java > 85dd975 > ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java > 0686d9b > ql/src/test/queries/clientpositive/file_with_header_footer.q PRE-CREATION > ql/src/test/results/clientpositive/file_with_header_footer.q.out > PRE-CREATION > > Diff: https://reviews.apache.org/r/15663/diff/ > > > Testing > ------- > > > Thanks, > > Eric Hanson > >
