[ https://issues.apache.org/jira/browse/DRILL-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15153147#comment-15153147 ]
ASF GitHub Bot commented on DRILL-3688: --------------------------------------- GitHub user arina-ielchiieva opened a pull request: https://github.com/apache/drill/pull/382 DRILL-3688: Drill should honor "skip.header.line.count" and "skip.foo… Drill should honor "skip.header.line.count" and "skip.footer.line.count" attribute of Hive table: 1. Functionality to skip header and footer lines while reading Hive data. 2. Unit tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/arina-ielchiieva/drill DRILL-3688 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/382.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #382 ---- commit 0482cea2c771b167907e32258fb649979cddcf49 Author: Arina Ielchiieva <arina.yelchiy...@gmail.com> Date: 2016-02-11T17:16:30Z DRILL-3688: Drill should honor "skip.header.line.count" and "skip.footer.line.count" attribute of Hive table 1. Functionality to skip header and footer lines while reading Hive data. 2. Unit tests. ---- > Drill should honor "skip.header.line.count" and "skip.footer.line.count" > attributes of Hive table > ------------------------------------------------------------------------------------------------- > > Key: DRILL-3688 > URL: https://issues.apache.org/jira/browse/DRILL-3688 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Hive > Affects Versions: 1.1.0 > Environment: 1.1 > Reporter: Hao Zhu > Assignee: Arina Ielchiieva > Fix For: Future > > > Currently Drill does not honor the "skip.header.line.count" attribute of Hive > table. > It may cause some other format conversion issue. > Reproduce: > 1. Create a Hive table > {code} > create table h1db.testheader(col0 string) > ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' > STORED AS TEXTFILE > tblproperties("skip.header.line.count"="1"); > {code} > 2. Prepare a sample data: > {code} > # cat test.data > col0 > 2015-01-01 > {code} > 3. Load sample data into Hive > {code} > LOAD DATA LOCAL INPATH '/xxx/test.data' OVERWRITE INTO TABLE h1db.testheader; > {code} > 4. Hive > {code} > hive> select * from h1db.testheader ; > OK > 2015-01-01 > Time taken: 0.254 seconds, Fetched: 1 row(s) > {code} > 5. Drill > {code} > > select * from hive.h1db.testheader ; > +-------------+ > | col0 | > +-------------+ > | col0 | > | 2015-01-01 | > +-------------+ > 2 rows selected (0.257 seconds) > > select cast(col0 as date) from hive.h1db.testheader ; > Error: SYSTEM ERROR: IllegalFieldValueException: Value 0 for monthOfYear must > be in the range [1,12] > Fragment 0:0 > [Error Id: 34353702-ca27-440b-a4f4-0c9f79fc8ccd on h1.poc.com:31010] > (org.joda.time.IllegalFieldValueException) Value 0 for monthOfYear must be > in the range [1,12] > org.joda.time.field.FieldUtils.verifyValueBounds():236 > org.joda.time.chrono.BasicChronology.getDateMidnightMillis():613 > org.joda.time.chrono.BasicChronology.getDateTimeMillis():159 > org.joda.time.chrono.AssembledChronology.getDateTimeMillis():120 > org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.memGetDate():261 > org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getDate():218 > org.apache.drill.exec.test.generated.ProjectorGen0.doEval():67 > org.apache.drill.exec.test.generated.ProjectorGen0.projectRecords():62 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork():172 > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():93 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129 > org.apache.drill.exec.record.AbstractRecordBatch.next():147 > org.apache.drill.exec.physical.impl.BaseRootExec.next():83 > > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():79 > org.apache.drill.exec.physical.impl.BaseRootExec.next():73 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():261 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():255 > java.security.AccessController.doPrivileged():-2 > javax.security.auth.Subject.doAs():422 > org.apache.hadoop.security.UserGroupInformation.doAs():1566 > org.apache.drill.exec.work.fragment.FragmentExecutor.run():255 > org.apache.drill.common.SelfCleaningRunnable.run():38 > java.util.concurrent.ThreadPoolExecutor.runWorker():1142 > java.util.concurrent.ThreadPoolExecutor$Worker.run():617 > java.lang.Thread.run():745 (state=,code=0) > {code} > Also "skip.footer.line.count" should be taken into account. > If "skip.header.line.count" or "skip.footer.line.count" has incorrect value > in Hive, throw appropriate exception in Drill. > Ex: Hive table property skip.header.line.count value 'someValue' is > non-numeric -- This message was sent by Atlassian JIRA (v6.3.4#6332)