[jira] [Commented] (HIVE-12990) LLAP: ORC cache NPE without FileID support
[ https://issues.apache.org/jira/browse/HIVE-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134687#comment-15134687 ] Prasanth Jayachandran commented on HIVE-12990: -- LGTM, +1 > LLAP: ORC cache NPE without FileID support > -- > > Key: HIVE-12990 > URL: https://issues.apache.org/jira/browse/HIVE-12990 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Sergey Shelukhin > Attachments: HIVE-12990.01.patch, HIVE-12990.patch > > > {code} >OrcBatchKey stripeKey = hasFileId ? new OrcBatchKey(fileId, -1, 0) : null; >... > if (hasFileId && metadataCache != null) { > stripeKey.stripeIx = stripeIx; > stripeMetadata = metadataCache.getStripeMetadata(stripeKey); > } > ... > public void setStripeMetadata(OrcStripeMetadata m) { > assert stripes != null; > stripes[m.getStripeIx()] = m; > } > {code} > {code} > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.llap.io.metadata.OrcStripeMetadata.getStripeIx(OrcStripeMetadata.java:106) > at > org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.setStripeMetadata(OrcEncodedDataConsumer.java:70) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.readStripesMetadata(OrcEncodedDataReader.java:685) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:283) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:215) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:212) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:93) > ... 5 more > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12990) LLAP: ORC cache NPE without FileID support
[ https://issues.apache.org/jira/browse/HIVE-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133919#comment-15133919 ] Hive QA commented on HIVE-12990: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12786316/HIVE-12990.01.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10051 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hive.jdbc.TestSSL.testSSLVersion {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6871/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6871/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6871/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12786316 - PreCommit-HIVE-TRUNK-Build > LLAP: ORC cache NPE without FileID support > -- > > Key: HIVE-12990 > URL: https://issues.apache.org/jira/browse/HIVE-12990 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Sergey Shelukhin > Attachments: HIVE-12990.01.patch, HIVE-12990.patch > > > {code} >OrcBatchKey stripeKey = hasFileId ? new OrcBatchKey(fileId, -1, 0) : null; >... > if (hasFileId && metadataCache != null) { > stripeKey.stripeIx = stripeIx; > stripeMetadata = metadataCache.getStripeMetadata(stripeKey); > } > ... > public void setStripeMetadata(OrcStripeMetadata m) { > assert stripes != null; > stripes[m.getStripeIx()] = m; > } > {code} > {code} > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.llap.io.metadata.OrcStripeMetadata.getStripeIx(OrcStripeMetadata.java:106) > at > org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.setStripeMetadata(OrcEncodedDataConsumer.java:70) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.readStripesMetadata(OrcEncodedDataReader.java:685) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:283) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:215) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:212) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:93) > ... 5 more > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12990) LLAP: ORC cache NPE without FileID support
[ https://issues.apache.org/jira/browse/HIVE-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15131722#comment-15131722 ] Gopal V commented on HIVE-12990: Finally figure out what's happening there. The CLI parameters set aren't passed to the IO Elevator, so the params have to be setup in llap-daemon-site.xml for the system to pick it up. Which brings a very different problem, FS configs (like auth tokens) are passed via the jobconf today. > LLAP: ORC cache NPE without FileID support > -- > > Key: HIVE-12990 > URL: https://issues.apache.org/jira/browse/HIVE-12990 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Sergey Shelukhin > Attachments: HIVE-12990.patch > > > {code} >OrcBatchKey stripeKey = hasFileId ? new OrcBatchKey(fileId, -1, 0) : null; >... > if (hasFileId && metadataCache != null) { > stripeKey.stripeIx = stripeIx; > stripeMetadata = metadataCache.getStripeMetadata(stripeKey); > } > ... > public void setStripeMetadata(OrcStripeMetadata m) { > assert stripes != null; > stripes[m.getStripeIx()] = m; > } > {code} > {code} > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.llap.io.metadata.OrcStripeMetadata.getStripeIx(OrcStripeMetadata.java:106) > at > org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.setStripeMetadata(OrcEncodedDataConsumer.java:70) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.readStripesMetadata(OrcEncodedDataReader.java:685) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:283) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:215) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:212) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:93) > ... 5 more > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12990) LLAP: ORC cache NPE without FileID support
[ https://issues.apache.org/jira/browse/HIVE-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15131662#comment-15131662 ] Gopal V commented on HIVE-12990: Looks like this still sends through a NULL fileId through - the auto-unbox here throws an NPE {code} @Override public EncodedReader encodedReader( Long fileId, DataCache dataCache, DataReader dataReader, PoolFactory pf) throws IOException { return new EncodedReaderImpl(fileId, types, codec, bufferSize, rowIndexStride, dataCache, dataReader, pf); } {code} {code} Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.io.orc.encoded.ReaderImpl.encodedReader(ReaderImpl.java:38) at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:331) at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:215) at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:212) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:212) at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:93) {code} > LLAP: ORC cache NPE without FileID support > -- > > Key: HIVE-12990 > URL: https://issues.apache.org/jira/browse/HIVE-12990 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Sergey Shelukhin > Attachments: HIVE-12990.patch > > > {code} >OrcBatchKey stripeKey = hasFileId ? new OrcBatchKey(fileId, -1, 0) : null; >... > if (hasFileId && metadataCache != null) { > stripeKey.stripeIx = stripeIx; > stripeMetadata = metadataCache.getStripeMetadata(stripeKey); > } > ... > public void setStripeMetadata(OrcStripeMetadata m) { > assert stripes != null; > stripes[m.getStripeIx()] = m; > } > {code} > {code} > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.llap.io.metadata.OrcStripeMetadata.getStripeIx(OrcStripeMetadata.java:106) > at > org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.setStripeMetadata(OrcEncodedDataConsumer.java:70) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.readStripesMetadata(OrcEncodedDataReader.java:685) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:283) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:215) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:212) > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:93) > ... 5 more > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)