[jira] [Commented] (HIVE-13884) Disallow queries in HMS fetching more than a configured number of partitions
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370264#comment-15370264
]
Lefty Leverenz commented on HIVE-13884:
---
Okay, found it on the dev@hive list. You have edit privileges now.
> Disallow queries in HMS fetching more than a configured number of partitions
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-13884.1.patch, HIVE-13884.10.patch,
> HIVE-13884.2.patch, HIVE-13884.3.patch, HIVE-13884.4.patch,
> HIVE-13884.5.patch, HIVE-13884.6.patch, HIVE-13884.7.patch,
> HIVE-13884.8.patch, HIVE-13884.9.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[
https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370259#comment-15370259
]
Hive QA commented on HIVE-13974:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12817072/HIVE-13974.091.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10296 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/462/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/462/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-462/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12817072 - PreCommit-HIVE-MASTER-Build
> ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
> ---
>
> Key: HIVE-13974
> URL: https://issues.apache.org/jira/browse/HIVE-13974
> Project: Hive
> Issue Type: Bug
> Components: Hive, ORC, Transactions
>Affects Versions: 1.3.0, 2.1.0, 2.2.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Blocker
> Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch,
> HIVE-13974.03.patch, HIVE-13974.04.patch, HIVE-13974.05.WIP.patch,
> HIVE-13974.06.patch, HIVE-13974.07.patch, HIVE-13974.08.patch,
> HIVE-13974.09.patch, HIVE-13974.091.patch
>
>
> Currently, the included columns are based on the fileSchema and not the
> readerSchema which doesn't work for adding columns to non-last STRUCT data
> type columns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13644) Remove hardcoded groovy.grape.report.downloads=true from DependencyResolver
[
https://issues.apache.org/jira/browse/HIVE-13644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370258#comment-15370258
]
Lefty Leverenz commented on HIVE-13644:
---
Should this be documented in the wiki? (If so, where?)
> Remove hardcoded groovy.grape.report.downloads=true from DependencyResolver
> ---
>
> Key: HIVE-13644
> URL: https://issues.apache.org/jira/browse/HIVE-13644
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Fix For: 2.2.0
>
> Attachments: HIVE-13644.1.patch
>
>
> Currently, in Hive's
> [DependencyResolver.java|https://github.com/apache/hive/blob/8dd1d1966f2f0b86604b4e991ebc865224f42b41/ql/src/java/org/apache/hadoop/hive/ql/util/DependencyResolver.java#L176],
> the system property {{groovy.grape.report.downloads}} is hardcoded to
> {{true}} and there is no way to override it and disable the logging. We
> should remove this hardcoded value and allow users to configure it as they
> see fit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14158) deal with derived column names
[
https://issues.apache.org/jira/browse/HIVE-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370254#comment-15370254
]
Pengcheng Xiong commented on HIVE-14158:
[~ashutoshc] and [~hagleitn]. The previous patch 03 does not work for (1)
enforced view column names and (2) existing view column authorization
mechanism. Thus I worked out patch 04. It will solve both the view auto
generated column alias problem and the virtual column problem. The design is
that (1) give a different auto generated column alias. (to solve the view auto
generated column alias problem) and (2) add the virtual columns for masking
tables and unparse translate the "*". It seems that the price is that we have
to go through genOp twice, i.e., the first time, unparse the "select *", the
second time, add all the virtual columns. See the below query
{code}
select * from maskTable
{code}
If we only go through genOp once, then it will be rewritten to
{code}
select * from (select key, mask(value) as value, ROW__ID... etc )maskTable
{code}
It will change the RR of the maskTable to the outside.
> deal with derived column names
> --
>
> Key: HIVE-14158
> URL: https://issues.apache.org/jira/browse/HIVE-14158
> Project: Hive
> Issue Type: Sub-task
> Components: Security
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.1.0
>
> Attachments: HIVE-14158.01.patch, HIVE-14158.02.patch,
> HIVE-14158.03.patch, HIVE-14158.04.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14158) deal with derived column names
[ https://issues.apache.org/jira/browse/HIVE-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14158: --- Status: Open (was: Patch Available) > deal with derived column names > -- > > Key: HIVE-14158 > URL: https://issues.apache.org/jira/browse/HIVE-14158 > Project: Hive > Issue Type: Sub-task > Components: Security >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-14158.01.patch, HIVE-14158.02.patch, > HIVE-14158.03.patch, HIVE-14158.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14158) deal with derived column names
[ https://issues.apache.org/jira/browse/HIVE-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14158: --- Attachment: HIVE-14158.04.patch > deal with derived column names > -- > > Key: HIVE-14158 > URL: https://issues.apache.org/jira/browse/HIVE-14158 > Project: Hive > Issue Type: Sub-task > Components: Security >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-14158.01.patch, HIVE-14158.02.patch, > HIVE-14158.03.patch, HIVE-14158.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14158) deal with derived column names
[ https://issues.apache.org/jira/browse/HIVE-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14158: --- Status: Patch Available (was: Open) > deal with derived column names > -- > > Key: HIVE-14158 > URL: https://issues.apache.org/jira/browse/HIVE-14158 > Project: Hive > Issue Type: Sub-task > Components: Security >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-14158.01.patch, HIVE-14158.02.patch, > HIVE-14158.03.patch, HIVE-14158.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14139) NPE dropping permanent function
[
https://issues.apache.org/jira/browse/HIVE-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-14139:
--
Attachment: HIVE-14139.4.patch
Patch v4 adds safe check to avoid the NPE.
[~sershe], could you take a look and let me know if it reflects what you have
in mind? Thanks.
> NPE dropping permanent function
> ---
>
> Key: HIVE-14139
> URL: https://issues.apache.org/jira/browse/HIVE-14139
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14139.1.patch, HIVE-14139.2.patch,
> HIVE-14139.3.patch, HIVE-14139.4.patch
>
>
> To reproduce:
> 1. Start a CLI session and create a permanent function.
> 2. Exit current CLI session.
> 3. Start a new CLI session and drop the function.
> Stack trace:
> {noformat}
> FAILED: error during drop function: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.Registry.removePersistentFunctionUnderLock(Registry.java:513)
> at
> org.apache.hadoop.hive.ql.exec.Registry.unregisterFunction(Registry.java:501)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.unregisterPermanentFunction(FunctionRegistry.java:1532)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.dropPermanentFunction(FunctionTask.java:228)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:95)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1860)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1564)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1316)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1085)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1073)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14152) datanucleus.autoStartMechanismMode should set to 'Ignored' to allow rolling downgrade
[
https://issues.apache.org/jira/browse/HIVE-14152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-14152:
-
Attachment: HIVE-14152.3.patch
3.patch - fix test case
> datanucleus.autoStartMechanismMode should set to 'Ignored' to allow rolling
> downgrade
> --
>
> Key: HIVE-14152
> URL: https://issues.apache.org/jira/browse/HIVE-14152
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Daniel Dai
>Assignee: Thejas M Nair
> Attachments: HIVE-14152.1.patch, HIVE-14152.2.patch,
> HIVE-14152.3.patch
>
>
> We see the following issue when downgrading metastore:
> 1. Run some query using new tables
> 2. Downgrade metastore
> 3. Restart metastore will complain the new table does not exist
> In particular, constaints tables does not exist in branch-1. If we run Hive 2
> and create a constraint, then downgrade metastore to Hive 1, datanucleus will
> complain:
> {code}
> javax.jdo.JDOFatalUserException: Error starting up DataNucleus : a class
> "org.apache.hadoop.hive.metastore.model.MConstraint" was listed as being
> persisted previously in this datastore, yet the class wasnt found. Perhaps it
> is used by a different DataNucleus-enabled application in this datastore, or
> you have changed your class names.
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:528)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
> at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:377)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:406)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:299)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:266)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:60)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:69)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:650)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:628)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:677)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:484)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:77)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:83)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5905)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5900)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6159)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6084)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar
[jira] [Commented] (HIVE-14203) ORC should throw if null row is passed to writer
[ https://issues.apache.org/jira/browse/HIVE-14203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370204#comment-15370204 ] Prasanth Jayachandran commented on HIVE-14203: -- Couple of things - Can you add a msg to the the exception saying that it is not expected (per writer contract) to write null rows? or something like that.. - Can you please update the Writer.java interface that the row should be non-null - Vectorized writers use addRowBatch() API. We might have to do the same for addRowBatch() API as well along with above steps > ORC should throw if null row is passed to writer > > > Key: HIVE-14203 > URL: https://issues.apache.org/jira/browse/HIVE-14203 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1, 2.0.0, 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14203.patch > > > Currently it silently drops the row (doesn't write in file) but updates > metadata. As a result of this data being inconsistent with metadata, its > impossible to read it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13614) Implementation for PreparedStatement#setBigDecimal
[ https://issues.apache.org/jira/browse/HIVE-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13614: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to mater. Thanks Iwao! > Implementation for PreparedStatement#setBigDecimal > -- > > Key: HIVE-13614 > URL: https://issues.apache.org/jira/browse/HIVE-13614 > Project: Hive > Issue Type: Improvement > Components: JDBC >Reporter: Iwao AVE! >Assignee: Iwao AVE! >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-13614.patch > > > Hi, > I am a committer of MyBatis project (https://github.com/mybatis) and some of > our users have difficulty with using MyBatis-Migrations (a db migration tool) > with Hive. > https://github.com/mybatis/migrations/issues/25 > Basically, Migrations uses BigDecimal as the ID type and the lack of support > in Hive's JDBC driver prevents our users from using Migrations with Hive. > So, it would be beneficial to both of Hive and MyBatis users if you could > apply the following change to mitigate the situation. > https://github.com/harawata/hive/commit/1910632442ad9b70b41b28e37596843d1b9f7d3f > Please let me know if I should send the modification as a GitHub pull request. > Thanks in advance! > Iwao -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13625) Hive Prepared Statement when executed with escape characters in parameter fails
[
https://issues.apache.org/jira/browse/HIVE-13625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-13625:
Component/s: JDBC
> Hive Prepared Statement when executed with escape characters in parameter
> fails
> ---
>
> Key: HIVE-13625
> URL: https://issues.apache.org/jira/browse/HIVE-13625
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: Daya Venkatesan
>Assignee: Zoltan Haindrich
> Fix For: 2.2.0
>
> Attachments: HIVE-13625.patch
>
>
> When setting parameters to a Hive Prepared Statement, if the parameter has
> an odd number of escape characters, then the Statement fails.
> For example, I set one of the parameters to
> "/somepath/\044{}/\044{MM}/\044{dd}/". Here, I have escaped the dollar
> character with \044 because Hive gives an Atlas exception with "$" character.
> Now, when the parameters are getting set inside Hive,
> getCharIndexFromSqlByParamLocation throws an Exception.
> Hive records something called signal count.
> if (c == '\'' || c == '\\')// record the count of char "'" and char "\"
> {
> signalCount++;
> }
> And the parameter is set only if the signalCount %2 is 0.
> else if (c == cchar && signalCount % 2 == 0) {// check if the ? is really the
> parameter
> num++;
> if (num == paramLoc) {
> charIndex = i;
> break;
> }
> Since my parameter has three "\" characters, the signal Count modulo is not 0
> and the parameter is not set at all throwing an exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13625) Hive Prepared Statement when executed with escape characters in parameter fails
[
https://issues.apache.org/jira/browse/HIVE-13625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-13625:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Zoltan!
> Hive Prepared Statement when executed with escape characters in parameter
> fails
> ---
>
> Key: HIVE-13625
> URL: https://issues.apache.org/jira/browse/HIVE-13625
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: Daya Venkatesan
>Assignee: Zoltan Haindrich
> Fix For: 2.2.0
>
> Attachments: HIVE-13625.patch
>
>
> When setting parameters to a Hive Prepared Statement, if the parameter has
> an odd number of escape characters, then the Statement fails.
> For example, I set one of the parameters to
> "/somepath/\044{}/\044{MM}/\044{dd}/". Here, I have escaped the dollar
> character with \044 because Hive gives an Atlas exception with "$" character.
> Now, when the parameters are getting set inside Hive,
> getCharIndexFromSqlByParamLocation throws an Exception.
> Hive records something called signal count.
> if (c == '\'' || c == '\\')// record the count of char "'" and char "\"
> {
> signalCount++;
> }
> And the parameter is set only if the signalCount %2 is 0.
> else if (c == cchar && signalCount % 2 == 0) {// check if the ? is really the
> parameter
> num++;
> if (num == paramLoc) {
> charIndex = i;
> break;
> }
> Since my parameter has three "\" characters, the signal Count modulo is not 0
> and the parameter is not set at all throwing an exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13644) Remove hardcoded groovy.grape.report.downloads=true from DependencyResolver
[
https://issues.apache.org/jira/browse/HIVE-13644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-13644:
Component/s: UDF
> Remove hardcoded groovy.grape.report.downloads=true from DependencyResolver
> ---
>
> Key: HIVE-13644
> URL: https://issues.apache.org/jira/browse/HIVE-13644
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Fix For: 2.2.0
>
> Attachments: HIVE-13644.1.patch
>
>
> Currently, in Hive's
> [DependencyResolver.java|https://github.com/apache/hive/blob/8dd1d1966f2f0b86604b4e991ebc865224f42b41/ql/src/java/org/apache/hadoop/hive/ql/util/DependencyResolver.java#L176],
> the system property {{groovy.grape.report.downloads}} is hardcoded to
> {{true}} and there is no way to override it and disable the logging. We
> should remove this hardcoded value and allow users to configure it as they
> see fit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13644) Remove hardcoded groovy.grape.report.downloads=true from DependencyResolver
[
https://issues.apache.org/jira/browse/HIVE-13644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-13644:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Anthony!
> Remove hardcoded groovy.grape.report.downloads=true from DependencyResolver
> ---
>
> Key: HIVE-13644
> URL: https://issues.apache.org/jira/browse/HIVE-13644
> Project: Hive
> Issue Type: Improvement
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Fix For: 2.2.0
>
> Attachments: HIVE-13644.1.patch
>
>
> Currently, in Hive's
> [DependencyResolver.java|https://github.com/apache/hive/blob/8dd1d1966f2f0b86604b4e991ebc865224f42b41/ql/src/java/org/apache/hadoop/hive/ql/util/DependencyResolver.java#L176],
> the system property {{groovy.grape.report.downloads}} is hardcoded to
> {{true}} and there is no way to override it and disable the logging. We
> should remove this hardcoded value and allow users to configure it as they
> see fit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14152) datanucleus.autoStartMechanismMode should set to 'Ignored' to allow rolling downgrade
[
https://issues.apache.org/jira/browse/HIVE-14152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370164#comment-15370164
]
Hive QA commented on HIVE-14152:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12817053/HIVE-14152.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10282 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-auto_sortmerge_join_7.q-cbo_windowing.q-vector_interval_2.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.metastore.TestMetastoreVersion.testMetastoreVersion
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/461/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/461/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-461/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12817053 - PreCommit-HIVE-MASTER-Build
> datanucleus.autoStartMechanismMode should set to 'Ignored' to allow rolling
> downgrade
> --
>
> Key: HIVE-14152
> URL: https://issues.apache.org/jira/browse/HIVE-14152
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Daniel Dai
>Assignee: Thejas M Nair
> Attachments: HIVE-14152.1.patch, HIVE-14152.2.patch
>
>
> We see the following issue when downgrading metastore:
> 1. Run some query using new tables
> 2. Downgrade metastore
> 3. Restart metastore will complain the new table does not exist
> In particular, constaints tables does not exist in branch-1. If we run Hive 2
> and create a constraint, then downgrade metastore to Hive 1, datanucleus will
> complain:
> {code}
> javax.jdo.JDOFatalUserException: Error starting up DataNucleus : a class
> "org.apache.hadoop.hive.metastore.model.MConstraint" was listed as being
> persisted previously in this datastore, yet the class wasnt found. Perhaps it
> is used by a different DataNucleus-enabled application in this datastore, or
> you have changed your class names.
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:528)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
> at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:377)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:406)
> at
> org.apache.hadoop.hive.metastore.ObjectStore
[jira] [Updated] (HIVE-14203) ORC should throw if null row is passed to writer
[ https://issues.apache.org/jira/browse/HIVE-14203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14203: Status: Patch Available (was: Open) > ORC should throw if null row is passed to writer > > > Key: HIVE-14203 > URL: https://issues.apache.org/jira/browse/HIVE-14203 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0, 2.0.0, 1.2.1 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14203.patch > > > Currently it silently drops the row (doesn't write in file) but updates > metadata. As a result of this data being inconsistent with metadata, its > impossible to read it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Attachment: HIVE-13974.091.patch > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch, > HIVE-13974.03.patch, HIVE-13974.04.patch, HIVE-13974.05.WIP.patch, > HIVE-13974.06.patch, HIVE-13974.07.patch, HIVE-13974.08.patch, > HIVE-13974.09.patch, HIVE-13974.091.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Attachment: (was: HIVE-13974.091.patch) > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch, > HIVE-13974.03.patch, HIVE-13974.04.patch, HIVE-13974.05.WIP.patch, > HIVE-13974.06.patch, HIVE-13974.07.patch, HIVE-13974.08.patch, > HIVE-13974.09.patch, HIVE-13974.091.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Status: Patch Available (was: In Progress) > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 2.1.0, 1.3.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch, > HIVE-13974.03.patch, HIVE-13974.04.patch, HIVE-13974.05.WIP.patch, > HIVE-13974.06.patch, HIVE-13974.07.patch, HIVE-13974.08.patch, > HIVE-13974.09.patch, HIVE-13974.091.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370110#comment-15370110 ] Matt McCline commented on HIVE-13974: - Patch #091 includes HIVE-14004 patch #3. > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch, > HIVE-13974.03.patch, HIVE-13974.04.patch, HIVE-13974.05.WIP.patch, > HIVE-13974.06.patch, HIVE-13974.07.patch, HIVE-13974.08.patch, > HIVE-13974.09.patch, HIVE-13974.091.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Attachment: HIVE-13974.091.patch > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch, > HIVE-13974.03.patch, HIVE-13974.04.patch, HIVE-13974.05.WIP.patch, > HIVE-13974.06.patch, HIVE-13974.07.patch, HIVE-13974.08.patch, > HIVE-13974.09.patch, HIVE-13974.091.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13548) hive-jdbc isn't escaping slashes during PreparedStatement
[
https://issues.apache.org/jira/browse/HIVE-13548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370109#comment-15370109
]
Hive QA commented on HIVE-13548:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12799539/HIVE-13548.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10298 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/460/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/460/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-460/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12799539 - PreCommit-HIVE-MASTER-Build
> hive-jdbc isn't escaping slashes during PreparedStatement
> -
>
> Key: HIVE-13548
> URL: https://issues.apache.org/jira/browse/HIVE-13548
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: Nasron Cheong
>Assignee: Nasron Cheong
> Attachments: HIVE-13548.patch
>
>
> Calling setString on a prepared statement with a string containing a '\' will
> cause the SQL construction to fail.
> I believe the slash should be escaped by the setString function.
> There may be other characters that require escaping during the same call.
> Failure from the unittest without the patch:
> {code}
> Running org.apache.hive.jdbc.TestJdbcDriver2
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 9.738 sec <<<
> FAILURE! - in org.apache.hive.jdbc.TestJdbcDriver2
> testSlashPreparedStatement(org.apache.hive.jdbc.TestJdbcDriver2) Time
> elapsed: 3.867 sec <<< FAILURE!
> java.lang.AssertionError: java.lang.StringIndexOutOfBoundsException: String
> index out of range: -1
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hive.jdbc.TestJdbcDriver2.testSlashPreparedStatement(TestJdbcDriver2.java:522)
> Results :
> Failed tests:
> TestJdbcDriver2.testSlashPreparedStatement:522
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Status: In Progress (was: Patch Available) > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 2.1.0, 1.3.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch, > HIVE-13974.03.patch, HIVE-13974.04.patch, HIVE-13974.05.WIP.patch, > HIVE-13974.06.patch, HIVE-13974.07.patch, HIVE-13974.08.patch, > HIVE-13974.09.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13644) Remove hardcoded groovy.grape.report.downloads=true from DependencyResolver
[
https://issues.apache.org/jira/browse/HIVE-13644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370043#comment-15370043
]
Hive QA commented on HIVE-13644:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801275/HIVE-13644.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10282 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-tez_union_group_by.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-vector_left_outer_join2.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/459/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/459/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-459/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12801275 - PreCommit-HIVE-MASTER-Build
> Remove hardcoded groovy.grape.report.downloads=true from DependencyResolver
> ---
>
> Key: HIVE-13644
> URL: https://issues.apache.org/jira/browse/HIVE-13644
> Project: Hive
> Issue Type: Improvement
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13644.1.patch
>
>
> Currently, in Hive's
> [DependencyResolver.java|https://github.com/apache/hive/blob/8dd1d1966f2f0b86604b4e991ebc865224f42b41/ql/src/java/org/apache/hadoop/hive/ql/util/DependencyResolver.java#L176],
> the system property {{groovy.grape.report.downloads}} is hardcoded to
> {{true}} and there is no way to override it and disable the logging. We
> should remove this hardcoded value and allow users to configure it as they
> see fit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14004) Minor compaction produces ArrayIndexOutOfBoundsException: 7 in SchemaEvolution.getFileType
[
https://issues.apache.org/jira/browse/HIVE-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370003#comment-15370003
]
Matt McCline commented on HIVE-14004:
-
Eugene: thank you for looking at the changes.
> Minor compaction produces ArrayIndexOutOfBoundsException: 7 in
> SchemaEvolution.getFileType
> --
>
> Key: HIVE-14004
> URL: https://issues.apache.org/jira/browse/HIVE-14004
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Eugene Koifman
>Assignee: Matt McCline
> Attachments: HIVE-14004.01.patch, HIVE-14004.02.patch,
> HIVE-14004.03.patch
>
>
> Easiest way to repro is to add TestTxnCommands2
> {noformat}
> @Test
> public void testCompactWithDelete() throws Exception {
> int[][] tableData = {{1,2},{3,4}};
> runStatementOnDriver("insert into " + Table.ACIDTBL + "(a,b) " +
> makeValuesClause(tableData));
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MAJOR'");
> Worker t = new Worker();
> t.setThreadId((int) t.getId());
> t.setHiveConf(hiveConf);
> AtomicBoolean stop = new AtomicBoolean();
> AtomicBoolean looped = new AtomicBoolean();
> stop.set(true);
> t.init(stop, looped);
> t.run();
> runStatementOnDriver("delete from " + Table.ACIDTBL + " where b = 4");
> runStatementOnDriver("update " + Table.ACIDTBL + " set b = -2 where b =
> 2");
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MINOR'");
> t.run();
> }
> {noformat}
> to TestTxnCommands2 and run it.
> Test won't fail but if you look
> in target/tmp/log/hive.log for the following exception (from Minor
> compaction).
> {noformat}
> 2016-06-09T18:36:39,071 WARN [Thread-190[]]: mapred.LocalJobRunner
> (LocalJobRunner.java:run(560)) - job_local1233973168_0005
> java.lang.Exception: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.6.1.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.6.1.jar:?]
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.orc.impl.SchemaEvolution.getFileType(SchemaEvolution.java:67)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2031)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.RecordReaderImpl.(RecordReaderImpl.java:208)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:63)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:365)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:207)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:508)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1977)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:630)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:609)
> ~[classes/:?]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.6.1.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
> ~[hadoop-mapreduce-client-core-2.6.1.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.6.1.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.6.1.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_71]
> at java.util.concurrent.FutureTask.run(FutureTas
[jira] [Commented] (HIVE-14129) Execute move tasks in parallel
[
https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369993#comment-15369993
]
Hive QA commented on HIVE-14129:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12817051/HIVE-14129.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10297 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/458/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/458/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-458/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12817051 - PreCommit-HIVE-MASTER-Build
> Execute move tasks in parallel
> --
>
> Key: HIVE-14129
> URL: https://issues.apache.org/jira/browse/HIVE-14129
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-14129.2.patch, HIVE-14129.patch, HIVE-14129.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13884) Disallow queries in HMS fetching more than a configured number of partitions
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369968#comment-15369968
]
Sergio Peña commented on HIVE-13884:
[~leftylev] Yes, thanks.
I sent an email to user@ 2 days ago asking for wiki access, but I just found
out that email was not sent.
Just resent again.
> Disallow queries in HMS fetching more than a configured number of partitions
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-13884.1.patch, HIVE-13884.10.patch,
> HIVE-13884.2.patch, HIVE-13884.3.patch, HIVE-13884.4.patch,
> HIVE-13884.5.patch, HIVE-13884.6.patch, HIVE-13884.7.patch,
> HIVE-13884.8.patch, HIVE-13884.9.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14004) Minor compaction produces ArrayIndexOutOfBoundsException: 7 in SchemaEvolution.getFileType
[
https://issues.apache.org/jira/browse/HIVE-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369956#comment-15369956
]
Eugene Koifman edited comment on HIVE-14004 at 7/10/16 9:24 PM:
[~mmccline] I left a few comments on RB. Mostly superficial except
"clone()/toString()" wrt Reader.Options.
I understand what you are trying to do and generally it makes sense.
One concern I have is that that the fact that orc.RecordReader need to know
that it's doing Acid read vs regular read. Given how things are currently
implemented, I'm not sure how to avoid that. It would be better if higher
layer just specified what columns it wants - user cols and acid meta cols and
interpret them so that RecordReader doesn't have to. Perhaps down the road we
can make the layout be just 1 struct where "c" are user columns and include
some version number in the ORC footer to know where the offset for the user
columns start (in case we add more metadata columns). I think this may have
other advantages make acid look more like non-acid equivalent.
I think someone more familiar with the code patch should look at this as well.
was (Author: ekoifman):
[~mmccline] I left a few comments on RB. Mostly superficial except
"clone()/toString()" wrt Reader.Options.
I understand what you are trying to do and generally it makes sense.
One concern I have is that that the fact that orc.RecordReader need to know
that it's doing Acid read vs regular read. Given how things are currently
implemented, I'm not sure how to avoid that. It would be better if higher
layer just specified what columns it wants - user cols and acid meta cols and
interpret them so that RecordReader doesn't have to. Perhaps down the road we
can make the layout be just 1 struct where "c" are user columns and include
some version number in the ORC footer to know where the offset for the user
columns start (in case we add more metadata columns). I think this may have
other advantages.
I think someone more familiar with the code patch should look at this as well.
> Minor compaction produces ArrayIndexOutOfBoundsException: 7 in
> SchemaEvolution.getFileType
> --
>
> Key: HIVE-14004
> URL: https://issues.apache.org/jira/browse/HIVE-14004
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Eugene Koifman
>Assignee: Matt McCline
> Attachments: HIVE-14004.01.patch, HIVE-14004.02.patch,
> HIVE-14004.03.patch
>
>
> Easiest way to repro is to add TestTxnCommands2
> {noformat}
> @Test
> public void testCompactWithDelete() throws Exception {
> int[][] tableData = {{1,2},{3,4}};
> runStatementOnDriver("insert into " + Table.ACIDTBL + "(a,b) " +
> makeValuesClause(tableData));
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MAJOR'");
> Worker t = new Worker();
> t.setThreadId((int) t.getId());
> t.setHiveConf(hiveConf);
> AtomicBoolean stop = new AtomicBoolean();
> AtomicBoolean looped = new AtomicBoolean();
> stop.set(true);
> t.init(stop, looped);
> t.run();
> runStatementOnDriver("delete from " + Table.ACIDTBL + " where b = 4");
> runStatementOnDriver("update " + Table.ACIDTBL + " set b = -2 where b =
> 2");
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MINOR'");
> t.run();
> }
> {noformat}
> to TestTxnCommands2 and run it.
> Test won't fail but if you look
> in target/tmp/log/hive.log for the following exception (from Minor
> compaction).
> {noformat}
> 2016-06-09T18:36:39,071 WARN [Thread-190[]]: mapred.LocalJobRunner
> (LocalJobRunner.java:run(560)) - job_local1233973168_0005
> java.lang.Exception: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.6.1.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.6.1.jar:?]
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.orc.impl.SchemaEvolution.getFileType(SchemaEvolution.java:67)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2031)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$St
[jira] [Commented] (HIVE-14004) Minor compaction produces ArrayIndexOutOfBoundsException: 7 in SchemaEvolution.getFileType
[
https://issues.apache.org/jira/browse/HIVE-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369956#comment-15369956
]
Eugene Koifman commented on HIVE-14004:
---
[~mmccline] I left a few comments on RB. Mostly superficial except
"clone()/toString()" wrt Reader.Options.
I understand what you are trying to do and generally it makes sense.
One concern I have is that that the fact that orc.RecordReader need to know
that it's doing Acid read vs regular read. Given how things are currently
implemented, I'm not sure how to avoid that. It would be better if higher
layer just specified what columns it wants - user cols and acid meta cols and
interpret them so that RecordReader doesn't have to. Perhaps down the road we
can make the layout be just 1 struct where "c" are user columns and include
some version number in the ORC footer to know where the offset for the user
columns start (in case we add more metadata columns). I think this may have
other advantages.
I think someone more familiar with the code patch should look at this as well.
> Minor compaction produces ArrayIndexOutOfBoundsException: 7 in
> SchemaEvolution.getFileType
> --
>
> Key: HIVE-14004
> URL: https://issues.apache.org/jira/browse/HIVE-14004
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Eugene Koifman
>Assignee: Matt McCline
> Attachments: HIVE-14004.01.patch, HIVE-14004.02.patch,
> HIVE-14004.03.patch
>
>
> Easiest way to repro is to add TestTxnCommands2
> {noformat}
> @Test
> public void testCompactWithDelete() throws Exception {
> int[][] tableData = {{1,2},{3,4}};
> runStatementOnDriver("insert into " + Table.ACIDTBL + "(a,b) " +
> makeValuesClause(tableData));
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MAJOR'");
> Worker t = new Worker();
> t.setThreadId((int) t.getId());
> t.setHiveConf(hiveConf);
> AtomicBoolean stop = new AtomicBoolean();
> AtomicBoolean looped = new AtomicBoolean();
> stop.set(true);
> t.init(stop, looped);
> t.run();
> runStatementOnDriver("delete from " + Table.ACIDTBL + " where b = 4");
> runStatementOnDriver("update " + Table.ACIDTBL + " set b = -2 where b =
> 2");
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MINOR'");
> t.run();
> }
> {noformat}
> to TestTxnCommands2 and run it.
> Test won't fail but if you look
> in target/tmp/log/hive.log for the following exception (from Minor
> compaction).
> {noformat}
> 2016-06-09T18:36:39,071 WARN [Thread-190[]]: mapred.LocalJobRunner
> (LocalJobRunner.java:run(560)) - job_local1233973168_0005
> java.lang.Exception: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.6.1.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.6.1.jar:?]
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.orc.impl.SchemaEvolution.getFileType(SchemaEvolution.java:67)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2031)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.RecordReaderImpl.(RecordReaderImpl.java:208)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:63)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:365)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:207)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:508)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1977)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.txn.compactor.Compactor
[jira] [Commented] (HIVE-14175) Fix creating buckets without scheme information
[
https://issues.apache.org/jira/browse/HIVE-14175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369951#comment-15369951
]
Hive QA commented on HIVE-14175:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12817050/HIVE-14175.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 38 failed/errored test(s), 10302 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_15
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_gby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge11
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf_matchpath
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_self_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_two_cols
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_aggregate_without_gby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_precision
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_nullsafe_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_string_funcs
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_avro_decimal_native
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cross_product_check_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_enforce_order
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_lateral_view_explode2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_join_union
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_udf_min
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_17
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_rcfile_columnar
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/457/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/457/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-457/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 38 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12817050 - PreCommit-HIVE-MASTER-Build
> Fix creating buckets without scheme information
> ---
>
> Key: HIVE-14175
> URL: https://issues.apache.org/jira/browse/HIVE-14175
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Thomas Poepping
>Assignee: Thomas Poepping
> Labels: patch
> Attachments: HIVE-14175.2.patch, HIVE-14175.patch, HIVE-14175.patch
>
>
> If a table is created on a non-default filesystem (i.e. non-hdfs), the empty
> files will be created with incorrect scheme information. This p
[jira] [Updated] (HIVE-14152) datanucleus.autoStartMechanismMode should set to 'Ignored' to allow rolling downgrade
[
https://issues.apache.org/jira/browse/HIVE-14152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-14152:
-
Attachment: HIVE-14152.2.patch
2.patch - setting the autostartmechanism in ObjectStore to prevent cases where
old config has set it to incorrect value.
> datanucleus.autoStartMechanismMode should set to 'Ignored' to allow rolling
> downgrade
> --
>
> Key: HIVE-14152
> URL: https://issues.apache.org/jira/browse/HIVE-14152
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Daniel Dai
>Assignee: Thejas M Nair
> Attachments: HIVE-14152.1.patch, HIVE-14152.2.patch
>
>
> We see the following issue when downgrading metastore:
> 1. Run some query using new tables
> 2. Downgrade metastore
> 3. Restart metastore will complain the new table does not exist
> In particular, constaints tables does not exist in branch-1. If we run Hive 2
> and create a constraint, then downgrade metastore to Hive 1, datanucleus will
> complain:
> {code}
> javax.jdo.JDOFatalUserException: Error starting up DataNucleus : a class
> "org.apache.hadoop.hive.metastore.model.MConstraint" was listed as being
> persisted previously in this datastore, yet the class wasnt found. Perhaps it
> is used by a different DataNucleus-enabled application in this datastore, or
> you have changed your class names.
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:528)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
> at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:377)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:406)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:299)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:266)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:60)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:69)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:650)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:628)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:677)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:484)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:77)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:83)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5905)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5900)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6159)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6084)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJa
[jira] [Commented] (HIVE-13425) Fix partition addition in MSCK REPAIR TABLE command
[
https://issues.apache.org/jira/browse/HIVE-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369933#comment-15369933
]
Ashutosh Chauhan commented on HIVE-13425:
-
I don't understand following:
{code}
if (!path.toString().contains(checkPath)) {
+ String partKey = "";
+ for (int i = 0; i < kvs.size(); i++) {
+if (partKey.equals(kvs.get(i)[0])) {
+ partSpec.clear();
+ break;
+}
+partKey = kvs.get(i)[0];
+ }
{code}
your .4 patch made more sense to me. Can you describe motivation for diff
between .4 & .5?
> Fix partition addition in MSCK REPAIR TABLE command
> ---
>
> Key: HIVE-13425
> URL: https://issues.apache.org/jira/browse/HIVE-13425
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
>Priority: Minor
> Attachments: HIVE-13425.1.patch, HIVE-13425.2.patch,
> HIVE-13425.3.patch, HIVE-13425.4.patch, HIVE-13425.5.patch
>
>
> I create a table in following HiveQL.
> {code}
> hive> create table example (name string) partitioned by (id int);
> {code}
> And, I make some directories about this table in HDFS.
> {code}
> [root@hadoop ~]# hdfs dfs -ls -R /user/hive/warehouse/example
> drwxr-xr-x - root hadoop 0 2016-04-05 22:21
> /user/hive/warehouse/example/id=1
> drwxr-xr-x - root hadoop 0 2016-04-05 22:22
> /user/hive/warehouse/example/id=1/id=2
> -rw-r--r-- 1 root hadoop 8 2016-04-05 22:22
> /user/hive/warehouse/example/id=1/id=2/example.txt
> {code}
> Next I executed MSCK REPAIR TABLE command and added a partition. And this
> result became as follows.
> {code}
> [root@hadoop ~]# hive -e 'msck repair table example'
> OK
> Partitions not in metastore: example:id=1/id=2
> Repair: Added partition to metastore example:id=1/id=2
> Time taken: 1.243 seconds, Fetched: 2 row(s)
> [root@hadoop ~]# hive -e 'show partitions example'
> OK
> id=2
> {code}
> "id=1" should be a partition, but "id=2" was added. I will fix this problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13548) hive-jdbc isn't escaping slashes during PreparedStatement
[
https://issues.apache.org/jira/browse/HIVE-13548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369918#comment-15369918
]
Ashutosh Chauhan commented on HIVE-13548:
-
Patch on HIVE-13625 should resolve this too.
> hive-jdbc isn't escaping slashes during PreparedStatement
> -
>
> Key: HIVE-13548
> URL: https://issues.apache.org/jira/browse/HIVE-13548
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: Nasron Cheong
>Assignee: Nasron Cheong
> Attachments: HIVE-13548.patch
>
>
> Calling setString on a prepared statement with a string containing a '\' will
> cause the SQL construction to fail.
> I believe the slash should be escaped by the setString function.
> There may be other characters that require escaping during the same call.
> Failure from the unittest without the patch:
> {code}
> Running org.apache.hive.jdbc.TestJdbcDriver2
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 9.738 sec <<<
> FAILURE! - in org.apache.hive.jdbc.TestJdbcDriver2
> testSlashPreparedStatement(org.apache.hive.jdbc.TestJdbcDriver2) Time
> elapsed: 3.867 sec <<< FAILURE!
> java.lang.AssertionError: java.lang.StringIndexOutOfBoundsException: String
> index out of range: -1
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hive.jdbc.TestJdbcDriver2.testSlashPreparedStatement(TestJdbcDriver2.java:522)
> Results :
> Failed tests:
> TestJdbcDriver2.testSlashPreparedStatement:522
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13614) Implementation for PreparedStatement#setBigDecimal
[ https://issues.apache.org/jira/browse/HIVE-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369917#comment-15369917 ] Ashutosh Chauhan commented on HIVE-13614: - +1 > Implementation for PreparedStatement#setBigDecimal > -- > > Key: HIVE-13614 > URL: https://issues.apache.org/jira/browse/HIVE-13614 > Project: Hive > Issue Type: Improvement > Components: JDBC >Reporter: Iwao AVE! >Priority: Minor > Attachments: HIVE-13614.patch > > > Hi, > I am a committer of MyBatis project (https://github.com/mybatis) and some of > our users have difficulty with using MyBatis-Migrations (a db migration tool) > with Hive. > https://github.com/mybatis/migrations/issues/25 > Basically, Migrations uses BigDecimal as the ID type and the lack of support > in Hive's JDBC driver prevents our users from using Migrations with Hive. > So, it would be beneficial to both of Hive and MyBatis users if you could > apply the following change to mitigate the situation. > https://github.com/harawata/hive/commit/1910632442ad9b70b41b28e37596843d1b9f7d3f > Please let me know if I should send the modification as a GitHub pull request. > Thanks in advance! > Iwao -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13614) Implementation for PreparedStatement#setBigDecimal
[ https://issues.apache.org/jira/browse/HIVE-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13614: Assignee: Iwao AVE! > Implementation for PreparedStatement#setBigDecimal > -- > > Key: HIVE-13614 > URL: https://issues.apache.org/jira/browse/HIVE-13614 > Project: Hive > Issue Type: Improvement > Components: JDBC >Reporter: Iwao AVE! >Assignee: Iwao AVE! >Priority: Minor > Attachments: HIVE-13614.patch > > > Hi, > I am a committer of MyBatis project (https://github.com/mybatis) and some of > our users have difficulty with using MyBatis-Migrations (a db migration tool) > with Hive. > https://github.com/mybatis/migrations/issues/25 > Basically, Migrations uses BigDecimal as the ID type and the lack of support > in Hive's JDBC driver prevents our users from using Migrations with Hive. > So, it would be beneficial to both of Hive and MyBatis users if you could > apply the following change to mitigate the situation. > https://github.com/harawata/hive/commit/1910632442ad9b70b41b28e37596843d1b9f7d3f > Please let me know if I should send the modification as a GitHub pull request. > Thanks in advance! > Iwao -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13625) Hive Prepared Statement when executed with escape characters in parameter fails
[
https://issues.apache.org/jira/browse/HIVE-13625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369913#comment-15369913
]
Ashutosh Chauhan commented on HIVE-13625:
-
+1
> Hive Prepared Statement when executed with escape characters in parameter
> fails
> ---
>
> Key: HIVE-13625
> URL: https://issues.apache.org/jira/browse/HIVE-13625
> Project: Hive
> Issue Type: Bug
>Reporter: Daya Venkatesan
>Assignee: Zoltan Haindrich
> Attachments: HIVE-13625.patch
>
>
> When setting parameters to a Hive Prepared Statement, if the parameter has
> an odd number of escape characters, then the Statement fails.
> For example, I set one of the parameters to
> "/somepath/\044{}/\044{MM}/\044{dd}/". Here, I have escaped the dollar
> character with \044 because Hive gives an Atlas exception with "$" character.
> Now, when the parameters are getting set inside Hive,
> getCharIndexFromSqlByParamLocation throws an Exception.
> Hive records something called signal count.
> if (c == '\'' || c == '\\')// record the count of char "'" and char "\"
> {
> signalCount++;
> }
> And the parameter is set only if the signalCount %2 is 0.
> else if (c == cchar && signalCount % 2 == 0) {// check if the ? is really the
> parameter
> num++;
> if (num == paramLoc) {
> charIndex = i;
> break;
> }
> Since my parameter has three "\" characters, the signal Count modulo is not 0
> and the parameter is not set at all throwing an exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13644) Remove hardcoded groovy.grape.report.downloads=true from DependencyResolver
[
https://issues.apache.org/jira/browse/HIVE-13644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369907#comment-15369907
]
Ashutosh Chauhan commented on HIVE-13644:
-
+1
> Remove hardcoded groovy.grape.report.downloads=true from DependencyResolver
> ---
>
> Key: HIVE-13644
> URL: https://issues.apache.org/jira/browse/HIVE-13644
> Project: Hive
> Issue Type: Improvement
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13644.1.patch
>
>
> Currently, in Hive's
> [DependencyResolver.java|https://github.com/apache/hive/blob/8dd1d1966f2f0b86604b4e991ebc865224f42b41/ql/src/java/org/apache/hadoop/hive/ql/util/DependencyResolver.java#L176],
> the system property {{groovy.grape.report.downloads}} is hardcoded to
> {{true}} and there is no way to override it and disable the logging. We
> should remove this hardcoded value and allow users to configure it as they
> see fit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split
[
https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Svetozar Ivanov updated HIVE-13584:
---
Release Note:
- This patch gives opportunity for creation of pre-splitted HBase table via
Hive. Via TBLPROPERTIES could be configured following for HBase table:
- Splits number
- Split algorithm (if is not configured by default would be used
'org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit' algorithm)
- Split algorithm first row key
- Split algorithm last row key
For more details see provided example in description field of this issue.
was:
- This patch gives opportunity for creation of pre-splitted HBase table via
Hive. Via TBLPROPERTIES could be configured following for HBase table:
- Splits number
- Split algorithm (if is not configured by default would be used
'org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit' algorithm)
- Split algorithm first row key
- Split algorithm last row key
> HBaseStorageHandler should support table pre-split
> --
>
> Key: HIVE-13584
> URL: https://issues.apache.org/jira/browse/HIVE-13584
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler
>Affects Versions: 2.0.0
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
> Labels: HBase, PreSplit
> Attachments: HIVE-13584-branch-2.0.patch, HIVE-13584.1.patch,
> HIVE-13584.2.patch, HIVE-13584.3.patch, HIVE-13584.4.patch, HIVE-13584.patch
>
>
> Currently is not possible creation of HBase table via HBaseStorageHandler
> with specified pre-split strategy.
> I use it in my code with applied patch in that way:
> {code}
> public String buildCreateTblStatement(TableDescriptor tableDescriptor) {
> StringBuilder createTableStatement = ...;
> ...
> ...
> String tblProperties = buildTblProperties(tableDescriptor);
> if (StringUtils.isNotBlank(tblProperties)) {
> createTableStatement.append(" TBLPROPERTIES (" + tblProperties +
> ")");
> }
> ...
> return createTableStatement.toString();
> }
> private static String buildTblProperties(TableDescriptor tableDescriptor) {
> ...
> ...
> if (supportsPreSplit(tableDescriptor)) {
> sb.append(", ");
> sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLITS_NUMBER));
> sb.append(" = ");
>
> sb.append(quoteInSingleQuotes(String.valueOf(tableDescriptor.getNumRegions(;
> GeneratedPKDescriptor columnDescriptor = (GeneratedPKDescriptor)
> tableDescriptor.getPkDescriptors().get(0);
> sb.append(", ");
>
> sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_FIRSTROW_KEY));
> sb.append(" = ");
>
> sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().firstRowKey()));
> sb.append(", ");
>
> sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_LASTROW_KEY));
> sb.append(" = ");
>
> sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().lastRowKey()));
> }
> logger.trace("TBLPROPERTIES are [{}]", sb);
> return sb.toString();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split
[
https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Svetozar Ivanov updated HIVE-13584:
---
Release Note:
- This patch gives opportunity for creation of pre-splitted HBase table via
Hive. Via TBLPROPERTIES could be configured following for HBase table:
- Splits number
- Split algorithm (if is not configured by default would be used
'org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit' algorithm)
- Split algorithm first row key
- Split algorithm last row key
> HBaseStorageHandler should support table pre-split
> --
>
> Key: HIVE-13584
> URL: https://issues.apache.org/jira/browse/HIVE-13584
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler
>Affects Versions: 2.0.0
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
> Labels: HBase, PreSplit
> Attachments: HIVE-13584-branch-2.0.patch, HIVE-13584.1.patch,
> HIVE-13584.2.patch, HIVE-13584.3.patch, HIVE-13584.4.patch, HIVE-13584.patch
>
>
> Currently is not possible creation of HBase table via HBaseStorageHandler
> with specified pre-split strategy.
> I use it in my code with applied patch in that way:
> {code}
> public String buildCreateTblStatement(TableDescriptor tableDescriptor) {
> StringBuilder createTableStatement = ...;
> ...
> ...
> String tblProperties = buildTblProperties(tableDescriptor);
> if (StringUtils.isNotBlank(tblProperties)) {
> createTableStatement.append(" TBLPROPERTIES (" + tblProperties +
> ")");
> }
> ...
> return createTableStatement.toString();
> }
> private static String buildTblProperties(TableDescriptor tableDescriptor) {
> ...
> ...
> if (supportsPreSplit(tableDescriptor)) {
> sb.append(", ");
> sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLITS_NUMBER));
> sb.append(" = ");
>
> sb.append(quoteInSingleQuotes(String.valueOf(tableDescriptor.getNumRegions(;
> GeneratedPKDescriptor columnDescriptor = (GeneratedPKDescriptor)
> tableDescriptor.getPkDescriptors().get(0);
> sb.append(", ");
>
> sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_FIRSTROW_KEY));
> sb.append(" = ");
>
> sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().firstRowKey()));
> sb.append(", ");
>
> sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_LASTROW_KEY));
> sb.append(" = ");
>
> sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().lastRowKey()));
> }
> logger.trace("TBLPROPERTIES are [{}]", sb);
> return sb.toString();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13745) UDF current_date、current_timestamp、unix_timestamp NPE
[ https://issues.apache.org/jira/browse/HIVE-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369895#comment-15369895 ] Ashutosh Chauhan commented on HIVE-13745: - [~bill] Instead of storing timestamp in config, I think better approach is to store Timestamp value in constructor of udfs. That way when its deserialized on backend later we can use value of compile time. > UDF current_date、current_timestamp、unix_timestamp NPE > - > > Key: HIVE-13745 > URL: https://issues.apache.org/jira/browse/HIVE-13745 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Biao Wu >Assignee: Biao Wu > Attachments: HIVE-13745.patch > > > NullPointerException when current_date is used in mapreduce -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14203) ORC should throw if null row is passed to writer
[ https://issues.apache.org/jira/browse/HIVE-14203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14203: Attachment: HIVE-14203.patch [~prasanth_j] Can you please take a look? > ORC should throw if null row is passed to writer > > > Key: HIVE-14203 > URL: https://issues.apache.org/jira/browse/HIVE-14203 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1, 2.0.0, 2.1.0 >Reporter: Ashutosh Chauhan > Attachments: HIVE-14203.patch > > > Currently it silently drops the row (doesn't write in file) but updates > metadata. As a result of this data being inconsistent with metadata, its > impossible to read it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14203) ORC should throw if null row is passed to writer
[ https://issues.apache.org/jira/browse/HIVE-14203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reassigned HIVE-14203: --- Assignee: Ashutosh Chauhan > ORC should throw if null row is passed to writer > > > Key: HIVE-14203 > URL: https://issues.apache.org/jira/browse/HIVE-14203 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1, 2.0.0, 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14203.patch > > > Currently it silently drops the row (doesn't write in file) but updates > metadata. As a result of this data being inconsistent with metadata, its > impossible to read it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14129) Execute move tasks in parallel
[ https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14129: Status: Patch Available (was: Open) > Execute move tasks in parallel > -- > > Key: HIVE-14129 > URL: https://issues.apache.org/jira/browse/HIVE-14129 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14129.2.patch, HIVE-14129.patch, HIVE-14129.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14129) Execute move tasks in parallel
[ https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14129: Attachment: HIVE-14129.2.patch > Execute move tasks in parallel > -- > > Key: HIVE-14129 > URL: https://issues.apache.org/jira/browse/HIVE-14129 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14129.2.patch, HIVE-14129.patch, HIVE-14129.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14129) Execute move tasks in parallel
[ https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14129: Status: Open (was: Patch Available) > Execute move tasks in parallel > -- > > Key: HIVE-14129 > URL: https://issues.apache.org/jira/browse/HIVE-14129 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14129.patch, HIVE-14129.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14175) Fix creating buckets without scheme information
[ https://issues.apache.org/jira/browse/HIVE-14175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14175: Attachment: HIVE-14175.2.patch > Fix creating buckets without scheme information > --- > > Key: HIVE-14175 > URL: https://issues.apache.org/jira/browse/HIVE-14175 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1, 2.1.0 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Labels: patch > Attachments: HIVE-14175.2.patch, HIVE-14175.patch, HIVE-14175.patch > > > If a table is created on a non-default filesystem (i.e. non-hdfs), the empty > files will be created with incorrect scheme information. This patch extracts > the scheme and authority information for the new paths. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14175) Fix creating buckets without scheme information
[ https://issues.apache.org/jira/browse/HIVE-14175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14175: Status: Patch Available (was: Open) > Fix creating buckets without scheme information > --- > > Key: HIVE-14175 > URL: https://issues.apache.org/jira/browse/HIVE-14175 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.1.0, 1.2.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Labels: patch > Attachments: HIVE-14175.2.patch, HIVE-14175.patch, HIVE-14175.patch > > > If a table is created on a non-default filesystem (i.e. non-hdfs), the empty > files will be created with incorrect scheme information. This patch extracts > the scheme and authority information for the new paths. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14175) Fix creating buckets without scheme information
[ https://issues.apache.org/jira/browse/HIVE-14175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14175: Status: Open (was: Patch Available) > Fix creating buckets without scheme information > --- > > Key: HIVE-14175 > URL: https://issues.apache.org/jira/browse/HIVE-14175 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.1.0, 1.2.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Labels: patch > Attachments: HIVE-14175.patch, HIVE-14175.patch > > > If a table is created on a non-default filesystem (i.e. non-hdfs), the empty > files will be created with incorrect scheme information. This patch extracts > the scheme and authority information for the new paths. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13887) LazySimpleSerDe should parse "NULL" dates faster
[ https://issues.apache.org/jira/browse/HIVE-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13887: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Gopal! > LazySimpleSerDe should parse "NULL" dates faster > > > Key: HIVE-13887 > URL: https://issues.apache.org/jira/browse/HIVE-13887 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers, Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Labels: Performance > Fix For: 2.2.0 > > Attachments: HIVE-13887.1.patch, HIVE-13887.1.patch > > > Date string which contain "NULL" or "(null)" are being parsed through a very > slow codepath involving exception handling as a normal codepath. > These are currently ~4x slower than parsing an actual date field. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13763) Update smart-apply-patch.sh with ability to use patches from git
[
https://issues.apache.org/jira/browse/HIVE-13763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-13763:
Component/s: Testing Infrastructure
> Update smart-apply-patch.sh with ability to use patches from git
>
>
> Key: HIVE-13763
> URL: https://issues.apache.org/jira/browse/HIVE-13763
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 2.2.0
>
> Attachments: HIVE-13763.patch
>
>
> Currently, the smart-apply-patch.sh doesn't understand git patches. It is
> relatively easy to make it understand patches generated by:
> {code}
> % git format-patch apache/master --stdout > HIVE-999.patch
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13763) Update smart-apply-patch.sh with ability to use patches from git
[
https://issues.apache.org/jira/browse/HIVE-13763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-13763:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Pushed to master.
> Update smart-apply-patch.sh with ability to use patches from git
>
>
> Key: HIVE-13763
> URL: https://issues.apache.org/jira/browse/HIVE-13763
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 2.2.0
>
> Attachments: HIVE-13763.patch
>
>
> Currently, the smart-apply-patch.sh doesn't understand git patches. It is
> relatively easy to make it understand patches generated by:
> {code}
> % git format-patch apache/master --stdout > HIVE-999.patch
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13937) Unit test for HIVE-13051
[ https://issues.apache.org/jira/browse/HIVE-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13937: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Zoltan! > Unit test for HIVE-13051 > > > Key: HIVE-13937 > URL: https://issues.apache.org/jira/browse/HIVE-13937 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-13937.01.patch > > > unit test for HIVE-13051 ; it checks the issue prior to the fix, which > prevented further usage of a thread after an exception have occured -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13937) Unit test for HIVE-13051
[ https://issues.apache.org/jira/browse/HIVE-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13937: Component/s: Test > Unit test for HIVE-13051 > > > Key: HIVE-13937 > URL: https://issues.apache.org/jira/browse/HIVE-13937 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-13937.01.patch > > > unit test for HIVE-13051 ; it checks the issue prior to the fix, which > prevented further usage of a thread after an exception have occured -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14113) Create function failed but function in show function list
[
https://issues.apache.org/jira/browse/HIVE-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-14113:
Resolution: Fixed
Assignee: niklaus xiao (was: Navis)
Fix Version/s: (was: 1.3.0)
2.2.0
Status: Resolved (was: Patch Available)
Committed to master. Thanks, Niklaus!
> Create function failed but function in show function list
> -
>
> Key: HIVE-14113
> URL: https://issues.apache.org/jira/browse/HIVE-14113
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.2.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
> Fix For: 2.2.0
>
> Attachments: HIVE-14113.1.patch
>
>
> 1. create function with invalid hdfs path, /udf/udf-test.jar does not exists
> {quote}
> create function my_lower as 'com.tang.UDFLower' using jar
> 'hdfs:///udf/udf-test.jar';
> {quote}
> Failed with following exception:
> {quote}
> 0: jdbc:hive2://189.39.151.44:1/> create function my_lower as
> 'com.tang.UDFLower' using jar 'hdfs:///udf/udf-test.jar';
> INFO : converting to local hdfs:///udf/udf-test.jar
> ERROR : Failed to read external resource hdfs:///udf/udf-test.jar
> java.lang.RuntimeException: Failed to read external resource
> hdfs:///udf/udf-test.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1384)
> at
> org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1340)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1264)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1250)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:306)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:466)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerPermanentFunction(Registry.java:206)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerPermanentFunction(FunctionRegistry.java:1551)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.createPermanentFunction(FunctionTask.java:136)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:75)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1965)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1723)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1475)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1283)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:167)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:258)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: File does not exist:
> hdfs:/udf/udf-test.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1391)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1383)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2034)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2003)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1979)
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1370)
[jira] [Commented] (HIVE-14007) Replace ORC module with ORC release
[
https://issues.apache.org/jira/browse/HIVE-14007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369849#comment-15369849
]
Hive QA commented on HIVE-14007:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12817049/HIVE-14007.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/456/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/456/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-456/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-456/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
82b84ac..d09969b master -> origin/master
+ git reset --hard HEAD
HEAD is now at 82b84ac HIVE-14173: NPE was thrown after enabling directsql in
the middle of session (Chaoyu Tang, reviewed by Sergey Shelukhin)
+ git clean -f -d
Removing
hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseStorageHandlerTest.java
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at d09969b HIVE-14115 : Custom FetchFormatter is not supported (Ryu
Kobayashi via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12817049 - PreCommit-HIVE-MASTER-Build
> Replace ORC module with ORC release
> ---
>
> Key: HIVE-14007
> URL: https://issues.apache.org/jira/browse/HIVE-14007
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.2.0
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 2.2.0
>
> Attachments: HIVE-14007.patch, HIVE-14007.patch
>
>
> This completes moving the core ORC reader & writer to the ORC project.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14115) Custom FetchFormatter is not supported
[
https://issues.apache.org/jira/browse/HIVE-14115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-14115:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Committed to master. Thanks, Ryu!
> Custom FetchFormatter is not supported
> --
>
> Key: HIVE-14115
> URL: https://issues.apache.org/jira/browse/HIVE-14115
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14115.01.patch
>
>
> The following code is supported only FetchFormatter of ThriftFormatter and
> DefaultFetchFormatter. It can not be used Custom FetchFormatter.
> {code}
> if (SessionState.get().isHiveServerQuery()) {
>
> conf.set(SerDeUtils.LIST_SINK_OUTPUT_FORMATTER,ThriftFormatter.class.getName());
> } else {
> conf.set(SerDeUtils.LIST_SINK_OUTPUT_FORMATTER,
> DefaultFetchFormatter.class.getName());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14007) Replace ORC module with ORC release
[ https://issues.apache.org/jira/browse/HIVE-14007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-14007: - Attachment: HIVE-14007.patch This patch is rebased and moves to ORC 1.1.2. > Replace ORC module with ORC release > --- > > Key: HIVE-14007 > URL: https://issues.apache.org/jira/browse/HIVE-14007 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.2.0 >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.2.0 > > Attachments: HIVE-14007.patch, HIVE-14007.patch > > > This completes moving the core ORC reader & writer to the ORC project. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14004) Minor compaction produces ArrayIndexOutOfBoundsException: 7 in SchemaEvolution.getFileType
[
https://issues.apache.org/jira/browse/HIVE-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369834#comment-15369834
]
Matt McCline commented on HIVE-14004:
-
For patch #3, the list_bucket_dml_13 failure doesn't involve ORC and therefore
seems unrelated. The other failures are prior ones.
> Minor compaction produces ArrayIndexOutOfBoundsException: 7 in
> SchemaEvolution.getFileType
> --
>
> Key: HIVE-14004
> URL: https://issues.apache.org/jira/browse/HIVE-14004
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Eugene Koifman
>Assignee: Matt McCline
> Attachments: HIVE-14004.01.patch, HIVE-14004.02.patch,
> HIVE-14004.03.patch
>
>
> Easiest way to repro is to add TestTxnCommands2
> {noformat}
> @Test
> public void testCompactWithDelete() throws Exception {
> int[][] tableData = {{1,2},{3,4}};
> runStatementOnDriver("insert into " + Table.ACIDTBL + "(a,b) " +
> makeValuesClause(tableData));
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MAJOR'");
> Worker t = new Worker();
> t.setThreadId((int) t.getId());
> t.setHiveConf(hiveConf);
> AtomicBoolean stop = new AtomicBoolean();
> AtomicBoolean looped = new AtomicBoolean();
> stop.set(true);
> t.init(stop, looped);
> t.run();
> runStatementOnDriver("delete from " + Table.ACIDTBL + " where b = 4");
> runStatementOnDriver("update " + Table.ACIDTBL + " set b = -2 where b =
> 2");
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MINOR'");
> t.run();
> }
> {noformat}
> to TestTxnCommands2 and run it.
> Test won't fail but if you look
> in target/tmp/log/hive.log for the following exception (from Minor
> compaction).
> {noformat}
> 2016-06-09T18:36:39,071 WARN [Thread-190[]]: mapred.LocalJobRunner
> (LocalJobRunner.java:run(560)) - job_local1233973168_0005
> java.lang.Exception: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.6.1.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.6.1.jar:?]
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.orc.impl.SchemaEvolution.getFileType(SchemaEvolution.java:67)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2031)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.RecordReaderImpl.(RecordReaderImpl.java:208)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:63)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:365)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:207)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:508)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1977)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:630)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:609)
> ~[classes/:?]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.6.1.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
> ~[hadoop-mapreduce-client-core-2.6.1.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.6.1.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.6.1.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executor
[jira] [Commented] (HIVE-13584) HBaseStorageHandler should support table pre-split
[
https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369830#comment-15369830
]
Hive QA commented on HIVE-13584:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12817040/HIVE-13584.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10296 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/455/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/455/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-455/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12817040 - PreCommit-HIVE-MASTER-Build
> HBaseStorageHandler should support table pre-split
> --
>
> Key: HIVE-13584
> URL: https://issues.apache.org/jira/browse/HIVE-13584
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler
>Affects Versions: 2.0.0
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
> Labels: HBase, PreSplit
> Attachments: HIVE-13584-branch-2.0.patch, HIVE-13584.1.patch,
> HIVE-13584.2.patch, HIVE-13584.3.patch, HIVE-13584.4.patch, HIVE-13584.patch
>
>
> Currently is not possible creation of HBase table via HBaseStorageHandler
> with specified pre-split strategy.
> I use it in my code with applied patch in that way:
> {code}
> public String buildCreateTblStatement(TableDescriptor tableDescriptor) {
> StringBuilder createTableStatement = ...;
> ...
> ...
> String tblProperties = buildTblProperties(tableDescriptor);
> if (StringUtils.isNotBlank(tblProperties)) {
> createTableStatement.append(" TBLPROPERTIES (" + tblProperties +
> ")");
> }
> ...
> return createTableStatement.toString();
> }
> private static String buildTblProperties(TableDescriptor tableDescriptor) {
> ...
> ...
> if (supportsPreSplit(tableDescriptor)) {
> sb.append(", ");
> sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLITS_NUMBER));
> sb.append(" = ");
>
> sb.append(quoteInSingleQuotes(String.valueOf(tableDescriptor.getNumRegions(;
> GeneratedPKDescriptor columnDescriptor = (GeneratedPKDescriptor)
> tableDescriptor.getPkDescriptors().get(0);
> sb.append(", ");
>
> sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_FIRSTROW_KEY));
> sb.append(" = ");
>
> sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().firstRowKey()));
> sb.append(", ");
>
> sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_LASTROW_KEY));
> sb.append(" = ");
>
> sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().lastRowKey()));
> }
> logger.trace("TBLPROPERTIES are [{}]", sb);
> return sb.toString();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Svetozar Ivanov updated HIVE-11233:
---
Description:
Currently HBaseStorageHandler doesn't provide mechanism for storage of binary
sortable key and values. It is necessary when given HBase table is used for
persistence by Apache Hive and Apache Phoenix. In that way all byte arrays read
or written by Hive will be compatible with binary sortable format used in
Phoenix.
It turns out the major difference is in all numeric data types accordingly
officially provided documentation -
https://phoenix.apache.org/language/datatypes.html.
That's how I use it in my code with applied patch:
{code}
private static String buildWithSerDeProperties(TableDescriptor
tableDescriptor) {
Map serdePropertiesMap = new HashMap<>();
serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME,
tableDescriptor.getTableName());
serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE,
BINARY_STORAGE_TYPE);
serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING,
buildHBaseColumnsDefinition(tableDescriptor));
serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS,
PhoenixValueFactory.class.getName());
/* Use different key factory for simple and composite primary key */
if (tableDescriptor.getPkDescriptors().size() == 1) {
serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS,
PhoenixKeyFactory.class.getName());
} else {
serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY,
PhoenixCompositeKeyFactory.class.getName());
}
String serDeProperties = serdePropertiesMap.entrySet().stream()
.map(e -> quoteInSingleQuotes(e.getKey()) + " = " +
quoteInSingleQuotes(e.getValue()))
.collect(Collectors.joining(COLUMNS_SEPARATOR));
logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
return serDeProperties;
}
{code}
was:
Currently HBaseStorageHandler doesn't provide mechanism for storage of binary
sortable key and values. It is necessary when given HBase table is used for
persistence by Apache Hive and Apache Phoenix. In that way all byte arrays read
or written by Hive will be compatible with binary sortable format used in
Phoenix.
It turns out the major difference is in all numeric data types accordingly
officially provided documentation -
https://phoenix.apache.org/language/datatypes.html.
That's how I'm using it in my code with applied patch:
{code}
private static String buildWithSerDeProperties(TableDescriptor
tableDescriptor) {
Map serdePropertiesMap = new HashMap<>();
serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME,
tableDescriptor.getTableName());
serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE,
BINARY_STORAGE_TYPE);
serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING,
buildHBaseColumnsDefinition(tableDescriptor));
serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS,
PhoenixValueFactory.class.getName());
/* Use different key factory for simple and composite primary key */
if (tableDescriptor.getPkDescriptors().size() == 1) {
serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS,
PhoenixKeyFactory.class.getName());
} else {
serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY,
PhoenixCompositeKeyFactory.class.getName());
}
String serDeProperties = serdePropertiesMap.entrySet().stream()
.map(e -> quoteInSingleQuotes(e.getKey()) + " = " +
quoteInSingleQuotes(e.getValue()))
.collect(Collectors.joining(COLUMNS_SEPARATOR));
logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
return serDeProperties;
}
{code}
> Include Apache Phoenix support in HBaseStorageHandler
> -
>
> Key: HIVE-11233
> URL: https://issues.apache.org/jira/browse/HIVE-11233
> Project: Hive
> Issue Type: New Feature
> Components: HBase Handler
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
> Labels: Binary, Hbase, Numeric, Phoenix, Sortable
> Attachments: HIVE-11233-branch-1.2.patch,
> HIVE-11233-branch-2.0.patch, HIVE-11233.1.patch, HIVE-11233.2.patch,
> HIVE-11233.3.patch, HIVE-11233.4.patch, HIVE-11233.5.patch, HIVE-11233.patch
>
>
> Currently HBaseStorageHandler doesn't provide mechanism for storage of binary
> sortable key and values. It is necessary when given HBase table is used for
> persistence by Apache Hive and Apache Phoenix. In that way all byte arrays
> read or written by Hive will be compatible with binary sortable format used
> in Phoenix.
> It turns out the major diff
[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split
[
https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Svetozar Ivanov updated HIVE-13584:
---
Description:
Currently is not possible creation of HBase table via HBaseStorageHandler with
specified pre-split strategy.
I use it in my code with applied patch in that way:
{code}
public String buildCreateTblStatement(TableDescriptor tableDescriptor) {
StringBuilder createTableStatement = ...;
...
...
String tblProperties = buildTblProperties(tableDescriptor);
if (StringUtils.isNotBlank(tblProperties)) {
createTableStatement.append(" TBLPROPERTIES (" + tblProperties +
")");
}
...
return createTableStatement.toString();
}
private static String buildTblProperties(TableDescriptor tableDescriptor) {
...
...
if (supportsPreSplit(tableDescriptor)) {
sb.append(", ");
sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLITS_NUMBER));
sb.append(" = ");
sb.append(quoteInSingleQuotes(String.valueOf(tableDescriptor.getNumRegions(;
GeneratedPKDescriptor columnDescriptor = (GeneratedPKDescriptor)
tableDescriptor.getPkDescriptors().get(0);
sb.append(", ");
sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_FIRSTROW_KEY));
sb.append(" = ");
sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().firstRowKey()));
sb.append(", ");
sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_LASTROW_KEY));
sb.append(" = ");
sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().lastRowKey()));
}
logger.trace("TBLPROPERTIES are [{}]", sb);
return sb.toString();
}
{code}
was:Currently is not possible creation of HBase table via HBaseStorageHandler
with specified pre-split strategy.
> HBaseStorageHandler should support table pre-split
> --
>
> Key: HIVE-13584
> URL: https://issues.apache.org/jira/browse/HIVE-13584
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler
>Affects Versions: 2.0.0
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
> Labels: HBase, PreSplit
> Attachments: HIVE-13584-branch-2.0.patch, HIVE-13584.1.patch,
> HIVE-13584.2.patch, HIVE-13584.3.patch, HIVE-13584.4.patch, HIVE-13584.patch
>
>
> Currently is not possible creation of HBase table via HBaseStorageHandler
> with specified pre-split strategy.
> I use it in my code with applied patch in that way:
> {code}
> public String buildCreateTblStatement(TableDescriptor tableDescriptor) {
> StringBuilder createTableStatement = ...;
> ...
> ...
> String tblProperties = buildTblProperties(tableDescriptor);
> if (StringUtils.isNotBlank(tblProperties)) {
> createTableStatement.append(" TBLPROPERTIES (" + tblProperties +
> ")");
> }
> ...
> return createTableStatement.toString();
> }
> private static String buildTblProperties(TableDescriptor tableDescriptor) {
> ...
> ...
> if (supportsPreSplit(tableDescriptor)) {
> sb.append(", ");
> sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLITS_NUMBER));
> sb.append(" = ");
>
> sb.append(quoteInSingleQuotes(String.valueOf(tableDescriptor.getNumRegions(;
> GeneratedPKDescriptor columnDescriptor = (GeneratedPKDescriptor)
> tableDescriptor.getPkDescriptors().get(0);
> sb.append(", ");
>
> sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_FIRSTROW_KEY));
> sb.append(" = ");
>
> sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().firstRowKey()));
> sb.append(", ");
>
> sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_LASTROW_KEY));
> sb.append(" = ");
>
> sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().lastRowKey()));
> }
> logger.trace("TBLPROPERTIES are [{}]", sb);
> return sb.toString();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Svetozar Ivanov updated HIVE-11233:
---
Description:
Currently HBaseStorageHandler doesn't provide mechanism for storage of binary
sortable key and values. It is necessary when given HBase table is used for
persistence by Apache Hive and Apache Phoenix. In that way all byte arrays read
or written by Hive will be compatible with binary sortable format used in
Phoenix.
It turns out the major difference is in all numeric data types accordingly
officially provided documentation -
https://phoenix.apache.org/language/datatypes.html.
That's how I'm using it in my code with applied patch:
{code}
private static String buildWithSerDeProperties(TableDescriptor
tableDescriptor) {
Map serdePropertiesMap = new HashMap<>();
serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME,
tableDescriptor.getTableName());
serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE,
BINARY_STORAGE_TYPE);
serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING,
buildHBaseColumnsDefinition(tableDescriptor));
serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS,
PhoenixValueFactory.class.getName());
/* Use different key factory for simple and composite primary key */
if (tableDescriptor.getPkDescriptors().size() == 1) {
serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS,
PhoenixKeyFactory.class.getName());
} else {
serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY,
PhoenixCompositeKeyFactory.class.getName());
}
String serDeProperties = serdePropertiesMap.entrySet().stream()
.map(e -> quoteInSingleQuotes(e.getKey()) + " = " +
quoteInSingleQuotes(e.getValue()))
.collect(Collectors.joining(COLUMNS_SEPARATOR));
logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
return serDeProperties;
}
{code}
was:
Currently HBaseStorageHandler doesn't provide mechanism for storage of binary
sortable key and values. It is necessary when given HBase table is used for
persistence by Apache Hive and Apache Phoenix. In that way all byte arrays read
or written by Hive will be compatible with binary sortable format used in
Phoenix.
It turns out the major difference is in all numeric data types accordingly
officially provided documentation -
https://phoenix.apache.org/language/datatypes.html.
That's how I'm using it in my code:
{code}
private static String buildWithSerDeProperties(TableDescriptor
tableDescriptor) {
Map serdePropertiesMap = new HashMap<>();
serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME,
tableDescriptor.getTableName());
serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE,
BINARY_STORAGE_TYPE);
serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING,
buildHBaseColumnsDefinition(tableDescriptor));
serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS,
PhoenixValueFactory.class.getName());
/* Use different key factory for simple and composite primary key */
if (tableDescriptor.getPkDescriptors().size() == 1) {
serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS,
PhoenixKeyFactory.class.getName());
} else {
serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY,
PhoenixCompositeKeyFactory.class.getName());
}
String serDeProperties = serdePropertiesMap.entrySet().stream()
.map(e -> quoteInSingleQuotes(e.getKey()) + " = " +
quoteInSingleQuotes(e.getValue()))
.collect(Collectors.joining(COLUMNS_SEPARATOR));
logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
return serDeProperties;
}
{code}
> Include Apache Phoenix support in HBaseStorageHandler
> -
>
> Key: HIVE-11233
> URL: https://issues.apache.org/jira/browse/HIVE-11233
> Project: Hive
> Issue Type: New Feature
> Components: HBase Handler
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
> Labels: Binary, Hbase, Numeric, Phoenix, Sortable
> Attachments: HIVE-11233-branch-1.2.patch,
> HIVE-11233-branch-2.0.patch, HIVE-11233.1.patch, HIVE-11233.2.patch,
> HIVE-11233.3.patch, HIVE-11233.4.patch, HIVE-11233.5.patch, HIVE-11233.patch
>
>
> Currently HBaseStorageHandler doesn't provide mechanism for storage of binary
> sortable key and values. It is necessary when given HBase table is used for
> persistence by Apache Hive and Apache Phoenix. In that way all byte arrays
> read or written by Hive will be compatible with binary sortable format used
> in Phoenix.
> It turns out the major difference is in al
[jira] [Updated] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Svetozar Ivanov updated HIVE-11233:
---
Release Note:
Thanks of this patch are possible read or write operations for HBase table in
Apache Phoenix persistence mode. Following classes could be considered as 3
keypoints:
- org.apache.hadoop.hive.hbase.phoenix.PhoenixValueFactory
- org.apache.hadoop.hive.hbase.phoenix.PhoenixKeyFactory
- org.apache.hadoop.hive.hbase.phoenix.PhoenixCompositeKeyFactory
Note: See the code sample provided in description field of that issue for more
details.
was:
Thanks of this patch are possible read or write operations for HBase table in
Apache Phoenix persistence mode. Following classes could be considered as 3
keypoints:
- org.apache.hadoop.hive.hbase.phoenix.PhoenixValueFactory
- org.apache.hadoop.hive.hbase.phoenix.PhoenixKeyFactory
- org.apache.hadoop.hive.hbase.phoenix.PhoenixCompositeKeyFactory
> Include Apache Phoenix support in HBaseStorageHandler
> -
>
> Key: HIVE-11233
> URL: https://issues.apache.org/jira/browse/HIVE-11233
> Project: Hive
> Issue Type: New Feature
> Components: HBase Handler
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
> Labels: Binary, Hbase, Numeric, Phoenix, Sortable
> Attachments: HIVE-11233-branch-1.2.patch,
> HIVE-11233-branch-2.0.patch, HIVE-11233.1.patch, HIVE-11233.2.patch,
> HIVE-11233.3.patch, HIVE-11233.4.patch, HIVE-11233.5.patch, HIVE-11233.patch
>
>
> Currently HBaseStorageHandler doesn't provide mechanism for storage of binary
> sortable key and values. It is necessary when given HBase table is used for
> persistence by Apache Hive and Apache Phoenix. In that way all byte arrays
> read or written by Hive will be compatible with binary sortable format used
> in Phoenix.
> It turns out the major difference is in all numeric data types accordingly
> officially provided documentation -
> https://phoenix.apache.org/language/datatypes.html.
> That's how I'm using it in my code:
> {code}
> private static String buildWithSerDeProperties(TableDescriptor
> tableDescriptor) {
> Map serdePropertiesMap = new HashMap<>();
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME,
> tableDescriptor.getTableName());
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE,
> BINARY_STORAGE_TYPE);
> serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING,
> buildHBaseColumnsDefinition(tableDescriptor));
> serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS,
> PhoenixValueFactory.class.getName());
> /* Use different key factory for simple and composite primary key */
> if (tableDescriptor.getPkDescriptors().size() == 1) {
> serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS,
> PhoenixKeyFactory.class.getName());
> } else {
> serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY,
> PhoenixCompositeKeyFactory.class.getName());
> }
> String serDeProperties = serdePropertiesMap.entrySet().stream()
> .map(e -> quoteInSingleQuotes(e.getKey()) + " = " +
> quoteInSingleQuotes(e.getValue()))
> .collect(Collectors.joining(COLUMNS_SEPARATOR));
> logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
> return serDeProperties;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Svetozar Ivanov updated HIVE-11233:
---
Release Note:
Thanks of this patch are possible read or write operations for HBase table in
Apache Phoenix persistence mode. Following classes could be considered as 3
keypoints:
- org.apache.hadoop.hive.hbase.phoenix.PhoenixValueFactory
- org.apache.hadoop.hive.hbase.phoenix.PhoenixKeyFactory
- org.apache.hadoop.hive.hbase.phoenix.PhoenixCompositeKeyFactory
> Include Apache Phoenix support in HBaseStorageHandler
> -
>
> Key: HIVE-11233
> URL: https://issues.apache.org/jira/browse/HIVE-11233
> Project: Hive
> Issue Type: New Feature
> Components: HBase Handler
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
> Labels: Binary, Hbase, Numeric, Phoenix, Sortable
> Attachments: HIVE-11233-branch-1.2.patch,
> HIVE-11233-branch-2.0.patch, HIVE-11233.1.patch, HIVE-11233.2.patch,
> HIVE-11233.3.patch, HIVE-11233.4.patch, HIVE-11233.5.patch, HIVE-11233.patch
>
>
> Currently HBaseStorageHandler doesn't provide mechanism for storage of binary
> sortable key and values. It is necessary when given HBase table is used for
> persistence by Apache Hive and Apache Phoenix. In that way all byte arrays
> read or written by Hive will be compatible with binary sortable format used
> in Phoenix.
> It turns out the major difference is in all numeric data types accordingly
> officially provided documentation -
> https://phoenix.apache.org/language/datatypes.html.
> That's how I'm using it in my code:
> {code}
> private static String buildWithSerDeProperties(TableDescriptor
> tableDescriptor) {
> Map serdePropertiesMap = new HashMap<>();
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME,
> tableDescriptor.getTableName());
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE,
> BINARY_STORAGE_TYPE);
> serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING,
> buildHBaseColumnsDefinition(tableDescriptor));
> serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS,
> PhoenixValueFactory.class.getName());
> /* Use different key factory for simple and composite primary key */
> if (tableDescriptor.getPkDescriptors().size() == 1) {
> serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS,
> PhoenixKeyFactory.class.getName());
> } else {
> serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY,
> PhoenixCompositeKeyFactory.class.getName());
> }
> String serDeProperties = serdePropertiesMap.entrySet().stream()
> .map(e -> quoteInSingleQuotes(e.getKey()) + " = " +
> quoteInSingleQuotes(e.getValue()))
> .collect(Collectors.joining(COLUMNS_SEPARATOR));
> logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
> return serDeProperties;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split
[ https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-13584: --- Status: Patch Available (was: Open) > HBaseStorageHandler should support table pre-split > -- > > Key: HIVE-13584 > URL: https://issues.apache.org/jira/browse/HIVE-13584 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 2.0.0 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: HBase, PreSplit > Attachments: HIVE-13584-branch-2.0.patch, HIVE-13584.1.patch, > HIVE-13584.2.patch, HIVE-13584.3.patch, HIVE-13584.4.patch, HIVE-13584.patch > > > Currently is not possible creation of HBase table via HBaseStorageHandler > with specified pre-split strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split
[ https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-13584: --- Attachment: HIVE-13584.4.patch > HBaseStorageHandler should support table pre-split > -- > > Key: HIVE-13584 > URL: https://issues.apache.org/jira/browse/HIVE-13584 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 2.0.0 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: HBase, PreSplit > Attachments: HIVE-13584-branch-2.0.patch, HIVE-13584.1.patch, > HIVE-13584.2.patch, HIVE-13584.3.patch, HIVE-13584.4.patch, HIVE-13584.patch > > > Currently is not possible creation of HBase table via HBaseStorageHandler > with specified pre-split strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split
[ https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-13584: --- Status: Open (was: Patch Available) > HBaseStorageHandler should support table pre-split > -- > > Key: HIVE-13584 > URL: https://issues.apache.org/jira/browse/HIVE-13584 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 2.0.0 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: HBase, PreSplit > Attachments: HIVE-13584-branch-2.0.patch, HIVE-13584.1.patch, > HIVE-13584.2.patch, HIVE-13584.3.patch, HIVE-13584.patch > > > Currently is not possible creation of HBase table via HBaseStorageHandler > with specified pre-split strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14004) Minor compaction produces ArrayIndexOutOfBoundsException: 7 in SchemaEvolution.getFileType
[
https://issues.apache.org/jira/browse/HIVE-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369616#comment-15369616
]
Hive QA commented on HIVE-14004:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12817027/HIVE-14004.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10297 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/454/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/454/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-454/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12817027 - PreCommit-HIVE-MASTER-Build
> Minor compaction produces ArrayIndexOutOfBoundsException: 7 in
> SchemaEvolution.getFileType
> --
>
> Key: HIVE-14004
> URL: https://issues.apache.org/jira/browse/HIVE-14004
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Eugene Koifman
>Assignee: Matt McCline
> Attachments: HIVE-14004.01.patch, HIVE-14004.02.patch,
> HIVE-14004.03.patch
>
>
> Easiest way to repro is to add TestTxnCommands2
> {noformat}
> @Test
> public void testCompactWithDelete() throws Exception {
> int[][] tableData = {{1,2},{3,4}};
> runStatementOnDriver("insert into " + Table.ACIDTBL + "(a,b) " +
> makeValuesClause(tableData));
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MAJOR'");
> Worker t = new Worker();
> t.setThreadId((int) t.getId());
> t.setHiveConf(hiveConf);
> AtomicBoolean stop = new AtomicBoolean();
> AtomicBoolean looped = new AtomicBoolean();
> stop.set(true);
> t.init(stop, looped);
> t.run();
> runStatementOnDriver("delete from " + Table.ACIDTBL + " where b = 4");
> runStatementOnDriver("update " + Table.ACIDTBL + " set b = -2 where b =
> 2");
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MINOR'");
> t.run();
> }
> {noformat}
> to TestTxnCommands2 and run it.
> Test won't fail but if you look
> in target/tmp/log/hive.log for the following exception (from Minor
> compaction).
> {noformat}
> 2016-06-09T18:36:39,071 WARN [Thread-190[]]: mapred.LocalJobRunner
> (LocalJobRunner.java:run(560)) - job_local1233973168_0005
> java.lang.Exception: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.6.1.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.6.1.jar:?]
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.orc.impl.SchemaEvolution.getFileType(SchemaEvolution.java:67)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2031)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc
[jira] [Assigned] (HIVE-14179) Too many delta files causes select queries on the table to fail with OOM
[
https://issues.apache.org/jira/browse/HIVE-14179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rammohan Krishnan reassigned HIVE-14179:
Assignee: Rammohan Krishnan
> Too many delta files causes select queries on the table to fail with OOM
>
>
> Key: HIVE-14179
> URL: https://issues.apache.org/jira/browse/HIVE-14179
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.2.0
>Reporter: Deepesh Khandelwal
>Assignee: Rammohan Krishnan
>
> When a large number of delta files get generated during ACID operations, a
> select query on the ACID table fails with OOM.
> {noformat}
> ERROR [main]: SessionState (SessionState.java:printError(942)) - Vertex
> failed, vertexName=Map 1, vertexId=vertex_1465431842106_0014_1_00,
> diagnostics=[Task failed, taskId=task_1465431842106_0014_1_00_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Direct buffer
> memory
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:159)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:693)
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
> at
> org.apache.hadoop.util.DirectBufferPool.getBuffer(DirectBufferPool.java:72)
> at
> org.apache.hadoop.hdfs.BlockReaderLocal.createDataBufIfNeeded(BlockReaderLocal.java:260)
> at
> org.apache.hadoop.hdfs.BlockReaderLocal.readWithBounceBuffer(BlockReaderLocal.java:601)
> at
> org.apache.hadoop.hdfs.BlockReaderLocal.read(BlockReaderLocal.java:569)
> at
> org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:789)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:845)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:905)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:953)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:377)
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:323)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:238)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:462)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1372)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1264)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:251)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:193)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:135)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:101)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:149)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:80)
> at
> org.apache.tez.mapreduce.input.MRInput.init
[jira] [Updated] (HIVE-14004) Minor compaction produces ArrayIndexOutOfBoundsException: 7 in SchemaEvolution.getFileType
[
https://issues.apache.org/jira/browse/HIVE-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-14004:
Status: Patch Available (was: In Progress)
> Minor compaction produces ArrayIndexOutOfBoundsException: 7 in
> SchemaEvolution.getFileType
> --
>
> Key: HIVE-14004
> URL: https://issues.apache.org/jira/browse/HIVE-14004
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Eugene Koifman
>Assignee: Matt McCline
> Attachments: HIVE-14004.01.patch, HIVE-14004.02.patch,
> HIVE-14004.03.patch
>
>
> Easiest way to repro is to add TestTxnCommands2
> {noformat}
> @Test
> public void testCompactWithDelete() throws Exception {
> int[][] tableData = {{1,2},{3,4}};
> runStatementOnDriver("insert into " + Table.ACIDTBL + "(a,b) " +
> makeValuesClause(tableData));
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MAJOR'");
> Worker t = new Worker();
> t.setThreadId((int) t.getId());
> t.setHiveConf(hiveConf);
> AtomicBoolean stop = new AtomicBoolean();
> AtomicBoolean looped = new AtomicBoolean();
> stop.set(true);
> t.init(stop, looped);
> t.run();
> runStatementOnDriver("delete from " + Table.ACIDTBL + " where b = 4");
> runStatementOnDriver("update " + Table.ACIDTBL + " set b = -2 where b =
> 2");
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MINOR'");
> t.run();
> }
> {noformat}
> to TestTxnCommands2 and run it.
> Test won't fail but if you look
> in target/tmp/log/hive.log for the following exception (from Minor
> compaction).
> {noformat}
> 2016-06-09T18:36:39,071 WARN [Thread-190[]]: mapred.LocalJobRunner
> (LocalJobRunner.java:run(560)) - job_local1233973168_0005
> java.lang.Exception: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.6.1.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.6.1.jar:?]
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.orc.impl.SchemaEvolution.getFileType(SchemaEvolution.java:67)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2031)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.RecordReaderImpl.(RecordReaderImpl.java:208)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:63)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:365)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:207)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:508)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1977)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:630)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:609)
> ~[classes/:?]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.6.1.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
> ~[hadoop-mapreduce-client-core-2.6.1.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.6.1.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.6.1.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_71]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_71]
> at
> java.util.c
[jira] [Updated] (HIVE-14004) Minor compaction produces ArrayIndexOutOfBoundsException: 7 in SchemaEvolution.getFileType
[
https://issues.apache.org/jira/browse/HIVE-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-14004:
Attachment: HIVE-14004.03.patch
> Minor compaction produces ArrayIndexOutOfBoundsException: 7 in
> SchemaEvolution.getFileType
> --
>
> Key: HIVE-14004
> URL: https://issues.apache.org/jira/browse/HIVE-14004
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Eugene Koifman
>Assignee: Matt McCline
> Attachments: HIVE-14004.01.patch, HIVE-14004.02.patch,
> HIVE-14004.03.patch
>
>
> Easiest way to repro is to add TestTxnCommands2
> {noformat}
> @Test
> public void testCompactWithDelete() throws Exception {
> int[][] tableData = {{1,2},{3,4}};
> runStatementOnDriver("insert into " + Table.ACIDTBL + "(a,b) " +
> makeValuesClause(tableData));
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MAJOR'");
> Worker t = new Worker();
> t.setThreadId((int) t.getId());
> t.setHiveConf(hiveConf);
> AtomicBoolean stop = new AtomicBoolean();
> AtomicBoolean looped = new AtomicBoolean();
> stop.set(true);
> t.init(stop, looped);
> t.run();
> runStatementOnDriver("delete from " + Table.ACIDTBL + " where b = 4");
> runStatementOnDriver("update " + Table.ACIDTBL + " set b = -2 where b =
> 2");
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MINOR'");
> t.run();
> }
> {noformat}
> to TestTxnCommands2 and run it.
> Test won't fail but if you look
> in target/tmp/log/hive.log for the following exception (from Minor
> compaction).
> {noformat}
> 2016-06-09T18:36:39,071 WARN [Thread-190[]]: mapred.LocalJobRunner
> (LocalJobRunner.java:run(560)) - job_local1233973168_0005
> java.lang.Exception: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.6.1.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.6.1.jar:?]
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.orc.impl.SchemaEvolution.getFileType(SchemaEvolution.java:67)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2031)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.RecordReaderImpl.(RecordReaderImpl.java:208)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:63)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:365)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:207)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:508)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1977)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:630)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:609)
> ~[classes/:?]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.6.1.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
> ~[hadoop-mapreduce-client-core-2.6.1.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.6.1.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.6.1.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_71]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_71]
> at
> java.util.concurrent.Th
[jira] [Updated] (HIVE-14004) Minor compaction produces ArrayIndexOutOfBoundsException: 7 in SchemaEvolution.getFileType
[
https://issues.apache.org/jira/browse/HIVE-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-14004:
Status: In Progress (was: Patch Available)
> Minor compaction produces ArrayIndexOutOfBoundsException: 7 in
> SchemaEvolution.getFileType
> --
>
> Key: HIVE-14004
> URL: https://issues.apache.org/jira/browse/HIVE-14004
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Eugene Koifman
>Assignee: Matt McCline
> Attachments: HIVE-14004.01.patch, HIVE-14004.02.patch,
> HIVE-14004.03.patch
>
>
> Easiest way to repro is to add TestTxnCommands2
> {noformat}
> @Test
> public void testCompactWithDelete() throws Exception {
> int[][] tableData = {{1,2},{3,4}};
> runStatementOnDriver("insert into " + Table.ACIDTBL + "(a,b) " +
> makeValuesClause(tableData));
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MAJOR'");
> Worker t = new Worker();
> t.setThreadId((int) t.getId());
> t.setHiveConf(hiveConf);
> AtomicBoolean stop = new AtomicBoolean();
> AtomicBoolean looped = new AtomicBoolean();
> stop.set(true);
> t.init(stop, looped);
> t.run();
> runStatementOnDriver("delete from " + Table.ACIDTBL + " where b = 4");
> runStatementOnDriver("update " + Table.ACIDTBL + " set b = -2 where b =
> 2");
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MINOR'");
> t.run();
> }
> {noformat}
> to TestTxnCommands2 and run it.
> Test won't fail but if you look
> in target/tmp/log/hive.log for the following exception (from Minor
> compaction).
> {noformat}
> 2016-06-09T18:36:39,071 WARN [Thread-190[]]: mapred.LocalJobRunner
> (LocalJobRunner.java:run(560)) - job_local1233973168_0005
> java.lang.Exception: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.6.1.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.6.1.jar:?]
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.orc.impl.SchemaEvolution.getFileType(SchemaEvolution.java:67)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2031)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.RecordReaderImpl.(RecordReaderImpl.java:208)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:63)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:365)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:207)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:508)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1977)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:630)
> ~[classes/:?]
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:609)
> ~[classes/:?]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.6.1.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
> ~[hadoop-mapreduce-client-core-2.6.1.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.6.1.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.6.1.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_71]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_71]
> at
> java.util.c
[jira] [Commented] (HIVE-14004) Minor compaction produces ArrayIndexOutOfBoundsException: 7 in SchemaEvolution.getFileType
[
https://issues.apache.org/jira/browse/HIVE-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369520#comment-15369520
]
Hive QA commented on HIVE-14004:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12817014/HIVE-14004.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10297 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/453/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/453/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-453/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12817014 - PreCommit-HIVE-MASTER-Build
> Minor compaction produces ArrayIndexOutOfBoundsException: 7 in
> SchemaEvolution.getFileType
> --
>
> Key: HIVE-14004
> URL: https://issues.apache.org/jira/browse/HIVE-14004
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Eugene Koifman
>Assignee: Matt McCline
> Attachments: HIVE-14004.01.patch, HIVE-14004.02.patch
>
>
> Easiest way to repro is to add TestTxnCommands2
> {noformat}
> @Test
> public void testCompactWithDelete() throws Exception {
> int[][] tableData = {{1,2},{3,4}};
> runStatementOnDriver("insert into " + Table.ACIDTBL + "(a,b) " +
> makeValuesClause(tableData));
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MAJOR'");
> Worker t = new Worker();
> t.setThreadId((int) t.getId());
> t.setHiveConf(hiveConf);
> AtomicBoolean stop = new AtomicBoolean();
> AtomicBoolean looped = new AtomicBoolean();
> stop.set(true);
> t.init(stop, looped);
> t.run();
> runStatementOnDriver("delete from " + Table.ACIDTBL + " where b = 4");
> runStatementOnDriver("update " + Table.ACIDTBL + " set b = -2 where b =
> 2");
> runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MINOR'");
> t.run();
> }
> {noformat}
> to TestTxnCommands2 and run it.
> Test won't fail but if you look
> in target/tmp/log/hive.log for the following exception (from Minor
> compaction).
> {noformat}
> 2016-06-09T18:36:39,071 WARN [Thread-190[]]: mapred.LocalJobRunner
> (LocalJobRunner.java:run(560)) - job_local1233973168_0005
> java.lang.Exception: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.6.1.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.6.1.jar:?]
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 7
> at
> org.apache.orc.impl.SchemaEvolution.getFileType(SchemaEvolution.java:67)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2031)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:1716)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2077)
> ~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.orc.impl.RecordReaderImpl.(Re
[jira] [Commented] (HIVE-14202) Change tez version used to 0.8.4
[
https://issues.apache.org/jira/browse/HIVE-14202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369467#comment-15369467
]
Hive QA commented on HIVE-14202:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12817005/HIVE-14202.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10296 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/452/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/452/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-452/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12817005 - PreCommit-HIVE-MASTER-Build
> Change tez version used to 0.8.4
>
>
> Key: HIVE-14202
> URL: https://issues.apache.org/jira/browse/HIVE-14202
> Project: Hive
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14202.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
