[jira] [Updated] (HIVE-26685) Improve Path name escaping / unescaping performance
[ https://issues.apache.org/jira/browse/HIVE-26685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-26685: - Assignee: James Petty Resolution: Fixed Status: Resolved (was: Patch Available) > Improve Path name escaping / unescaping performance > --- > > Key: HIVE-26685 > URL: https://issues.apache.org/jira/browse/HIVE-26685 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: All Versions >Reporter: James Petty >Assignee: James Petty >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-26685.1.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > When escaping / unescaping partition path part names, the existing logic > incurs significant avoidable overhead by copying each character sequentially > into a new StringBuilder even when no escaping/unescaping is necessary as > well as using String.format to escape characters inside of the inner loop. > > The included patch to improve the performance of these operations refactors > two static method implementations, but requires no external API surface or > user-visible behavior changes. This change is applicable and portable to a > wide range of Hive versions from branch-0.6 onward when the initial method > implementations were added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26685) Improve Path name escaping / unescaping performance
[ https://issues.apache.org/jira/browse/HIVE-26685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-26685: - Fix Version/s: 4.0.0 > Improve Path name escaping / unescaping performance > --- > > Key: HIVE-26685 > URL: https://issues.apache.org/jira/browse/HIVE-26685 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: All Versions >Reporter: James Petty >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-26685.1.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > When escaping / unescaping partition path part names, the existing logic > incurs significant avoidable overhead by copying each character sequentially > into a new StringBuilder even when no escaping/unescaping is necessary as > well as using String.format to escape characters inside of the inner loop. > > The included patch to improve the performance of these operations refactors > two static method implementations, but requires no external API surface or > user-visible behavior changes. This change is applicable and portable to a > wide range of Hive versions from branch-0.6 onward when the initial method > implementations were added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26179) In tez reuse container mode, asyncInitOperations are not clear.
[
https://issues.apache.org/jira/browse/HIVE-26179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528865#comment-17528865
]
Wei Zheng commented on HIVE-26179:
--
+1 Thanks for fixing this
> In tez reuse container mode, asyncInitOperations are not clear.
> ---
>
> Key: HIVE-26179
> URL: https://issues.apache.org/jira/browse/HIVE-26179
> Project: Hive
> Issue Type: Bug
> Components: Hive, Tez
>Affects Versions: 1.2.1
> Environment: engine: Tez (Note: tez.am.container.reuse.enabled is
> true)
>
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In our cluster, we found error like this.
> {code:java}
> Vertex failed, vertexName=Map 1, vertexId=vertex_1650608671415_321290_1_11,
> diagnostics=[Task failed, taskId=task_1650608671415_321290_1_11_000422,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1650608671415_321290_1_11_000422_0:java.lang.RuntimeException:
> java.lang.RuntimeException: Hive Runtime Error while closing operators
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:135)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: Hive Runtime Error while closing
> operators
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:349)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:161)
> ... 16 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:488)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:684)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:698)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:338)
> ... 17 more
> {code}
> When tez reuse container is enable, and use MapJoinOperator, if same tasks's
> different taskattemp execute in same container, will throw NPE.
> By my debug, I found the second task attempt use first task's
> asyncInitOperations. asyncInitOperations are not clear when close op, then
> second taskattemp may use first taskattepmt's mapJoinTables which
> HybridHashTableContainer.HashPartition is closed, so throw NPE.
> We must clear asyncInitOperations when op is closed.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (HIVE-20858) Serializer is not correctly initialized with configuration in Utilities.createEmptyBuckets()
[ https://issues.apache.org/jira/browse/HIVE-20858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-20858: - Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Thanks Daniel. Committed. > Serializer is not correctly initialized with configuration in > Utilities.createEmptyBuckets() > > > Key: HIVE-20858 > URL: https://issues.apache.org/jira/browse/HIVE-20858 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20858.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20858) Serializer is not correctly initialized with configuration in Utilities.createEmptyBuckets()
[ https://issues.apache.org/jira/browse/HIVE-20858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675919#comment-16675919 ] Wei Zheng commented on HIVE-20858: -- Nothing specific. I discovered this bug when reviewing another ticket HIVE-9651. They suffer the same initialization issue. > Serializer is not correctly initialized with configuration in > Utilities.createEmptyBuckets() > > > Key: HIVE-20858 > URL: https://issues.apache.org/jira/browse/HIVE-20858 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng >Priority: Major > Attachments: HIVE-20858.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20858) Serializer is not correctly initialized with configuration in Utilities.createEmptyBuckets()
[ https://issues.apache.org/jira/browse/HIVE-20858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675491#comment-16675491 ] Wei Zheng commented on HIVE-20858: -- [~daijy] Can you take a look please :) > Serializer is not correctly initialized with configuration in > Utilities.createEmptyBuckets() > > > Key: HIVE-20858 > URL: https://issues.apache.org/jira/browse/HIVE-20858 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng >Priority: Major > Attachments: HIVE-20858.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20858) Serializer is not correctly initialized with configuration in Utilities.createEmptyBuckets()
[ https://issues.apache.org/jira/browse/HIVE-20858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-20858: - Status: Patch Available (was: Open) > Serializer is not correctly initialized with configuration in > Utilities.createEmptyBuckets() > > > Key: HIVE-20858 > URL: https://issues.apache.org/jira/browse/HIVE-20858 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng >Priority: Major > Attachments: HIVE-20858.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20858) Serializer is not correctly initialized with configuration in Utilities.createEmptyBuckets()
[ https://issues.apache.org/jira/browse/HIVE-20858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-20858: > Serializer is not correctly initialized with configuration in > Utilities.createEmptyBuckets() > > > Key: HIVE-20858 > URL: https://issues.apache.org/jira/browse/HIVE-20858 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng >Priority: Major > Attachments: HIVE-20858.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20858) Serializer is not correctly initialized with configuration in Utilities.createEmptyBuckets()
[ https://issues.apache.org/jira/browse/HIVE-20858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-20858: - Attachment: HIVE-20858.1.patch > Serializer is not correctly initialized with configuration in > Utilities.createEmptyBuckets() > > > Key: HIVE-20858 > URL: https://issues.apache.org/jira/browse/HIVE-20858 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng >Priority: Major > Attachments: HIVE-20858.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663139#comment-16663139
]
Wei Zheng commented on HIVE-20786:
--
I also came across this issue today. Opened HIVE-20800 without noticing this
ticket. Thanks Vihang.
Here's what I found:
[https://maven.apache.org/plugins/maven-assembly-plugin/faq.html#tarFileModes]
We do need to make sure it's "posix" since it's better than "gnu". And we need
the change for packaging/pom.xml as well.
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Assignee: Szehon Ho
>Priority: Major
> Labels: maven
> Attachments: HIVE-20786.patch, hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator
[jira] [Commented] (HIVE-20800) Use "posix" for property tarLongFileMode for maven-assembly-plugin
[
https://issues.apache.org/jira/browse/HIVE-20800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663135#comment-16663135
]
Wei Zheng commented on HIVE-20800:
--
[~vihangk1] Indeed. I will close this one as dup.
> Use "posix" for property tarLongFileMode for maven-assembly-plugin
> --
>
> Key: HIVE-20800
> URL: https://issues.apache.org/jira/browse/HIVE-20800
> Project: Hive
> Issue Type: Improvement
> Components: Build Infrastructure
>Affects Versions: 3.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20800.1.patch
>
>
> Came across this error when building hive using "mvn clean install
> -DskipTests"
> {code}
> [INFO] Building tar:
> /Users/wei/apache/hive/standalone-metastore/target/apache-hive-standalone-metastore-4.0.0-SNAPSHOT-src.tar.gz
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.656
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.779
> s]
> [INFO] Hive Classifications ... SUCCESS [ 0.908
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.217
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 7.102
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.069
> s]
> [INFO] Hive Shims . SUCCESS [ 1.905
> s]
> [INFO] Hive Common SUCCESS [ 8.185
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.603
> s]
> [INFO] Hive Serde . SUCCESS [ 7.438
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.576
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging . SKIPPED
> [INFO] Hive Metastore Tools ... SKIPPED
> [INFO] Hive Metastore Tools common libraries .. SKIPPED
> [INFO] Hive metastore benchmarks .. SKIPPED
> [INFO] Hive Upgrade Acid .. SKIPPED
> [INFO] Hive Pre Upgrade Acid 4.0.0-SNAPSHOT ... SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
[jira] [Updated] (HIVE-20800) Use "posix" for property tarLongFileMode for maven-assembly-plugin
[
https://issues.apache.org/jira/browse/HIVE-20800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-20800:
-
Resolution: Duplicate
Status: Resolved (was: Patch Available)
> Use "posix" for property tarLongFileMode for maven-assembly-plugin
> --
>
> Key: HIVE-20800
> URL: https://issues.apache.org/jira/browse/HIVE-20800
> Project: Hive
> Issue Type: Improvement
> Components: Build Infrastructure
>Affects Versions: 3.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20800.1.patch
>
>
> Came across this error when building hive using "mvn clean install
> -DskipTests"
> {code}
> [INFO] Building tar:
> /Users/wei/apache/hive/standalone-metastore/target/apache-hive-standalone-metastore-4.0.0-SNAPSHOT-src.tar.gz
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.656
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.779
> s]
> [INFO] Hive Classifications ... SUCCESS [ 0.908
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.217
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 7.102
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.069
> s]
> [INFO] Hive Shims . SUCCESS [ 1.905
> s]
> [INFO] Hive Common SUCCESS [ 8.185
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.603
> s]
> [INFO] Hive Serde . SUCCESS [ 7.438
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.576
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging . SKIPPED
> [INFO] Hive Metastore Tools ... SKIPPED
> [INFO] Hive Metastore Tools common libraries .. SKIPPED
> [INFO] Hive metastore benchmarks .. SKIPPED
> [INFO] Hive Upgrade Acid .. SKIPPED
> [INFO] Hive Pre Upgrade Acid 4.0.0-SNAPSHOT ... SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
[jira] [Updated] (HIVE-20800) Use "posix" for property tarLongFileMode for maven-assembly-plugin
[
https://issues.apache.org/jira/browse/HIVE-20800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-20800:
-
Status: Patch Available (was: Open)
> Use "posix" for property tarLongFileMode for maven-assembly-plugin
> --
>
> Key: HIVE-20800
> URL: https://issues.apache.org/jira/browse/HIVE-20800
> Project: Hive
> Issue Type: Improvement
> Components: Build Infrastructure
>Affects Versions: 3.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20800.1.patch
>
>
> Came across this error when building hive using "mvn clean install
> -DskipTests"
> {code}
> [INFO] Building tar:
> /Users/wei/apache/hive/standalone-metastore/target/apache-hive-standalone-metastore-4.0.0-SNAPSHOT-src.tar.gz
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.656
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.779
> s]
> [INFO] Hive Classifications ... SUCCESS [ 0.908
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.217
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 7.102
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.069
> s]
> [INFO] Hive Shims . SUCCESS [ 1.905
> s]
> [INFO] Hive Common SUCCESS [ 8.185
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.603
> s]
> [INFO] Hive Serde . SUCCESS [ 7.438
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.576
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging . SKIPPED
> [INFO] Hive Metastore Tools ... SKIPPED
> [INFO] Hive Metastore Tools common libraries .. SKIPPED
> [INFO] Hive metastore benchmarks .. SKIPPED
> [INFO] Hive Upgrade Acid .. SKIPPED
> [INFO] Hive Pre Upgrade Acid 4.0.0-SNAPSHOT ... SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
>
[jira] [Updated] (HIVE-20800) Use "posix" for property tarLongFileMode for maven-assembly-plugin
[
https://issues.apache.org/jira/browse/HIVE-20800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-20800:
-
Attachment: HIVE-20800.1.patch
> Use "posix" for property tarLongFileMode for maven-assembly-plugin
> --
>
> Key: HIVE-20800
> URL: https://issues.apache.org/jira/browse/HIVE-20800
> Project: Hive
> Issue Type: Improvement
> Components: Build Infrastructure
>Affects Versions: 3.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20800.1.patch
>
>
> Came across this error when building hive using "mvn clean install
> -DskipTests"
> {code}
> [INFO] Building tar:
> /Users/wei/apache/hive/standalone-metastore/target/apache-hive-standalone-metastore-4.0.0-SNAPSHOT-src.tar.gz
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.656
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.779
> s]
> [INFO] Hive Classifications ... SUCCESS [ 0.908
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.217
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 7.102
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.069
> s]
> [INFO] Hive Shims . SUCCESS [ 1.905
> s]
> [INFO] Hive Common SUCCESS [ 8.185
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.603
> s]
> [INFO] Hive Serde . SUCCESS [ 7.438
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.576
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging . SKIPPED
> [INFO] Hive Metastore Tools ... SKIPPED
> [INFO] Hive Metastore Tools common libraries .. SKIPPED
> [INFO] Hive metastore benchmarks .. SKIPPED
> [INFO] Hive Upgrade Acid .. SKIPPED
> [INFO] Hive Pre Upgrade Acid 4.0.0-SNAPSHOT ... SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO]
[jira] [Assigned] (HIVE-20800) Use "posix" for property tarLongFileMode for maven-assembly-plugin
[
https://issues.apache.org/jira/browse/HIVE-20800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng reassigned HIVE-20800:
> Use "posix" for property tarLongFileMode for maven-assembly-plugin
> --
>
> Key: HIVE-20800
> URL: https://issues.apache.org/jira/browse/HIVE-20800
> Project: Hive
> Issue Type: Improvement
> Components: Build Infrastructure
>Affects Versions: 3.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>Priority: Major
> Fix For: 4.0.0
>
>
> Came across this error when building hive using "mvn clean install
> -DskipTests"
> {code}
> [INFO] Building tar:
> /Users/wei/apache/hive/standalone-metastore/target/apache-hive-standalone-metastore-4.0.0-SNAPSHOT-src.tar.gz
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.656
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.779
> s]
> [INFO] Hive Classifications ... SUCCESS [ 0.908
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.217
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 7.102
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.069
> s]
> [INFO] Hive Shims . SUCCESS [ 1.905
> s]
> [INFO] Hive Common SUCCESS [ 8.185
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.603
> s]
> [INFO] Hive Serde . SUCCESS [ 7.438
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.576
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging . SKIPPED
> [INFO] Hive Metastore Tools ... SKIPPED
> [INFO] Hive Metastore Tools common libraries .. SKIPPED
> [INFO] Hive metastore benchmarks .. SKIPPED
> [INFO] Hive Upgrade Acid .. SKIPPED
> [INFO] Hive Pre Upgrade Acid 4.0.0-SNAPSHOT ... SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 42.026 s
> [INFO] Finished at: 2018-10-24T15:34:40-07:00
>
[jira] [Commented] (HIVE-17783) Hybrid Grace Hash Join has performance degradation for N-way join using Hive on Tez
[ https://issues.apache.org/jira/browse/HIVE-17783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225678#comment-16225678 ] Wei Zheng commented on HIVE-17783: -- [~Ferd] Sorry for the late reply. Yes the spilling part is the bottleneck and there's no easy way to get around it. In your case for the n-way joins, the optimizer stats estimation may not be accurate which makes the situation worse. Anyway, the ultimate way to solve this problem is to have a reliable memory manager which can provide memory usage/quota at any moment. Right now we're following a conservative approach, which is to use a soft (possibly inaccurate) memory limit. That way we can avoid unnecessary spilling if there is enough memory for loading the hashtable. > Hybrid Grace Hash Join has performance degradation for N-way join using Hive > on Tez > --- > > Key: HIVE-17783 > URL: https://issues.apache.org/jira/browse/HIVE-17783 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 > Environment: 8*Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz > 1 master + 7 workers > TPC-DS at 3TB data scales > Hive version : 2.2.0 >Reporter: Ferdinand Xu > Attachments: Hybrid_Grace_Hash_Join.xlsx, screenshot-1.png > > > Most configurations are using default value. And the benchmark is to test > enabling against disabling hybrid grace hash join using TPC-DS queries at 3TB > data scales. Many queries related to N-way join has performance degradation > over three times test. Detailed result is attached. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202268#comment-16202268 ] Wei Zheng commented on HIVE-15212: -- +1 Thanks Sergey for putting it together! > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, > HIVE-15212.09.patch, HIVE-15212.10.patch, HIVE-15212.11.patch, > HIVE-15212.12.patch, HIVE-15212.12.patch, HIVE-15212.13.patch, > HIVE-15212.13.patch, HIVE-15212.14.patch, HIVE-15212.15.patch, > HIVE-15212.16.patch, HIVE-15212.17.patch, HIVE-15212.18.patch, > HIVE-15212.19.patch, HIVE-15212.20.patch, HIVE-15212.21.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15212) merge branch into master
[
https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179929#comment-16179929
]
Wei Zheng commented on HIVE-15212:
--
[~ekoifman] You at'ed the wrong person ;)
Sorry for the late update. I left a todo comment in
HiveInputFormat.java:processForWriteIds()
{code}
// todo for IOW, we also need to count in base dir, if any
for (AcidUtils.ParsedDelta delta : dirInfo.getCurrentDirectories())
{
Utilities.LOG14535.info("Adding input " + delta.getPath());
finalPaths.add(delta.getPath());
}
{code}
Here we just need to count in base dir if any.
> merge branch into master
>
>
> Key: HIVE-15212
> URL: https://issues.apache.org/jira/browse/HIVE-15212
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch,
> HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch,
> HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch,
> HIVE-15212.09.patch, HIVE-15212.10.patch, HIVE-15212.11.patch,
> HIVE-15212.12.patch, HIVE-15212.12.patch, HIVE-15212.13.patch,
> HIVE-15212.13.patch, HIVE-15212.14.patch, HIVE-15212.15.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15212: - Attachment: HIVE-15212.10.patch patch 10 for test > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, > HIVE-15212.09.patch, HIVE-15212.10.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158955#comment-16158955 ] Wei Zheng commented on HIVE-15212: -- [~sershe] [~ekoifman] Patch 9 is the latest diff'ed patch between master and hive-14535 branch. Looks like there are some relevant test failures as well as many related to replication. I will try to fix those non-replication related failures today. It's strange that the test run didn't generate a test report although it sent the result here to the JIRA. > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, > HIVE-15212.09.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15212: - Attachment: HIVE-15212.09.patch Upload patch 9 for testing > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, > HIVE-15212.09.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17460) `insert overwrite` should support table schema evolution (e.g. add columns)
[ https://issues.apache.org/jira/browse/HIVE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156423#comment-16156423 ] Wei Zheng commented on HIVE-17460: -- [~debugger87] I discussed with Matt regarding this issue as he is the domain expert for schema evolution. He's saying you can achieve what you want by adding CASCADE in your DDL. > `insert overwrite` should support table schema evolution (e.g. add columns) > --- > > Key: HIVE-17460 > URL: https://issues.apache.org/jira/browse/HIVE-17460 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.2.0 >Reporter: Chaozhong Yang >Assignee: Chaozhong Yang > Fix For: 3.0.0 > > Attachments: HIVE-17460.2.patch, HIVE-17460.patch > > > In Hive, adding columns into original table is a common use case. However, if > we insert overwrite older partitions after adding columns, added columns will > not be accessed. > ``` > create table src_table( > i int > ) > PARTITIONED BY (`date` string); > insert overwrite table src_table partition(`date`='20170905') valu > es (3); > select * from src_table where `date` = '20170905'; > alter table src_table add columns (bi bigint); > insert overwrite table src_table partition(`date`='20170905') valu > es (3, 5); > select * from src_table where `date` = '20170905'; > ``` > The result will be as follows: > ``` > 3, NULL, '20170905' > ``` > Obviously, it doesn't meet our expectation. The expected result should be: > ``` > 3, 5, '20170905' > ``` -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17460) `insert overwrite` should support table schema evolution (e.g. add columns)
[ https://issues.apache.org/jira/browse/HIVE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155845#comment-16155845 ] Wei Zheng commented on HIVE-17460: -- Some existing q.out files are wrong, but I noticed some other failures, e.g. autoColumnStats_5.q. I suggest you try moving the fix into alterPartitionSpecInMemory, under the "if (inheritTableSpecs)" block and have another test run. > `insert overwrite` should support table schema evolution (e.g. add columns) > --- > > Key: HIVE-17460 > URL: https://issues.apache.org/jira/browse/HIVE-17460 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.2.0 >Reporter: Chaozhong Yang >Assignee: Chaozhong Yang > Fix For: 3.0.0 > > Attachments: HIVE-17460.2.patch, HIVE-17460.patch > > > In Hive, adding columns into original table is a common use case. However, if > we insert overwrite older partitions after adding columns, added columns will > not be accessed. > ``` > create table src_table( > i int > ) > PARTITIONED BY (`date` string); > insert overwrite table src_table partition(`date`='20170905') valu > es (3); > select * from src_table where `date` = '20170905'; > alter table src_table add columns (bi bigint); > insert overwrite table src_table partition(`date`='20170905') valu > es (3, 5); > select * from src_table where `date` = '20170905'; > ``` > The result will be as follows: > ``` > 3, NULL, '20170905' > ``` > Obviously, it doesn't meet our expectation. The expected result should be: > ``` > 3, 5, '20170905' > ``` -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17460) `insert overwrite` should support table schema evolution (e.g. add columns)
[ https://issues.apache.org/jira/browse/HIVE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155616#comment-16155616 ] Wei Zheng commented on HIVE-17460: -- [~debugger87] Your fix looks good. Would it be better to move it inside alterPartitionSpecInMemory()? > `insert overwrite` should support table schema evolution (e.g. add columns) > --- > > Key: HIVE-17460 > URL: https://issues.apache.org/jira/browse/HIVE-17460 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.2.0 >Reporter: Chaozhong Yang >Assignee: Chaozhong Yang > Fix For: 3.0.0 > > Attachments: HIVE-17460.patch > > > In Hive, adding columns into original table is a common use case. However, if > we insert overwrite older partitions after adding columns, added columns will > not be accessed. > ``` > create table src_table( > i int > ) > PARTITIONED BY (`date` string); > insert overwrite table src_table partition(`date`='20170905') valu > es (3); > select * from src_table where `date` = '20170905'; > alter table src_table add columns (bi bigint); > insert overwrite table src_table partition(`date`='20170905') valu > es (3, 5); > select * from src_table where `date` = '20170905'; > ``` > The result will be as follows: > ``` > 3, NULL, '20170905' > ``` > Obviously, it doesn't meet our expectation. The expected result should be: > ``` > 3, 5, '20170905' > ``` -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154142#comment-16154142 ] Wei Zheng commented on HIVE-17361: -- Sure. Attached RB. > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-17361.1.patch, HIVE-17361.2.patch, > HIVE-17361.3.patch, HIVE-17361.4.patch > > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150048#comment-16150048 ] Wei Zheng commented on HIVE-17361: -- [~ekoifman] Can you take another look? > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-17361.1.patch, HIVE-17361.2.patch, > HIVE-17361.3.patch, HIVE-17361.4.patch > > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-17361: - Attachment: HIVE-17361.4.patch The build bot failed for patch 3. Made an identical patch 4 to trigger precommit test. > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-17361.1.patch, HIVE-17361.2.patch, > HIVE-17361.3.patch, HIVE-17361.4.patch > > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-17361: - Attachment: HIVE-17361.3.patch > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-17361.1.patch, HIVE-17361.2.patch, > HIVE-17361.3.patch > > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-17361: - Attachment: HIVE-17361.2.patch patch 2 with a different approach > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-17361.1.patch, HIVE-17361.2.patch > > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17205) add functional support
[ https://issues.apache.org/jira/browse/HIVE-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142453#comment-16142453 ] Wei Zheng commented on HIVE-17205: -- The patch looks good. +1 Thanks Eugene for relaxing the bucketing restriction! > add functional support > -- > > Key: HIVE-17205 > URL: https://issues.apache.org/jira/browse/HIVE-17205 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-17205.01.patch, HIVE-17205.02.patch, > HIVE-17205.03.patch, HIVE-17205.09.patch, HIVE-17205.10.patch, > HIVE-17205.11.patch, HIVE-17205.12.patch, HIVE-17205.13.patch, > HIVE-17205.14.patch, HIVE-17205.15.patch, HIVE-17205.16.patch > > > make sure unbucketed tables can be marked transactional=true > make insert/update/delete/compaction work -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17340) TxnHandler.checkLock() - reduce number of SQL statements
[
https://issues.apache.org/jira/browse/HIVE-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142023#comment-16142023
]
Wei Zheng commented on HIVE-17340:
--
Maybe it's better to move {code}dbConn.commit(){code} into the acquire() method
from checkLock().
nit: there's unused imports.
+1 otherwise.
> TxnHandler.checkLock() - reduce number of SQL statements
>
>
> Key: HIVE-17340
> URL: https://issues.apache.org/jira/browse/HIVE-17340
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-17340.03.patch
>
>
> This calls acquire(Connection dbConn, Statement stmt, long extLockId,
> LockInfo lockInfo)
> for each lock in the same DB transaction - 1 Update stmt per acquire().
> There is no reason all of them cannot be sent in 1 statement if all the locks
> are granted
> With a lot of partitions this can be a perf issue
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17341) DbTxnManger.startHeartbeat() - randomize initial delay
[ https://issues.apache.org/jira/browse/HIVE-17341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141949#comment-16141949 ] Wei Zheng commented on HIVE-17341: -- +1 > DbTxnManger.startHeartbeat() - randomize initial delay > -- > > Key: HIVE-17341 > URL: https://issues.apache.org/jira/browse/HIVE-17341 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-17341.01.patch > > > This sets up a fixed delay for all heartebeats. If many queries land on the > server at the same time, > they will wake up and start hearbeating at the same time causing a bottleneck. > Add some random element to heatbeat delay. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140387#comment-16140387 ] Wei Zheng commented on HIVE-17361: -- [~ekoifman] Can you take a look please? > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-17361.1.patch > > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17361) Support LOAD DATA for transactional tables
[
https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139203#comment-16139203
]
Wei Zheng commented on HIVE-17361:
--
{code}
LOAD DATA [LOCAL] INPATH 'filepath' [OVERWRITE] INTO TABLE tablename [PARTITION
(partcol1=val1, partcol2=val2 ...)]
{code}
Unlike non-ACID table, if the table is bucketed and there are more than 1
bucket file, then LOAD DATA on ACID table will require 'filepath' to refer to a
directory not a file. Otherwise, one may end up having a bucket file in one
load_delta directory and another bucket file in a different load_delta
directory.
The reason behind this is:
a) For a non-ACID table, say tbl1, one can continue loading files into the same
table via consecutive LOAD commands, that will just result more and more files
under tbl1/ directory
b) However, for a non-ACID table, since a new load_delta directory will be
created every time when LOAD DATA is run, consecutive LOAD commands will create
separate subdirectories for every single file, which may not be desirable, e.g.
if one wants to load a file for one bucket, and then a file for another bucket,
those two files will reside in two different load_delta directories.
> Support LOAD DATA for transactional tables
> --
>
> Key: HIVE-17361
> URL: https://issues.apache.org/jira/browse/HIVE-17361
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-17361.1.patch
>
>
> LOAD DATA was not supported since ACID was introduced. Need to fill this gap
> between ACID table and regular hive table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-17361: - Attachment: HIVE-17361.1.patch > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-17361.1.patch > > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-17361: - Status: Patch Available (was: Open) > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-17361.1.patch > > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-17361: > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-14988) Support INSERT OVERWRITE into a partition on transactional tables
[ https://issues.apache.org/jira/browse/HIVE-14988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091109#comment-16091109 ] Wei Zheng commented on HIVE-14988: -- Sure of course :) Thank you! > Support INSERT OVERWRITE into a partition on transactional tables > - > > Key: HIVE-14988 > URL: https://issues.apache.org/jira/browse/HIVE-14988 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Fix For: 3.0.0 > > Attachments: HIVE-14988.01.patch, HIVE-14988.02.patch, > HIVE-14988.03.patch, HIVE-14988.04.patch, HIVE-14988.05.patch, > HIVE-14988.06.patch > > > Insert overwrite operation on transactional table will currently raise an > error. > This can/should be supported -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-14988) Support INSERT OVERWRITE into a partition on transactional tables
[
https://issues.apache.org/jira/browse/HIVE-14988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091101#comment-16091101
]
Wei Zheng commented on HIVE-14988:
--
[~leftylev] Thanks for catching that. INSERT OVERWRITE is not a new feature, so
I don't think we need to highlight this. But we do need to remove one bullet
item under here:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Synopsis.1
{code}
"As of Hive 0.14, if a table has an OutputFormat that implements
AcidOutputFormat and the system is configured to use a transaction manager that
implements ACID, then INSERT OVERWRITE will be disabled for that table. This
is to avoid users unintentionally overwriting transaction history. The same
functionality can be achieved by using TRUNCATE TABLE (for non-partitioned
tables) or DROP PARTITION followed by INSERT INTO."
{code}
> Support INSERT OVERWRITE into a partition on transactional tables
> -
>
> Key: HIVE-14988
> URL: https://issues.apache.org/jira/browse/HIVE-14988
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
> Fix For: 3.0.0
>
> Attachments: HIVE-14988.01.patch, HIVE-14988.02.patch,
> HIVE-14988.03.patch, HIVE-14988.04.patch, HIVE-14988.05.patch,
> HIVE-14988.06.patch
>
>
> Insert overwrite operation on transactional table will currently raise an
> error.
> This can/should be supported
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-14988) Support INSERT OVERWRITE into a partition on transactional tables
[ https://issues.apache.org/jira/browse/HIVE-14988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14988: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed patch 6 to master. Thanks Eugene for the review. > Support INSERT OVERWRITE into a partition on transactional tables > - > > Key: HIVE-14988 > URL: https://issues.apache.org/jira/browse/HIVE-14988 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Fix For: 3.0.0 > > Attachments: HIVE-14988.01.patch, HIVE-14988.02.patch, > HIVE-14988.03.patch, HIVE-14988.04.patch, HIVE-14988.05.patch, > HIVE-14988.06.patch > > > Insert overwrite operation on transactional table will currently raise an > error. > This can/should be supported -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-14688) Hive drop call fails in presence of TDE
[
https://issues.apache.org/jira/browse/HIVE-14688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075976#comment-16075976
]
Wei Zheng commented on HIVE-14688:
--
[~thejas] Looks like HIVE-11418 was recently committed to master which is
solving the same problem. So this ticket becomes duplicate of HIVE-11418. It
won't work for older 2.x hadoop versions, as was also discussed in HIVE-11418.
> Hive drop call fails in presence of TDE
> ---
>
> Key: HIVE-14688
> URL: https://issues.apache.org/jira/browse/HIVE-14688
> Project: Hive
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Deepesh Khandelwal
>Assignee: Wei Zheng
> Attachments: HIVE-14688.1.patch, HIVE-14688.2.patch,
> HIVE-14688.3.patch, HIVE-14688.4.patch
>
>
> This should be committed to when Hive moves to Hadoop 2.8
> In Hadoop 2.8.0 TDE trash collection was fixed through HDFS-8831. This
> enables us to make drop table calls for Hive managed tables where Hive
> metastore warehouse directory is in encrypted zone. However even with the
> feature in HDFS, Hive drop table currently fail:
> {noformat}
> $ hdfs crypto -listZones
> /apps/hive/warehouse key2
> $ hdfs dfs -ls /apps/hive/warehouse
> Found 1 items
> drwxrwxrwt - hdfs hdfs 0 2016-09-01 02:54
> /apps/hive/warehouse/.Trash
> hive> create table abc(a string, b int);
> OK
> Time taken: 5.538 seconds
> hive> dfs -ls /apps/hive/warehouse;
> Found 2 items
> drwxrwxrwt - hdfs hdfs 0 2016-09-01 02:54
> /apps/hive/warehouse/.Trash
> drwxrwxrwx - deepesh hdfs 0 2016-09-01 17:15
> /apps/hive/warehouse/abc
> hive> drop table if exists abc;
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Unable to drop
> default.abc because it is in an encryption zone and trash is enabled. Use
> PURGE option to skip trash.)
> {noformat}
> The problem lies here:
> {code:title=metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java}
> private void checkTrashPurgeCombination(Path pathToData, String objectName,
> boolean ifPurge)
> ...
> if (trashEnabled) {
> try {
> HadoopShims.HdfsEncryptionShim shim =
>
> ShimLoader.getHadoopShims().createHdfsEncryptionShim(FileSystem.get(hiveConf),
> hiveConf);
> if (shim.isPathEncrypted(pathToData)) {
> throw new MetaException("Unable to drop " + objectName + "
> because it is in an encryption zone" +
> " and trash is enabled. Use PURGE option to skip trash.");
> }
> } catch (IOException ex) {
> MetaException e = new MetaException(ex.getMessage());
> e.initCause(ex);
> throw e;
> }
> }
> {code}
> As we can see that we are making an assumption that delete wouldn't be
> successful in encrypted zone. We need to modify this logic.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16972) FetchOperator: filter out inputSplits which length is zero
[
https://issues.apache.org/jira/browse/HIVE-16972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069525#comment-16069525
]
Wei Zheng commented on HIVE-16972:
--
[~debugger87] patch 3 caused many test failures. Can you take a look and run
the test again?
> FetchOperator: filter out inputSplits which length is zero
> --
>
> Key: HIVE-16972
> URL: https://issues.apache.org/jira/browse/HIVE-16972
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.1.0, 2.1.1
>Reporter: Chaozhong Yang
>Assignee: Chaozhong Yang
> Attachments: HIVE-16972.2.patch, HIVE-16972.3.patch, HIVE-16972.patch
>
>
> * Background
>We can describe the basic work flow of common HQL query as follows:
> 1. compile and execute
> 2. fetch results
> In many cases, we don't need to worry about the issues fetching results
> from HDFS(iff there are mapreduce jobs generated in planning step). However,
> the number of results files on HDFS and data distribution will affect the
> final status of HQL query, especially for HiveServer2. We have some map-only
> queries, e.g:
> {code:sql}
> select * from myTable where date > '20170101' and date <= '20170301' and id =
> 88;
> {code}
> This query will generate more than 20,000 files(look at screenshot image
> uploaded) on HDFS and most of those files are empty. Of course, they are very
> sparse. If we send TFetchResultsRequest from HiveServer2 client with some
> parameters(timeout:90s, maxRows:1024) , FetchOperator can not fetch 1024 rows
> in 90 seconds and our HiveServer2 client will mark this TFetchResultsRequest
> as timed out failure. Why? In fact, It's expensive to fetch results from
> empty file. In our HDFS cluster( 5000+ DataNodes) , reading data from an
> empty file will cost almost 100 ms (100ms * 1000 ==> 100s > 90s timeout).
> Obviously, we can filter out those empty files or splits to speed up the
> process of FetchResults.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-14988) Support INSERT OVERWRITE into a partition on transactional tables
[ https://issues.apache.org/jira/browse/HIVE-14988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069233#comment-16069233 ] Wei Zheng commented on HIVE-14988: -- The 4 failures with age==1 are not relevant org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite] org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed] [~ekoifman] Can you take another look please? > Support INSERT OVERWRITE into a partition on transactional tables > - > > Key: HIVE-14988 > URL: https://issues.apache.org/jira/browse/HIVE-14988 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-14988.01.patch, HIVE-14988.02.patch, > HIVE-14988.03.patch, HIVE-14988.04.patch, HIVE-14988.05.patch, > HIVE-14988.06.patch > > > Insert overwrite operation on transactional table will currently raise an > error. > This can/should be supported -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-14988) Support INSERT OVERWRITE into a partition on transactional tables
[ https://issues.apache.org/jira/browse/HIVE-14988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14988: - Attachment: HIVE-14988.06.patch > Support INSERT OVERWRITE into a partition on transactional tables > - > > Key: HIVE-14988 > URL: https://issues.apache.org/jira/browse/HIVE-14988 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-14988.01.patch, HIVE-14988.02.patch, > HIVE-14988.03.patch, HIVE-14988.04.patch, HIVE-14988.05.patch, > HIVE-14988.06.patch > > > Insert overwrite operation on transactional table will currently raise an > error. > This can/should be supported -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16972) FetchOperator: filter out inputSplits which length is zero
[
https://issues.apache.org/jira/browse/HIVE-16972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065520#comment-16065520
]
Wei Zheng commented on HIVE-16972:
--
patch looks good to me. Although the change can be simplified a little bit :)
I'm not sure why the picture was picked up for the precommit test. You may want
to remove/rename it to trigger the test again.
> FetchOperator: filter out inputSplits which length is zero
> --
>
> Key: HIVE-16972
> URL: https://issues.apache.org/jira/browse/HIVE-16972
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.1.0, 2.1.1
>Reporter: Chaozhong Yang
>Assignee: Chaozhong Yang
> Fix For: 2.1.2
>
> Attachments: HIVE-16972.patch, screenshot-1.png
>
>
> * Background
>We can describe the basic work flow of common HQL query as follows:
> 1. compile and execute
> 2. fetch results
> In many cases, we don't need to worry about the issues fetching results
> from HDFS(iff there are mapreduce jobs generated in planning step). However,
> the number of results files on HDFS and data distribution will affect the
> final status of HQL query, especially for HiveServer2. We have some map-only
> queries, e.g:
> {code:sql}
> select * from myTable where date > '20170101' and date <= '20170301' and id =
> 88;
> {code}
> This query will generate more than 20,000 files(look at screenshot image
> uploaded) on HDFS and most of those files are empty. Of course, they are very
> sparse. If we send TFetchResultsRequest from HiveServer2 client with some
> parameters(timeout:90s, maxRows:1024) , FetchOperator can not fetch 1024 rows
> in 90 seconds and our HiveServer2 client will mark this TFetchResultsRequest
> as timed out failure. Why? In fact, It's expensive to fetch results from
> empty file. In our HDFS cluster( 5000+ DataNodes) , reading data from an
> empty file will cost almost 100 ms (100ms * 1000 ==> 100s > 90s timeout).
> Obviously, we can filter out those empty files or splits to speed up the
> process of FetchResults.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (HIVE-16963) rely on AcidUtils.getAcidState() for read path
[ https://issues.apache.org/jira/browse/HIVE-16963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng resolved HIVE-16963. -- Resolution: Fixed Fix Version/s: hive-14535 > rely on AcidUtils.getAcidState() for read path > -- > > Key: HIVE-16963 > URL: https://issues.apache.org/jira/browse/HIVE-16963 > Project: Hive > Issue Type: Sub-task >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-16963.patch > > > This is to make MM table more consistent to full ACID table. Also it's a > prerequisite for Insert Overwrite support for MM table (refer to HIVE-14988). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16963) rely on AcidUtils.getAcidState() for read path
[ https://issues.apache.org/jira/browse/HIVE-16963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16963: - Attachment: HIVE-16963.patch The patch also fixed an issue for Import > rely on AcidUtils.getAcidState() for read path > -- > > Key: HIVE-16963 > URL: https://issues.apache.org/jira/browse/HIVE-16963 > Project: Hive > Issue Type: Sub-task >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-16963.patch > > > This is to make MM table more consistent to full ACID table. Also it's a > prerequisite for Insert Overwrite support for MM table (refer to HIVE-14988). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-16963) rely on AcidUtils.getAcidState() for read path
[ https://issues.apache.org/jira/browse/HIVE-16963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-16963: > rely on AcidUtils.getAcidState() for read path > -- > > Key: HIVE-16963 > URL: https://issues.apache.org/jira/browse/HIVE-16963 > Project: Hive > Issue Type: Sub-task >Reporter: Wei Zheng >Assignee: Wei Zheng > > This is to make MM table more consistent to full ACID table. Also it's a > prerequisite for Insert Overwrite support for MM table (refer to HIVE-14988). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-14988) Support INSERT OVERWRITE into a partition on transactional tables
[ https://issues.apache.org/jira/browse/HIVE-14988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058384#comment-16058384 ] Wei Zheng commented on HIVE-14988: -- org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype] is the only test that has age 1. But it doesn't fail locally. [~ekoifman] Can you review please? > Support INSERT OVERWRITE into a partition on transactional tables > - > > Key: HIVE-14988 > URL: https://issues.apache.org/jira/browse/HIVE-14988 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-14988.01.patch, HIVE-14988.02.patch, > HIVE-14988.03.patch, HIVE-14988.04.patch, HIVE-14988.05.patch > > > Insert overwrite operation on transactional table will currently raise an > error. > This can/should be supported -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-14988) Support INSERT OVERWRITE into a partition on transactional tables
[ https://issues.apache.org/jira/browse/HIVE-14988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14988: - Attachment: HIVE-14988.05.patch > Support INSERT OVERWRITE into a partition on transactional tables > - > > Key: HIVE-14988 > URL: https://issues.apache.org/jira/browse/HIVE-14988 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-14988.01.patch, HIVE-14988.02.patch, > HIVE-14988.03.patch, HIVE-14988.04.patch, HIVE-14988.05.patch > > > Insert overwrite operation on transactional table will currently raise an > error. > This can/should be supported -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-14988) Support INSERT OVERWRITE into a partition on transactional tables
[ https://issues.apache.org/jira/browse/HIVE-14988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14988: - Attachment: HIVE-14988.04.patch > Support INSERT OVERWRITE into a partition on transactional tables > - > > Key: HIVE-14988 > URL: https://issues.apache.org/jira/browse/HIVE-14988 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-14988.01.patch, HIVE-14988.02.patch, > HIVE-14988.03.patch, HIVE-14988.04.patch > > > Insert overwrite operation on transactional table will currently raise an > error. > This can/should be supported -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-14988) Support INSERT OVERWRITE into a partition on transactional tables
[
https://issues.apache.org/jira/browse/HIVE-14988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056621#comment-16056621
]
Wei Zheng commented on HIVE-14988:
--
patch 03 is following the "new base" approach proposed by Eugene.
For example, we have such directory layout:
{code}
delta_1_1
delta_2_2
base_2
delta_3
{code}
After an Insert Overwrite, it should become like this:
{code}
delta_1_1
delta_2_2
base_2
delta_3
base_4 <= new base. All other dirs become obsolete.
{code}
> Support INSERT OVERWRITE into a partition on transactional tables
> -
>
> Key: HIVE-14988
> URL: https://issues.apache.org/jira/browse/HIVE-14988
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
> Attachments: HIVE-14988.01.patch, HIVE-14988.02.patch,
> HIVE-14988.03.patch
>
>
> Insert overwrite operation on transactional table will currently raise an
> error.
> This can/should be supported
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-14988) Support INSERT OVERWRITE into a partition on transactional tables
[ https://issues.apache.org/jira/browse/HIVE-14988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14988: - Status: Patch Available (was: Open) > Support INSERT OVERWRITE into a partition on transactional tables > - > > Key: HIVE-14988 > URL: https://issues.apache.org/jira/browse/HIVE-14988 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-14988.01.patch, HIVE-14988.02.patch, > HIVE-14988.03.patch > > > Insert overwrite operation on transactional table will currently raise an > error. > This can/should be supported -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-14988) Support INSERT OVERWRITE into a partition on transactional tables
[ https://issues.apache.org/jira/browse/HIVE-14988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-14988: Assignee: Wei Zheng (was: Eugene Koifman) > Support INSERT OVERWRITE into a partition on transactional tables > - > > Key: HIVE-14988 > URL: https://issues.apache.org/jira/browse/HIVE-14988 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-14988.01.patch, HIVE-14988.02.patch, > HIVE-14988.03.patch > > > Insert overwrite operation on transactional table will currently raise an > error. > This can/should be supported -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-14988) Support INSERT OVERWRITE into a partition on transactional tables
[ https://issues.apache.org/jira/browse/HIVE-14988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14988: - Attachment: HIVE-14988.03.patch > Support INSERT OVERWRITE into a partition on transactional tables > - > > Key: HIVE-14988 > URL: https://issues.apache.org/jira/browse/HIVE-14988 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-14988.01.patch, HIVE-14988.02.patch, > HIVE-14988.03.patch > > > Insert overwrite operation on transactional table will currently raise an > error. > This can/should be supported -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15212: - Attachment: HIVE-15212.08.patch > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HIVE-16850) Only open a new transaction when there's no currently opened transaction
[ https://issues.apache.org/jira/browse/HIVE-16850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng resolved HIVE-16850. -- Resolution: Fixed Fix Version/s: hive-14535 > Only open a new transaction when there's no currently opened transaction > > > Key: HIVE-16850 > URL: https://issues.apache.org/jira/browse/HIVE-16850 > Project: Hive > Issue Type: Sub-task >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-16850.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16850) Only open a new transaction when there's no currently opened transaction
[ https://issues.apache.org/jira/browse/HIVE-16850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16850: - Attachment: HIVE-16850.patch > Only open a new transaction when there's no currently opened transaction > > > Key: HIVE-16850 > URL: https://issues.apache.org/jira/browse/HIVE-16850 > Project: Hive > Issue Type: Sub-task >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-16850.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16850) Only open a new transaction when there's no currently opened transaction
[ https://issues.apache.org/jira/browse/HIVE-16850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-16850: > Only open a new transaction when there's no currently opened transaction > > > Key: HIVE-16850 > URL: https://issues.apache.org/jira/browse/HIVE-16850 > Project: Hive > Issue Type: Sub-task >Reporter: Wei Zheng >Assignee: Wei Zheng > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15212: - Attachment: HIVE-15212.07.patch > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16819) Add MM test for temporary table
[ https://issues.apache.org/jira/browse/HIVE-16819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16819: - Fix Version/s: hive-14535 > Add MM test for temporary table > --- > > Key: HIVE-16819 > URL: https://issues.apache.org/jira/browse/HIVE-16819 > Project: Hive > Issue Type: Sub-task >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-16819.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HIVE-16819) Add MM test for temporary table
[ https://issues.apache.org/jira/browse/HIVE-16819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng resolved HIVE-16819. -- Resolution: Fixed > Add MM test for temporary table > --- > > Key: HIVE-16819 > URL: https://issues.apache.org/jira/browse/HIVE-16819 > Project: Hive > Issue Type: Sub-task >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-16819.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16819) Add MM test for temporary table
[ https://issues.apache.org/jira/browse/HIVE-16819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16819: - Attachment: HIVE-16819.patch > Add MM test for temporary table > --- > > Key: HIVE-16819 > URL: https://issues.apache.org/jira/browse/HIVE-16819 > Project: Hive > Issue Type: Sub-task >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-16819.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16819) Add MM test for temporary table
[ https://issues.apache.org/jira/browse/HIVE-16819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-16819: > Add MM test for temporary table > --- > > Key: HIVE-16819 > URL: https://issues.apache.org/jira/browse/HIVE-16819 > Project: Hive > Issue Type: Sub-task >Reporter: Wei Zheng >Assignee: Wei Zheng > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-11538) Add an option to skip init script while running tests
[
https://issues.apache.org/jira/browse/HIVE-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035482#comment-16035482
]
Wei Zheng commented on HIVE-11538:
--
For some reason this doesn't work for me -- it still runs the init script.
Here's the command that I used. Anything wrong?
{code}
mvn test -Dtest=TestCliDriver -Dqfile=mm_materializedview.q -DinitScript=
{code}
> Add an option to skip init script while running tests
> -
>
> Key: HIVE-11538
> URL: https://issues.apache.org/jira/browse/HIVE-11538
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 2.0.0
>
> Attachments: HIVE-11538.2.patch, HIVE-11538.3.patch, HIVE-11538.patch
>
>
> {{q_test_init.sql}} has grown over time. Now, it takes substantial amount of
> time. When debugging a particular query which doesn't need such
> initialization, this delay is annoyance.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16817) Restore CTAS tests in mm_all.q
[ https://issues.apache.org/jira/browse/HIVE-16817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16817: - Component/s: (was: Materialized views) (was: Parser) > Restore CTAS tests in mm_all.q > -- > > Key: HIVE-16817 > URL: https://issues.apache.org/jira/browse/HIVE-16817 > Project: Hive > Issue Type: Sub-task >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-16817.patch > > > In earlier ACID integration patch CTAS was not supported. (previously I used > a different approach in which I created a new data operation type for INSERT > ONLY, which got errored out in TxnHandler. Later I changed that to INSERT > which is working fine) As CTAS is working now, the corresponding tests for it > should be restored. > Note we still have the same limitations for MM tables as Hive regular tables > do in general: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTableAsSelect(CTAS) > > CTAS has these restrictions: > The target table cannot be a partitioned table. > The target table cannot be an external table. > The target table cannot be a list bucketing table. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HIVE-16817) Restore CTAS tests in mm_all.q
[ https://issues.apache.org/jira/browse/HIVE-16817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng resolved HIVE-16817. -- Resolution: Fixed Fix Version/s: (was: 2.2.0) hive-14535 > Restore CTAS tests in mm_all.q > -- > > Key: HIVE-16817 > URL: https://issues.apache.org/jira/browse/HIVE-16817 > Project: Hive > Issue Type: Sub-task >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-16817.patch > > > In earlier ACID integration patch CTAS was not supported. (previously I used > a different approach in which I created a new data operation type for INSERT > ONLY, which got errored out in TxnHandler. Later I changed that to INSERT > which is working fine) As CTAS is working now, the corresponding tests for it > should be restored. > Note we still have the same limitations for MM tables as Hive regular tables > do in general: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTableAsSelect(CTAS) > > CTAS has these restrictions: > The target table cannot be a partitioned table. > The target table cannot be an external table. > The target table cannot be a list bucketing table. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16817) Restore CTAS tests in mm_all.q
[ https://issues.apache.org/jira/browse/HIVE-16817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16817: - Attachment: HIVE-16817.patch > Restore CTAS tests in mm_all.q > -- > > Key: HIVE-16817 > URL: https://issues.apache.org/jira/browse/HIVE-16817 > Project: Hive > Issue Type: Sub-task > Components: Materialized views, Parser >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: 2.2.0 > > Attachments: HIVE-16817.patch > > > In earlier ACID integration patch CTAS was not supported. (previously I used > a different approach in which I created a new data operation type for INSERT > ONLY, which got errored out in TxnHandler. Later I changed that to INSERT > which is working fine) As CTAS is working now, the corresponding tests for it > should be restored. > Note we still have the same limitations for MM tables as Hive regular tables > do in general: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTableAsSelect(CTAS) > > CTAS has these restrictions: > The target table cannot be a partitioned table. > The target table cannot be an external table. > The target table cannot be a list bucketing table. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16817) Restore CTAS tests in mm_all.q
[ https://issues.apache.org/jira/browse/HIVE-16817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16817: - Parent Issue: HIVE-14535 (was: HIVE-14249) > Restore CTAS tests in mm_all.q > -- > > Key: HIVE-16817 > URL: https://issues.apache.org/jira/browse/HIVE-16817 > Project: Hive > Issue Type: Sub-task > Components: Materialized views, Parser >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: 2.2.0 > > > In earlier ACID integration patch CTAS was not supported. (previously I used > a different approach in which I created a new data operation type for INSERT > ONLY, which got errored out in TxnHandler. Later I changed that to INSERT > which is working fine) As CTAS is working now, the corresponding tests for it > should be restored. > Note we still have the same limitations for MM tables as Hive regular tables > do in general: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTableAsSelect(CTAS) > > CTAS has these restrictions: > The target table cannot be a partitioned table. > The target table cannot be an external table. > The target table cannot be a list bucketing table. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16817) Restore CTAS tests in mm_all.q
[ https://issues.apache.org/jira/browse/HIVE-16817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-16817: > Restore CTAS tests in mm_all.q > -- > > Key: HIVE-16817 > URL: https://issues.apache.org/jira/browse/HIVE-16817 > Project: Hive > Issue Type: Sub-task >Reporter: Wei Zheng >Assignee: Wei Zheng > > In earlier ACID integration patch CTAS was not supported. (previously I used > a different approach in which I created a new data operation type for INSERT > ONLY, which got errored out in TxnHandler. Later I changed that to INSERT > which is working fine) As CTAS is working now, the corresponding tests for it > should be restored. > Note we still have the same limitations for MM tables as Hive regular tables > do in general: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTableAsSelect(CTAS) > > CTAS has these restrictions: > The target table cannot be a partitioned table. > The target table cannot be an external table. > The target table cannot be a list bucketing table. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14249) Add simple materialized views with manual rebuilds
[ https://issues.apache.org/jira/browse/HIVE-14249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034912#comment-16034912 ] Wei Zheng commented on HIVE-14249: -- [~jcamachorodriguez] Looks like materialized view hasn't been documented in the wiki > Add simple materialized views with manual rebuilds > -- > > Key: HIVE-14249 > URL: https://issues.apache.org/jira/browse/HIVE-14249 > Project: Hive > Issue Type: New Feature > Components: Materialized views, Parser >Reporter: Alan Gates >Assignee: Jesus Camacho Rodriguez > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-10459.2.patch, HIVE-14249.05.patch, > HIVE-14249.06.patch > > > This patch is a start at implementing simple views. It doesn't have enough > testing yet (e.g. there's no negative testing). And I know it fails in the > partitioned case. I suspect things like security and locking don't work > properly yet either. But I'm posting it as a starting point. > In this initial patch I'm just handling simple materialized views with manual > rebuilds. In later JIRAs we can add features such as allowing the optimizer > to rewrite queries to use materialized views rather than tables named in the > queries, giving the optimizer the ability to determine when a materialized > view is stale, etc. > Also, I didn't rebase this patch against trunk after the migration from > svn->git so it may not apply cleanly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HIVE-16052) MM tables: add exchange partition test after ACID integration
[ https://issues.apache.org/jira/browse/HIVE-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng resolved HIVE-16052. -- Resolution: Fixed > MM tables: add exchange partition test after ACID integration > - > > Key: HIVE-16052 > URL: https://issues.apache.org/jira/browse/HIVE-16052 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-16052.patch > > > exchgpartition2lel test fails if all tables are changed to MM, because of > write ID mismatch between directories and tables when exchanging partition > directories between tables. ACID should probably fix this because transaction > IDs are global. > We should add a test after integrating with ACID; if it doesn't work for some > other reason, we can either implement it as moving to a new mm_id/txn_id in > each affected partition, or block it on MM tables. > cc [~wzheng] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16052) MM tables: add exchange partition test after ACID integration
[ https://issues.apache.org/jira/browse/HIVE-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16052: - Attachment: HIVE-16052.patch > MM tables: add exchange partition test after ACID integration > - > > Key: HIVE-16052 > URL: https://issues.apache.org/jira/browse/HIVE-16052 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-16052.patch > > > exchgpartition2lel test fails if all tables are changed to MM, because of > write ID mismatch between directories and tables when exchanging partition > directories between tables. ACID should probably fix this because transaction > IDs are global. > We should add a test after integrating with ACID; if it doesn't work for some > other reason, we can either implement it as moving to a new mm_id/txn_id in > each affected partition, or block it on MM tables. > cc [~wzheng] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HIVE-16810) Fix an export/import bug due to ACID integration
[ https://issues.apache.org/jira/browse/HIVE-16810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng resolved HIVE-16810. -- Resolution: Fixed Fix Version/s: hive-14535 > Fix an export/import bug due to ACID integration > > > Key: HIVE-16810 > URL: https://issues.apache.org/jira/browse/HIVE-16810 > Project: Hive > Issue Type: Sub-task >Affects Versions: hive-14535 >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-16810.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16810) Fix an export/import bug due to ACID integration
[ https://issues.apache.org/jira/browse/HIVE-16810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16810: - Attachment: HIVE-16810.1.patch > Fix an export/import bug due to ACID integration > > > Key: HIVE-16810 > URL: https://issues.apache.org/jira/browse/HIVE-16810 > Project: Hive > Issue Type: Sub-task >Affects Versions: hive-14535 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-16810.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16810) Fix an export/import bug due to ACID integration
[ https://issues.apache.org/jira/browse/HIVE-16810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-16810: > Fix an export/import bug due to ACID integration > > > Key: HIVE-16810 > URL: https://issues.apache.org/jira/browse/HIVE-16810 > Project: Hive > Issue Type: Sub-task >Affects Versions: hive-14535 >Reporter: Wei Zheng >Assignee: Wei Zheng > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (HIVE-16052) MM tables: add exchange partition test after ACID integration
[ https://issues.apache.org/jira/browse/HIVE-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reopened HIVE-16052: -- Assignee: Wei Zheng Sure, will add the test for MM > MM tables: add exchange partition test after ACID integration > - > > Key: HIVE-16052 > URL: https://issues.apache.org/jira/browse/HIVE-16052 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Fix For: hive-14535 > > > exchgpartition2lel test fails if all tables are changed to MM, because of > write ID mismatch between directories and tables when exchanging partition > directories between tables. ACID should probably fix this because transaction > IDs are global. > We should add a test after integrating with ACID; if it doesn't work for some > other reason, we can either implement it as moving to a new mm_id/txn_id in > each affected partition, or block it on MM tables. > cc [~wzheng] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16052) MM tables: add exchange partition test after ACID integration
[ https://issues.apache.org/jira/browse/HIVE-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032214#comment-16032214 ] Wei Zheng commented on HIVE-16052: -- exchgpartition2lel.q. When I convert the tables to MM, it produced correct result. Do we want a counterpart of this test for MM? > MM tables: add exchange partition test after ACID integration > - > > Key: HIVE-16052 > URL: https://issues.apache.org/jira/browse/HIVE-16052 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > Fix For: hive-14535 > > > exchgpartition2lel test fails if all tables are changed to MM, because of > write ID mismatch between directories and tables when exchanging partition > directories between tables. ACID should probably fix this because transaction > IDs are global. > We should add a test after integrating with ACID; if it doesn't work for some > other reason, we can either implement it as moving to a new mm_id/txn_id in > each affected partition, or block it on MM tables. > cc [~wzheng] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HIVE-16052) MM tables: add exchange partition test after ACID integration
[ https://issues.apache.org/jira/browse/HIVE-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng resolved HIVE-16052. -- Resolution: Fixed Fix Version/s: hive-14535 > MM tables: add exchange partition test after ACID integration > - > > Key: HIVE-16052 > URL: https://issues.apache.org/jira/browse/HIVE-16052 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > Fix For: hive-14535 > > > exchgpartition2lel test fails if all tables are changed to MM, because of > write ID mismatch between directories and tables when exchanging partition > directories between tables. ACID should probably fix this because transaction > IDs are global. > We should add a test after integrating with ACID; if it doesn't work for some > other reason, we can either implement it as moving to a new mm_id/txn_id in > each affected partition, or block it on MM tables. > cc [~wzheng] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16052) MM tables: add exchange partition test after ACID integration
[ https://issues.apache.org/jira/browse/HIVE-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032112#comment-16032112 ] Wei Zheng commented on HIVE-16052: -- I've run this test with all tables created as MM. The test ran fine and the result is correct. Closing this one. > MM tables: add exchange partition test after ACID integration > - > > Key: HIVE-16052 > URL: https://issues.apache.org/jira/browse/HIVE-16052 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > Fix For: hive-14535 > > > exchgpartition2lel test fails if all tables are changed to MM, because of > write ID mismatch between directories and tables when exchanging partition > directories between tables. ACID should probably fix this because transaction > IDs are global. > We should add a test after integrating with ACID; if it doesn't work for some > other reason, we can either implement it as moving to a new mm_id/txn_id in > each affected partition, or block it on MM tables. > cc [~wzheng] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16052) MM tables: add exchange partition test after ACID integration
[ https://issues.apache.org/jira/browse/HIVE-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032113#comment-16032113 ] Wei Zheng commented on HIVE-16052: -- p.s. This is after ACID integration. > MM tables: add exchange partition test after ACID integration > - > > Key: HIVE-16052 > URL: https://issues.apache.org/jira/browse/HIVE-16052 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > Fix For: hive-14535 > > > exchgpartition2lel test fails if all tables are changed to MM, because of > write ID mismatch between directories and tables when exchanging partition > directories between tables. ACID should probably fix this because transaction > IDs are global. > We should add a test after integrating with ACID; if it doesn't work for some > other reason, we can either implement it as moving to a new mm_id/txn_id in > each affected partition, or block it on MM tables. > cc [~wzheng] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14990) run all tests for MM tables and fix the issues that are found
[ https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14990: - Attachment: HIVE-14990.21.patch > run all tests for MM tables and fix the issues that are found > - > > Key: HIVE-14990 > URL: https://issues.apache.org/jira/browse/HIVE-14990 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-14990.01.patch, HIVE-14990.02.patch, > HIVE-14990.03.patch, HIVE-14990.04.patch, HIVE-14990.04.patch, > HIVE-14990.05.patch, HIVE-14990.05.patch, HIVE-14990.06.patch, > HIVE-14990.06.patch, HIVE-14990.07.patch, HIVE-14990.08.patch, > HIVE-14990.09.patch, HIVE-14990.10.patch, HIVE-14990.10.patch, > HIVE-14990.10.patch, HIVE-14990.12.patch, HIVE-14990.13.patch, > HIVE-14990.14.patch, HIVE-14990.15.patch, HIVE-14990.16.patch, > HIVE-14990.17.patch, HIVE-14990.18.patch, HIVE-14990.19.patch, > HIVE-14990.20.patch, HIVE-14990.21.patch, HIVE-14990.patch > > > I am running the tests with isMmTable returning true for most tables (except > ACID, temporary tables, views, etc.). > Many tests will fail because of various expected issues with such an > approach; however we can find issues in MM tables from other failures. > Expected failures > 1) All HCat tests (cannot write MM tables via the HCat writer) > 2) Almost all merge tests (alter .. concat is not supported). > 3) Tests that run dfs commands with specific paths (path changes). > 4) Truncate column (not supported). > 5) Describe formatted will have the new table fields in the output (before > merging MM with ACID). > 6) Many tests w/explain extended - diff in partition "base file name" (path > changes). > 7) TestTxnCommands - all the conversion tests, as they check for bucket count > using file lists (path changes). > 8) HBase metastore tests cause methods are not implemented. > 9) Some load and ExIm tests that export a table and then rely on specific > path for load (path changes). > 10) Bucket map join/etc. - diffs; disabled the optimization for MM tables due > to how it accounts for buckets > 11) rand - different results due to different sequence of processing. > 12) many (not all i.e. not the ones with just one insert) tests that have > stats output, such as file count, for obvious reasons > 13) materialized views, not handled by design - the test check erroneously > makes them "mm", no easy way to tell them apart, I don't want to plumb more > stuff thru just for this test > I'm filing jiras for some test failures that are not obvious and need an > investigation later -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14990) run all tests for MM tables and fix the issues that are found
[ https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14990: - Attachment: HIVE-14990.20.patch > run all tests for MM tables and fix the issues that are found > - > > Key: HIVE-14990 > URL: https://issues.apache.org/jira/browse/HIVE-14990 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-14990.01.patch, HIVE-14990.02.patch, > HIVE-14990.03.patch, HIVE-14990.04.patch, HIVE-14990.04.patch, > HIVE-14990.05.patch, HIVE-14990.05.patch, HIVE-14990.06.patch, > HIVE-14990.06.patch, HIVE-14990.07.patch, HIVE-14990.08.patch, > HIVE-14990.09.patch, HIVE-14990.10.patch, HIVE-14990.10.patch, > HIVE-14990.10.patch, HIVE-14990.12.patch, HIVE-14990.13.patch, > HIVE-14990.14.patch, HIVE-14990.15.patch, HIVE-14990.16.patch, > HIVE-14990.17.patch, HIVE-14990.18.patch, HIVE-14990.19.patch, > HIVE-14990.20.patch, HIVE-14990.patch > > > I am running the tests with isMmTable returning true for most tables (except > ACID, temporary tables, views, etc.). > Many tests will fail because of various expected issues with such an > approach; however we can find issues in MM tables from other failures. > Expected failures > 1) All HCat tests (cannot write MM tables via the HCat writer) > 2) Almost all merge tests (alter .. concat is not supported). > 3) Tests that run dfs commands with specific paths (path changes). > 4) Truncate column (not supported). > 5) Describe formatted will have the new table fields in the output (before > merging MM with ACID). > 6) Many tests w/explain extended - diff in partition "base file name" (path > changes). > 7) TestTxnCommands - all the conversion tests, as they check for bucket count > using file lists (path changes). > 8) HBase metastore tests cause methods are not implemented. > 9) Some load and ExIm tests that export a table and then rely on specific > path for load (path changes). > 10) Bucket map join/etc. - diffs; disabled the optimization for MM tables due > to how it accounts for buckets > 11) rand - different results due to different sequence of processing. > 12) many (not all i.e. not the ones with just one insert) tests that have > stats output, such as file count, for obvious reasons > 13) materialized views, not handled by design - the test check erroneously > makes them "mm", no easy way to tell them apart, I don't want to plumb more > stuff thru just for this test > I'm filing jiras for some test failures that are not obvious and need an > investigation later -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14990) run all tests for MM tables and fix the issues that are found
[
https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-14990:
-
Attachment: HIVE-14990.19.patch
Upload patch 19 for testing.
The reason why there were ~4000 tests not being run previously is due to setup
phase failure in q_test_init.sql. LOAD command failed with
{code}
FAILED: SemanticException [Error 10265]: This command is not allowed on an ACID
table default.src with a non-ACID transaction manager.
{code}
Adding the txn manager settings and try again.
> run all tests for MM tables and fix the issues that are found
> -
>
> Key: HIVE-14990
> URL: https://issues.apache.org/jira/browse/HIVE-14990
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Wei Zheng
> Fix For: hive-14535
>
> Attachments: HIVE-14990.01.patch, HIVE-14990.02.patch,
> HIVE-14990.03.patch, HIVE-14990.04.patch, HIVE-14990.04.patch,
> HIVE-14990.05.patch, HIVE-14990.05.patch, HIVE-14990.06.patch,
> HIVE-14990.06.patch, HIVE-14990.07.patch, HIVE-14990.08.patch,
> HIVE-14990.09.patch, HIVE-14990.10.patch, HIVE-14990.10.patch,
> HIVE-14990.10.patch, HIVE-14990.12.patch, HIVE-14990.13.patch,
> HIVE-14990.14.patch, HIVE-14990.15.patch, HIVE-14990.16.patch,
> HIVE-14990.17.patch, HIVE-14990.18.patch, HIVE-14990.19.patch,
> HIVE-14990.patch
>
>
> I am running the tests with isMmTable returning true for most tables (except
> ACID, temporary tables, views, etc.).
> Many tests will fail because of various expected issues with such an
> approach; however we can find issues in MM tables from other failures.
> Expected failures
> 1) All HCat tests (cannot write MM tables via the HCat writer)
> 2) Almost all merge tests (alter .. concat is not supported).
> 3) Tests that run dfs commands with specific paths (path changes).
> 4) Truncate column (not supported).
> 5) Describe formatted will have the new table fields in the output (before
> merging MM with ACID).
> 6) Many tests w/explain extended - diff in partition "base file name" (path
> changes).
> 7) TestTxnCommands - all the conversion tests, as they check for bucket count
> using file lists (path changes).
> 8) HBase metastore tests cause methods are not implemented.
> 9) Some load and ExIm tests that export a table and then rely on specific
> path for load (path changes).
> 10) Bucket map join/etc. - diffs; disabled the optimization for MM tables due
> to how it accounts for buckets
> 11) rand - different results due to different sequence of processing.
> 12) many (not all i.e. not the ones with just one insert) tests that have
> stats output, such as file count, for obvious reasons
> 13) materialized views, not handled by design - the test check erroneously
> makes them "mm", no easy way to tell them apart, I don't want to plumb more
> stuff thru just for this test
> I'm filing jiras for some test failures that are not obvious and need an
> investigation later
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-14990) run all tests for MM tables and fix the issues that are found
[ https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025871#comment-16025871 ] Wei Zheng commented on HIVE-14990: -- That's strange. I didn't notice that. The test run here has 11k tests run. The only difference between the two patches is the fake logic for returning true (mm table) all the time. > run all tests for MM tables and fix the issues that are found > - > > Key: HIVE-14990 > URL: https://issues.apache.org/jira/browse/HIVE-14990 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-14990.01.patch, HIVE-14990.02.patch, > HIVE-14990.03.patch, HIVE-14990.04.patch, HIVE-14990.04.patch, > HIVE-14990.05.patch, HIVE-14990.05.patch, HIVE-14990.06.patch, > HIVE-14990.06.patch, HIVE-14990.07.patch, HIVE-14990.08.patch, > HIVE-14990.09.patch, HIVE-14990.10.patch, HIVE-14990.10.patch, > HIVE-14990.10.patch, HIVE-14990.12.patch, HIVE-14990.13.patch, > HIVE-14990.14.patch, HIVE-14990.15.patch, HIVE-14990.16.patch, > HIVE-14990.17.patch, HIVE-14990.18.patch, HIVE-14990.patch > > > I am running the tests with isMmTable returning true for most tables (except > ACID, temporary tables, views, etc.). > Many tests will fail because of various expected issues with such an > approach; however we can find issues in MM tables from other failures. > Expected failures > 1) All HCat tests (cannot write MM tables via the HCat writer) > 2) Almost all merge tests (alter .. concat is not supported). > 3) Tests that run dfs commands with specific paths (path changes). > 4) Truncate column (not supported). > 5) Describe formatted will have the new table fields in the output (before > merging MM with ACID). > 6) Many tests w/explain extended - diff in partition "base file name" (path > changes). > 7) TestTxnCommands - all the conversion tests, as they check for bucket count > using file lists (path changes). > 8) HBase metastore tests cause methods are not implemented. > 9) Some load and ExIm tests that export a table and then rely on specific > path for load (path changes). > 10) Bucket map join/etc. - diffs; disabled the optimization for MM tables due > to how it accounts for buckets > 11) rand - different results due to different sequence of processing. > 12) many (not all i.e. not the ones with just one insert) tests that have > stats output, such as file count, for obvious reasons > 13) materialized views, not handled by design - the test check erroneously > makes them "mm", no easy way to tell them apart, I don't want to plumb more > stuff thru just for this test > I'm filing jiras for some test failures that are not obvious and need an > investigation later -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14990) run all tests for MM tables and fix the issues that are found
[ https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14990: - Attachment: HIVE-14990.18.patch Upload patch 18 for testing > run all tests for MM tables and fix the issues that are found > - > > Key: HIVE-14990 > URL: https://issues.apache.org/jira/browse/HIVE-14990 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-14990.01.patch, HIVE-14990.02.patch, > HIVE-14990.03.patch, HIVE-14990.04.patch, HIVE-14990.04.patch, > HIVE-14990.05.patch, HIVE-14990.05.patch, HIVE-14990.06.patch, > HIVE-14990.06.patch, HIVE-14990.07.patch, HIVE-14990.08.patch, > HIVE-14990.09.patch, HIVE-14990.10.patch, HIVE-14990.10.patch, > HIVE-14990.10.patch, HIVE-14990.12.patch, HIVE-14990.13.patch, > HIVE-14990.14.patch, HIVE-14990.15.patch, HIVE-14990.16.patch, > HIVE-14990.17.patch, HIVE-14990.18.patch, HIVE-14990.patch > > > I am running the tests with isMmTable returning true for most tables (except > ACID, temporary tables, views, etc.). > Many tests will fail because of various expected issues with such an > approach; however we can find issues in MM tables from other failures. > Expected failures > 1) All HCat tests (cannot write MM tables via the HCat writer) > 2) Almost all merge tests (alter .. concat is not supported). > 3) Tests that run dfs commands with specific paths (path changes). > 4) Truncate column (not supported). > 5) Describe formatted will have the new table fields in the output (before > merging MM with ACID). > 6) Many tests w/explain extended - diff in partition "base file name" (path > changes). > 7) TestTxnCommands - all the conversion tests, as they check for bucket count > using file lists (path changes). > 8) HBase metastore tests cause methods are not implemented. > 9) Some load and ExIm tests that export a table and then rely on specific > path for load (path changes). > 10) Bucket map join/etc. - diffs; disabled the optimization for MM tables due > to how it accounts for buckets > 11) rand - different results due to different sequence of processing. > 12) many (not all i.e. not the ones with just one insert) tests that have > stats output, such as file count, for obvious reasons > 13) materialized views, not handled by design - the test check erroneously > makes them "mm", no easy way to tell them apart, I don't want to plumb more > stuff thru just for this test > I'm filing jiras for some test failures that are not obvious and need an > investigation later -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025480#comment-16025480 ] Wei Zheng commented on HIVE-15212: -- I agree with you :) Let me run precommit test for HIVE-14990 one more time and look thru the failures. > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025458#comment-16025458 ] Wei Zheng commented on HIVE-15212: -- I'm adding a dependency here: HIVE-14990 > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15212: - Attachment: HIVE-15212.06.patch Upload patch 6 for testing > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16760) Update errata.txt for HIVE-16743
[ https://issues.apache.org/jira/browse/HIVE-16760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025192#comment-16025192 ] Wei Zheng commented on HIVE-16760: -- Oh OK, first time doing this ;) I was referring to an example here: https://issues.apache.org/jira/browse/HIVE-11876 > Update errata.txt for HIVE-16743 > > > Key: HIVE-16760 > URL: https://issues.apache.org/jira/browse/HIVE-16760 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-16760.patch > > > Refer to: > https://issues.apache.org/jira/browse/HIVE-16743?focusedCommentId=16024139=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16024139 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HIVE-16760) Update errata.txt for HIVE-16743
[ https://issues.apache.org/jira/browse/HIVE-16760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng resolved HIVE-16760. -- Resolution: Fixed > Update errata.txt for HIVE-16743 > > > Key: HIVE-16760 > URL: https://issues.apache.org/jira/browse/HIVE-16760 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-16760.patch > > > Refer to: > https://issues.apache.org/jira/browse/HIVE-16743?focusedCommentId=16024139=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16024139 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16760) Update errata.txt for HIVE-16743
[ https://issues.apache.org/jira/browse/HIVE-16760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16760: - Attachment: HIVE-16760.patch > Update errata.txt for HIVE-16743 > > > Key: HIVE-16760 > URL: https://issues.apache.org/jira/browse/HIVE-16760 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-16760.patch > > > Refer to: > https://issues.apache.org/jira/browse/HIVE-16743?focusedCommentId=16024139=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16024139 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16760) Update errata.txt for HIVE-16743
[ https://issues.apache.org/jira/browse/HIVE-16760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16760: - Target Version/s: 3.0.0 (was: 3.0.0, hive-14535) > Update errata.txt for HIVE-16743 > > > Key: HIVE-16760 > URL: https://issues.apache.org/jira/browse/HIVE-16760 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > > Refer to: > https://issues.apache.org/jira/browse/HIVE-16743?focusedCommentId=16024139=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16024139 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16760) Update errata.txt for HIVE-16743
[ https://issues.apache.org/jira/browse/HIVE-16760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16760: - Affects Version/s: (was: hive-14535) > Update errata.txt for HIVE-16743 > > > Key: HIVE-16760 > URL: https://issues.apache.org/jira/browse/HIVE-16760 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > > Refer to: > https://issues.apache.org/jira/browse/HIVE-16743?focusedCommentId=16024139=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16024139 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16760) Update errata.txt for HIVE-16743
[ https://issues.apache.org/jira/browse/HIVE-16760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-16760: > Update errata.txt for HIVE-16743 > > > Key: HIVE-16760 > URL: https://issues.apache.org/jira/browse/HIVE-16760 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, hive-14535 >Reporter: Wei Zheng >Assignee: Wei Zheng > > Refer to: > https://issues.apache.org/jira/browse/HIVE-16743?focusedCommentId=16024139=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16024139 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16743) BitSet set() is incorrectly used in TxnUtils.createValidCompactTxnList()
[
https://issues.apache.org/jira/browse/HIVE-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024985#comment-16024985
]
Wei Zheng commented on HIVE-16743:
--
Woops.. Will do that. Thanks for finding it out!
> BitSet set() is incorrectly used in TxnUtils.createValidCompactTxnList()
>
>
> Key: HIVE-16743
> URL: https://issues.apache.org/jira/browse/HIVE-16743
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Fix For: 3.0.0
>
> Attachments: HIVE-16743.1.patch
>
>
> The second line is problematic
> {code}
> BitSet bitSet = new BitSet(exceptions.length);
> bitSet.set(0, bitSet.length()); // for ValidCompactorTxnList, everything
> in exceptions are aborted
> {code}
> For example, exceptions' length is 2. We declare a BitSet object with initial
> size of 2 via the first line above. But that's not the actual size of the
> BitSet. So bitSet.length() will still return 0.
> The intention of the second line above is to set all the bits to true. This
> was not achieved because bitSet.set(0, bitSet.length()) is equivalent to
> bitSet.set(0, 0).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Reopened] (HIVE-14990) run all tests for MM tables and fix the issues that are found
[ https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reopened HIVE-14990: -- Assignee: Wei Zheng (was: Sergey Shelukhin) As ACID integration is done, I want to re-run this to see what the feature parity is like. > run all tests for MM tables and fix the issues that are found > - > > Key: HIVE-14990 > URL: https://issues.apache.org/jira/browse/HIVE-14990 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-14990.01.patch, HIVE-14990.02.patch, > HIVE-14990.03.patch, HIVE-14990.04.patch, HIVE-14990.04.patch, > HIVE-14990.05.patch, HIVE-14990.05.patch, HIVE-14990.06.patch, > HIVE-14990.06.patch, HIVE-14990.07.patch, HIVE-14990.08.patch, > HIVE-14990.09.patch, HIVE-14990.10.patch, HIVE-14990.10.patch, > HIVE-14990.10.patch, HIVE-14990.12.patch, HIVE-14990.13.patch, > HIVE-14990.14.patch, HIVE-14990.15.patch, HIVE-14990.16.patch, > HIVE-14990.17.patch, HIVE-14990.patch > > > I am running the tests with isMmTable returning true for most tables (except > ACID, temporary tables, views, etc.). > Many tests will fail because of various expected issues with such an > approach; however we can find issues in MM tables from other failures. > Expected failures > 1) All HCat tests (cannot write MM tables via the HCat writer) > 2) Almost all merge tests (alter .. concat is not supported). > 3) Tests that run dfs commands with specific paths (path changes). > 4) Truncate column (not supported). > 5) Describe formatted will have the new table fields in the output (before > merging MM with ACID). > 6) Many tests w/explain extended - diff in partition "base file name" (path > changes). > 7) TestTxnCommands - all the conversion tests, as they check for bucket count > using file lists (path changes). > 8) HBase metastore tests cause methods are not implemented. > 9) Some load and ExIm tests that export a table and then rely on specific > path for load (path changes). > 10) Bucket map join/etc. - diffs; disabled the optimization for MM tables due > to how it accounts for buckets > 11) rand - different results due to different sequence of processing. > 12) many (not all i.e. not the ones with just one insert) tests that have > stats output, such as file count, for obvious reasons > 13) materialized views, not handled by design - the test check erroneously > makes them "mm", no easy way to tell them apart, I don't want to plumb more > stuff thru just for this test > I'm filing jiras for some test failures that are not obvious and need an > investigation later -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14990) run all tests for MM tables and fix the issues that are found
[ https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14990: - Status: Patch Available (was: Reopened) > run all tests for MM tables and fix the issues that are found > - > > Key: HIVE-14990 > URL: https://issues.apache.org/jira/browse/HIVE-14990 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-14990.01.patch, HIVE-14990.02.patch, > HIVE-14990.03.patch, HIVE-14990.04.patch, HIVE-14990.04.patch, > HIVE-14990.05.patch, HIVE-14990.05.patch, HIVE-14990.06.patch, > HIVE-14990.06.patch, HIVE-14990.07.patch, HIVE-14990.08.patch, > HIVE-14990.09.patch, HIVE-14990.10.patch, HIVE-14990.10.patch, > HIVE-14990.10.patch, HIVE-14990.12.patch, HIVE-14990.13.patch, > HIVE-14990.14.patch, HIVE-14990.15.patch, HIVE-14990.16.patch, > HIVE-14990.17.patch, HIVE-14990.patch > > > I am running the tests with isMmTable returning true for most tables (except > ACID, temporary tables, views, etc.). > Many tests will fail because of various expected issues with such an > approach; however we can find issues in MM tables from other failures. > Expected failures > 1) All HCat tests (cannot write MM tables via the HCat writer) > 2) Almost all merge tests (alter .. concat is not supported). > 3) Tests that run dfs commands with specific paths (path changes). > 4) Truncate column (not supported). > 5) Describe formatted will have the new table fields in the output (before > merging MM with ACID). > 6) Many tests w/explain extended - diff in partition "base file name" (path > changes). > 7) TestTxnCommands - all the conversion tests, as they check for bucket count > using file lists (path changes). > 8) HBase metastore tests cause methods are not implemented. > 9) Some load and ExIm tests that export a table and then rely on specific > path for load (path changes). > 10) Bucket map join/etc. - diffs; disabled the optimization for MM tables due > to how it accounts for buckets > 11) rand - different results due to different sequence of processing. > 12) many (not all i.e. not the ones with just one insert) tests that have > stats output, such as file count, for obvious reasons > 13) materialized views, not handled by design - the test check erroneously > makes them "mm", no easy way to tell them apart, I don't want to plumb more > stuff thru just for this test > I'm filing jiras for some test failures that are not obvious and need an > investigation later -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14990) run all tests for MM tables and fix the issues that are found
[ https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14990: - Attachment: HIVE-14990.17.patch Uploading patch 17 for testing > run all tests for MM tables and fix the issues that are found > - > > Key: HIVE-14990 > URL: https://issues.apache.org/jira/browse/HIVE-14990 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-14990.01.patch, HIVE-14990.02.patch, > HIVE-14990.03.patch, HIVE-14990.04.patch, HIVE-14990.04.patch, > HIVE-14990.05.patch, HIVE-14990.05.patch, HIVE-14990.06.patch, > HIVE-14990.06.patch, HIVE-14990.07.patch, HIVE-14990.08.patch, > HIVE-14990.09.patch, HIVE-14990.10.patch, HIVE-14990.10.patch, > HIVE-14990.10.patch, HIVE-14990.12.patch, HIVE-14990.13.patch, > HIVE-14990.14.patch, HIVE-14990.15.patch, HIVE-14990.16.patch, > HIVE-14990.17.patch, HIVE-14990.patch > > > I am running the tests with isMmTable returning true for most tables (except > ACID, temporary tables, views, etc.). > Many tests will fail because of various expected issues with such an > approach; however we can find issues in MM tables from other failures. > Expected failures > 1) All HCat tests (cannot write MM tables via the HCat writer) > 2) Almost all merge tests (alter .. concat is not supported). > 3) Tests that run dfs commands with specific paths (path changes). > 4) Truncate column (not supported). > 5) Describe formatted will have the new table fields in the output (before > merging MM with ACID). > 6) Many tests w/explain extended - diff in partition "base file name" (path > changes). > 7) TestTxnCommands - all the conversion tests, as they check for bucket count > using file lists (path changes). > 8) HBase metastore tests cause methods are not implemented. > 9) Some load and ExIm tests that export a table and then rely on specific > path for load (path changes). > 10) Bucket map join/etc. - diffs; disabled the optimization for MM tables due > to how it accounts for buckets > 11) rand - different results due to different sequence of processing. > 12) many (not all i.e. not the ones with just one insert) tests that have > stats output, such as file count, for obvious reasons > 13) materialized views, not handled by design - the test check erroneously > makes them "mm", no easy way to tell them apart, I don't want to plumb more > stuff thru just for this test > I'm filing jiras for some test failures that are not obvious and need an > investigation later -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15212: - Status: Patch Available (was: Open) > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
