[jira] [Commented] (HIVE-20152) reset db state, when repl dump fails, so rename table can be done
[
https://issues.apache.org/jira/browse/HIVE-20152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546121#comment-16546121
]
Hive QA commented on HIVE-20152:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 2273 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 3 new + 14 unchanged - 0 fixed
= 17 total (was 14) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s{color} | {color:green} ql generated 0 new + 2272 unchanged - 1 fixed = 2272
total (was 2273) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12650/dev-support/hive-personality.sh
|
| git revision | master / 85a3dd7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12650/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12650/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> reset db state, when repl dump fails, so rename table can be done
> -
>
> Key: HIVE-20152
> URL: https://issues.apache.org/jira/browse/HIVE-20152
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20152.1.patch, HIVE-20152.2.patch
>
>
> If a repl dump command is run and it fails for some reason while doing table
> level dumps, the state set on the db parameters is not reset and hence no
> table / partition renames can be done.
> the property to be reset is prefixed with key {code}bootstrap.dump.state
> {code}
> and it should be unset. meanwhile the workaround is
> {code}
> describe database extended [db_name];
> {code}
> assuming property is 'bootstrap.dump.state.something'
> {code}
> alter database [db_name] set dbproperties
> ('bootstrap.dump.state.something'='idle');"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20152) reset db state, when repl dump fails, so rename table can be done
[
https://issues.apache.org/jira/browse/HIVE-20152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546151#comment-16546151
]
Hive QA commented on HIVE-20152:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931884/HIVE-20152.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14661 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12650/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12650/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12650/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931884 - PreCommit-HIVE-Build
> reset db state, when repl dump fails, so rename table can be done
> -
>
> Key: HIVE-20152
> URL: https://issues.apache.org/jira/browse/HIVE-20152
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20152.1.patch, HIVE-20152.2.patch
>
>
> If a repl dump command is run and it fails for some reason while doing table
> level dumps, the state set on the db parameters is not reset and hence no
> table / partition renames can be done.
> the property to be reset is prefixed with key {code}bootstrap.dump.state
> {code}
> and it should be unset. meanwhile the workaround is
> {code}
> describe database extended [db_name];
> {code}
> assuming property is 'bootstrap.dump.state.something'
> {code}
> alter database [db_name] set dbproperties
> ('bootstrap.dump.state.something'='idle');"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20179) Some Tez jar-s are not on classpath so HS2 keeps too long to start
[
https://issues.apache.org/jira/browse/HIVE-20179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-20179:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks for the review [~kgyrtkirk]!
> Some Tez jar-s are not on classpath so HS2 keeps too long to start
> --
>
> Key: HIVE-20179
> URL: https://issues.apache.org/jira/browse/HIVE-20179
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20179.2.patch, HIVE-20179.patch
>
>
> aideWhen trying to start HS2 in IDEA without any changes in the pom the
> following exception is thrown:
> {code}
> java.lang.NoClassDefFoundError: org/apache/tez/dag/api/TezConfiguration
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession$AbstractTriggerValidator.startTriggerValidator(TezSessionPoolSession.java:74)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.initTriggers(TezSessionPoolManager.java:207)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:114)
> at
> org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:841)
> at
> org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:824)
> at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:746)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1039)
> at
> org.apache.hive.service.server.HiveServer2.access$1600(HiveServer2.java:140)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1307)
> at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1151)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.tez.dag.api.TezConfiguration
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 10 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-15131) Change Parquet reader to read metadata on the task side
[
https://issues.apache.org/jira/browse/HIVE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adesh Kumar Rao updated HIVE-15131:
---
Status: Patch Available (was: Open)
> Change Parquet reader to read metadata on the task side
> ---
>
> Key: HIVE-15131
> URL: https://issues.apache.org/jira/browse/HIVE-15131
> Project: Hive
> Issue Type: Bug
> Components: Reader
>Reporter: Chao Sun
>Assignee: Adesh Kumar Rao
>Priority: Major
> Attachments: HIVE-15131.1.patch
>
>
> Currently the {{ParquetRecordReaderWrapper}} still uses the {{readFooter}}
> API without filtering, which means it needs to read metadata about all row
> groups every time. This could some issues when input dataset is particularly
> big and has many columns.
> [Parquet-84|https://issues.apache.org/jira/browse/PARQUET-84] introduced
> another API which allows to do row group filtering on the task side. Hive
> should adopt this API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20152) reset db state, when repl dump fails, so rename table can be done
[
https://issues.apache.org/jira/browse/HIVE-20152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anishek updated HIVE-20152:
---
Attachment: HIVE-20152.3.patch
> reset db state, when repl dump fails, so rename table can be done
> -
>
> Key: HIVE-20152
> URL: https://issues.apache.org/jira/browse/HIVE-20152
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20152.1.patch, HIVE-20152.2.patch,
> HIVE-20152.3.patch
>
>
> If a repl dump command is run and it fails for some reason while doing table
> level dumps, the state set on the db parameters is not reset and hence no
> table / partition renames can be done.
> the property to be reset is prefixed with key {code}bootstrap.dump.state
> {code}
> and it should be unset. meanwhile the workaround is
> {code}
> describe database extended [db_name];
> {code}
> assuming property is 'bootstrap.dump.state.something'
> {code}
> alter database [db_name] set dbproperties
> ('bootstrap.dump.state.something'='idle');"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20152) reset db state, when repl dump fails, so rename table can be done
[
https://issues.apache.org/jira/browse/HIVE-20152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546193#comment-16546193
]
anishek commented on HIVE-20152:
[~sankarh] please review !
> reset db state, when repl dump fails, so rename table can be done
> -
>
> Key: HIVE-20152
> URL: https://issues.apache.org/jira/browse/HIVE-20152
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20152.1.patch, HIVE-20152.2.patch,
> HIVE-20152.3.patch
>
>
> If a repl dump command is run and it fails for some reason while doing table
> level dumps, the state set on the db parameters is not reset and hence no
> table / partition renames can be done.
> the property to be reset is prefixed with key {code}bootstrap.dump.state
> {code}
> and it should be unset. meanwhile the workaround is
> {code}
> describe database extended [db_name];
> {code}
> assuming property is 'bootstrap.dump.state.something'
> {code}
> alter database [db_name] set dbproperties
> ('bootstrap.dump.state.something'='idle');"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18705) Improve HiveMetaStoreClient.dropDatabase
[
https://issues.apache.org/jira/browse/HIVE-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546198#comment-16546198
]
Adam Szita commented on HIVE-18705:
---
Got green test, and +1 from Peter, pushed to master now.
Thanks [~pvary] for reviewing.
> Improve HiveMetaStoreClient.dropDatabase
>
>
> Key: HIVE-18705
> URL: https://issues.apache.org/jira/browse/HIVE-18705
> Project: Hive
> Issue Type: Improvement
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-18705.0.patch, HIVE-18705.1.patch,
> HIVE-18705.10.patch, HIVE-18705.11.patch, HIVE-18705.2.patch,
> HIVE-18705.4.patch, HIVE-18705.5.patch, HIVE-18705.6.patch,
> HIVE-18705.7.patch, HIVE-18705.8.patch, HIVE-18705.9.patch
>
>
> {{HiveMetaStoreClient.dropDatabase}} has a strange implementation to ensure
> dealing with client side hooks (for non-native tables e.g. HBase). Currently
> it starts by retrieving all the tables from HMS, and then sends {{dropTable}}
> calls to HMS table-by-table. At the end a {{dropDatabase}} just to be sure :)
> I believe this could be refactored so that it speeds up the dropDB in
> situations where the average table count per DB is very high.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18705) Improve HiveMetaStoreClient.dropDatabase
[
https://issues.apache.org/jira/browse/HIVE-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-18705:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
> Improve HiveMetaStoreClient.dropDatabase
>
>
> Key: HIVE-18705
> URL: https://issues.apache.org/jira/browse/HIVE-18705
> Project: Hive
> Issue Type: Improvement
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-18705.0.patch, HIVE-18705.1.patch,
> HIVE-18705.10.patch, HIVE-18705.11.patch, HIVE-18705.2.patch,
> HIVE-18705.4.patch, HIVE-18705.5.patch, HIVE-18705.6.patch,
> HIVE-18705.7.patch, HIVE-18705.8.patch, HIVE-18705.9.patch
>
>
> {{HiveMetaStoreClient.dropDatabase}} has a strange implementation to ensure
> dealing with client side hooks (for non-native tables e.g. HBase). Currently
> it starts by retrieving all the tables from HMS, and then sends {{dropTable}}
> calls to HMS table-by-table. At the end a {{dropDatabase}} just to be sure :)
> I believe this could be refactored so that it speeds up the dropDB in
> situations where the average table count per DB is very high.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546207#comment-16546207
]
Igor Kryvenko commented on HIVE-17683:
--
Seems like failure is not related
{code}
{code}
I will attach the same patch to check this.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch,
> HIVE-17683.03.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20181) Backport HIVE-20045 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-20181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546215#comment-16546215 ] Daniel Voros commented on HIVE-20181: - Test failure is unrelated and passed locally. > Backport HIVE-20045 to branch-3 > --- > > Key: HIVE-20181 > URL: https://issues.apache.org/jira/browse/HIVE-20181 > Project: Hive > Issue Type: Bug > Components: Configuration >Affects Versions: 3.0.0 >Reporter: Daniel Voros >Assignee: Daniel Voros >Priority: Major > Attachments: HIVE-20181.1.branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[ https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-17896: -- Attachment: HIVE-17896.12.patch > TopNKey: Create a standalone vectorizable TopNKey operator > -- > > Key: HIVE-17896 > URL: https://issues.apache.org/jira/browse/HIVE-17896 > Project: Hive > Issue Type: New Feature > Components: Operators >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-17896.1.patch, HIVE-17896.10.patch, > HIVE-17896.11.patch, HIVE-17896.12.patch, HIVE-17896.3.patch, > HIVE-17896.4.patch, HIVE-17896.5.patch, HIVE-17896.6.patch, > HIVE-17896.7.patch, HIVE-17896.8.patch, HIVE-17896.9.patch > > > For TPC-DS Query27, the TopN operation is delayed by the group-by - the > group-by operator buffers up all the rows before discarding the 99% of the > rows in the TopN Hash within the ReduceSink Operator. > The RS TopN operator is very restrictive as it only supports doing the > filtering on the shuffle keys, but it is better to do this before breaking > the vectors into rows and losing the isRepeating properties. > Adding a TopN Key operator in the physical operator tree allows the following > to happen. > GBY->RS(Top=1) > can become > TNK(1)->GBY->RS(Top=1) > So that, the TopNKey can remove rows before they are buffered into the GBY > and consume memory. > Here's the equivalent implementation in Presto > https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35 > Adding this as a sub-feature of GroupBy prevents further optimizations if the > GBY is on keys "a,b,c" and the TopNKey is on just "a". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[ https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-17896: -- Attachment: (was: HIVE-17896.12.patch) > TopNKey: Create a standalone vectorizable TopNKey operator > -- > > Key: HIVE-17896 > URL: https://issues.apache.org/jira/browse/HIVE-17896 > Project: Hive > Issue Type: New Feature > Components: Operators >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-17896.1.patch, HIVE-17896.10.patch, > HIVE-17896.11.patch, HIVE-17896.3.patch, HIVE-17896.4.patch, > HIVE-17896.5.patch, HIVE-17896.6.patch, HIVE-17896.7.patch, > HIVE-17896.8.patch, HIVE-17896.9.patch > > > For TPC-DS Query27, the TopN operation is delayed by the group-by - the > group-by operator buffers up all the rows before discarding the 99% of the > rows in the TopN Hash within the ReduceSink Operator. > The RS TopN operator is very restrictive as it only supports doing the > filtering on the shuffle keys, but it is better to do this before breaking > the vectors into rows and losing the isRepeating properties. > Adding a TopN Key operator in the physical operator tree allows the following > to happen. > GBY->RS(Top=1) > can become > TNK(1)->GBY->RS(Top=1) > So that, the TopNKey can remove rows before they are buffered into the GBY > and consume memory. > Here's the equivalent implementation in Presto > https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35 > Adding this as a sub-feature of GroupBy prevents further optimizations if the > GBY is on keys "a,b,c" and the TopNKey is on just "a". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[ https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-17896: -- Attachment: HIVE-17896.12.patch > TopNKey: Create a standalone vectorizable TopNKey operator > -- > > Key: HIVE-17896 > URL: https://issues.apache.org/jira/browse/HIVE-17896 > Project: Hive > Issue Type: New Feature > Components: Operators >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-17896.1.patch, HIVE-17896.10.patch, > HIVE-17896.11.patch, HIVE-17896.12.patch, HIVE-17896.3.patch, > HIVE-17896.4.patch, HIVE-17896.5.patch, HIVE-17896.6.patch, > HIVE-17896.7.patch, HIVE-17896.8.patch, HIVE-17896.9.patch > > > For TPC-DS Query27, the TopN operation is delayed by the group-by - the > group-by operator buffers up all the rows before discarding the 99% of the > rows in the TopN Hash within the ReduceSink Operator. > The RS TopN operator is very restrictive as it only supports doing the > filtering on the shuffle keys, but it is better to do this before breaking > the vectors into rows and losing the isRepeating properties. > Adding a TopN Key operator in the physical operator tree allows the following > to happen. > GBY->RS(Top=1) > can become > TNK(1)->GBY->RS(Top=1) > So that, the TopNKey can remove rows before they are buffered into the GBY > and consume memory. > Here's the equivalent implementation in Presto > https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35 > Adding this as a sub-feature of GroupBy prevents further optimizations if the > GBY is on keys "a,b,c" and the TopNKey is on just "a". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12342) Set default value of hive.optimize.index.filter to true
[
https://issues.apache.org/jira/browse/HIVE-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546233#comment-16546233
]
Igor Kryvenko commented on HIVE-12342:
--
[~djaiswal] I don't think that it changes product behavior. It wasn't done to
make tests pass it was done because it made unstably execution plan for the
{{TezCompiler}} . This change affects nothing except Tez. As far as I
understand it doesn't regress performance or anything else, anyway it looks odd
that we have the different plans for the same query.
Thanks, Igor.
> Set default value of hive.optimize.index.filter to true
> ---
>
> Key: HIVE-12342
> URL: https://issues.apache.org/jira/browse/HIVE-12342
> Project: Hive
> Issue Type: Task
> Components: Configuration
>Reporter: Ashutosh Chauhan
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-12342.05.patch, HIVE-12342.06.patch,
> HIVE-12342.07.patch, HIVE-12342.08.patch, HIVE-12342.09.patch,
> HIVE-12342.1.patch, HIVE-12342.10.patch, HIVE-12342.11.patch,
> HIVE-12342.12.patch, HIVE-12342.13.patch, HIVE-12342.14.patch,
> HIVE-12342.15.patch, HIVE-12342.16.patch, HIVE-12342.17.patch,
> HIVE-12342.18.patch, HIVE-12342.19.patch, HIVE-12342.2.patch,
> HIVE-12342.20.patch, HIVE-12342.21.patch, HIVE-12342.22.patch,
> HIVE-12342.23.patch, HIVE-12342.24.patch, HIVE-12342.3.patch,
> HIVE-12342.4.patch, HIVE-12342.patch
>
>
> This configuration governs ppd for storage layer. When applicable, it will
> always help. It should be on by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15131) Change Parquet reader to read metadata on the task side
[
https://issues.apache.org/jira/browse/HIVE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546237#comment-16546237
]
Hive QA commented on HIVE-15131:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 2273 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 1 new + 7 unchanged - 0 fixed
= 8 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12651/dev-support/hive-personality.sh
|
| git revision | master / 3e02354 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12651/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12651/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Change Parquet reader to read metadata on the task side
> ---
>
> Key: HIVE-15131
> URL: https://issues.apache.org/jira/browse/HIVE-15131
> Project: Hive
> Issue Type: Bug
> Components: Reader
>Reporter: Chao Sun
>Assignee: Adesh Kumar Rao
>Priority: Major
> Attachments: HIVE-15131.1.patch
>
>
> Currently the {{ParquetRecordReaderWrapper}} still uses the {{readFooter}}
> API without filtering, which means it needs to read metadata about all row
> groups every time. This could some issues when input dataset is particularly
> big and has many columns.
> [Parquet-84|https://issues.apache.org/jira/browse/PARQUET-84] introduced
> another API which allows to do row group filtering on the task side. Hive
> should adopt this API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546207#comment-16546207
]
Igor Kryvenko edited comment on HIVE-17683 at 7/17/18 9:26 AM:
---
Seems like failure is not related
{code:java}
{code}
I will attach the same patch to check this.
Locally it also works fine:
{code}
[INFO] Running org.apache.hive.beeline.TestBeeLineWithArgs
[WARNING] Tests run: 54, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
237.004 s - in org.apache.hive.beeline.TestBeeLineWithArgs
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 54, Failures: 0, Errors: 0, Skipped: 2
[INFO]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 05:00 min
[INFO] Finished at: 2018-07-17T12:25:43+03:00
[INFO] Final Memory: 136M/2524M
[INFO]
{code}
was (Author: ikryvenko):
Seems like failure is not related
{code}
{code}
I will attach the same patch to check this.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch,
> HIVE-17683.03.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Kryvenko updated HIVE-17683: - Attachment: HIVE-17683.04.patch > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch, > HIVE-17683.03.patch, HIVE-17683.04.patch > > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19486) Discrepancy in HikariCP config naming
[ https://issues.apache.org/jira/browse/HIVE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546253#comment-16546253 ] Adam Szita commented on HIVE-19486: --- Committed to master, thanks [~asinkovits] > Discrepancy in HikariCP config naming > - > > Key: HIVE-19486 > URL: https://issues.apache.org/jira/browse/HIVE-19486 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-19486.1.patch, HIVE-19486.2.patch > > > HiveConf hive.conf.restricted.list contains "hikari." instead of "hikaricp." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19486) Discrepancy in HikariCP config naming
[ https://issues.apache.org/jira/browse/HIVE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated HIVE-19486: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) > Discrepancy in HikariCP config naming > - > > Key: HIVE-19486 > URL: https://issues.apache.org/jira/browse/HIVE-19486 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19486.1.patch, HIVE-19486.2.patch > > > HiveConf hive.conf.restricted.list contains "hikari." instead of "hikaricp." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20191) PreCommit patch application doesn't fail if patch is empty
[
https://issues.apache.org/jira/browse/HIVE-20191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Voros reassigned HIVE-20191:
---
> PreCommit patch application doesn't fail if patch is empty
> --
>
> Key: HIVE-20191
> URL: https://issues.apache.org/jira/browse/HIVE-20191
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
>
> I've created some backport tickets to branch-3 (e.g. HIVE-20181) and made the
> mistake of uploading the patch files with wrong filename ({{.} instead of
> {{-}} between version and branch).
> These get applied on master, where they're already present, since {{git
> apply}} with {{-3}} won't fail if patch is already there. Tests are run on
> master instead of failing.
> I think the patch application should fail if the patch is empty and branch
> selection logic should probably fail too if the patch name is malformed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20192) HS2 is leaking JDOPersistenceManager objects.
[ https://issues.apache.org/jira/browse/HIVE-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan reassigned HIVE-20192: --- > HS2 is leaking JDOPersistenceManager objects. > - > > Key: HIVE-20192 > URL: https://issues.apache.org/jira/browse/HIVE-20192 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0, 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: HiveServer2 > > Hiveserver2 instances where every 3-4 days they are observing HS2 in an > unresponsive state, we observed that the FGC collection happening regularly > From JXray report it is seen that pmCache(List of JDOPersistenceManager > objects) is occupying 84% of the heap and there are around 16,000 references > of UDFClassLoader. > When the RawStore object is re-created, it is not allowed to be updated into > the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new > RawStore never gets cleaned-up when the thread exit. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20180) Backport HIVE-19759 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-20180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Voros updated HIVE-20180: Attachment: HIVE-20180.2-branch-3.patch > Backport HIVE-19759 to branch-3 > --- > > Key: HIVE-20180 > URL: https://issues.apache.org/jira/browse/HIVE-20180 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 3.0.0 >Reporter: Daniel Voros >Assignee: Daniel Voros >Priority: Major > Attachments: HIVE-20180.1.branch-3.patch, HIVE-20180.2-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20180) Backport HIVE-19759 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-20180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546272#comment-16546272 ] Daniel Voros commented on HIVE-20180: - Please ignore test results, first patch had wrong filename and tests were executed against master (see HIVE-20191). > Backport HIVE-19759 to branch-3 > --- > > Key: HIVE-20180 > URL: https://issues.apache.org/jira/browse/HIVE-20180 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 3.0.0 >Reporter: Daniel Voros >Assignee: Daniel Voros >Priority: Major > Attachments: HIVE-20180.1.branch-3.patch, HIVE-20180.2-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20181) Backport HIVE-20045 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-20181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Voros updated HIVE-20181: Attachment: HIVE-20181.2-branch-3.patch > Backport HIVE-20045 to branch-3 > --- > > Key: HIVE-20181 > URL: https://issues.apache.org/jira/browse/HIVE-20181 > Project: Hive > Issue Type: Bug > Components: Configuration >Affects Versions: 3.0.0 >Reporter: Daniel Voros >Assignee: Daniel Voros >Priority: Major > Attachments: HIVE-20181.1.branch-3.patch, HIVE-20181.2-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20182) Backport HIVE-20067 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-20182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546276#comment-16546276 ] Daniel Voros commented on HIVE-20182: - Please ignore test results, first patch had wrong filename and tests were executed against master (see HIVE-20191). > Backport HIVE-20067 to branch-3 > --- > > Key: HIVE-20182 > URL: https://issues.apache.org/jira/browse/HIVE-20182 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.0.0 >Reporter: Daniel Voros >Assignee: Daniel Voros >Priority: Major > Attachments: HIVE-20182.1.branch-3.patch, HIVE-20182.2-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20182) Backport HIVE-20067 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-20182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Voros updated HIVE-20182: Attachment: HIVE-20182.2-branch-3.patch > Backport HIVE-20067 to branch-3 > --- > > Key: HIVE-20182 > URL: https://issues.apache.org/jira/browse/HIVE-20182 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.0.0 >Reporter: Daniel Voros >Assignee: Daniel Voros >Priority: Major > Attachments: HIVE-20182.1.branch-3.patch, HIVE-20182.2-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20181) Backport HIVE-20045 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-20181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546275#comment-16546275 ] Daniel Voros commented on HIVE-20181: - Please ignore test results, first patch had wrong filename and tests were executed against master (see HIVE-20191). > Backport HIVE-20045 to branch-3 > --- > > Key: HIVE-20181 > URL: https://issues.apache.org/jira/browse/HIVE-20181 > Project: Hive > Issue Type: Bug > Components: Configuration >Affects Versions: 3.0.0 >Reporter: Daniel Voros >Assignee: Daniel Voros >Priority: Major > Attachments: HIVE-20181.1.branch-3.patch, HIVE-20181.2-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20152) reset db state, when repl dump fails, so rename table can be done
[
https://issues.apache.org/jira/browse/HIVE-20152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546274#comment-16546274
]
Sankar Hariappan commented on HIVE-20152:
-
+1, pending tests.
> reset db state, when repl dump fails, so rename table can be done
> -
>
> Key: HIVE-20152
> URL: https://issues.apache.org/jira/browse/HIVE-20152
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20152.1.patch, HIVE-20152.2.patch,
> HIVE-20152.3.patch
>
>
> If a repl dump command is run and it fails for some reason while doing table
> level dumps, the state set on the db parameters is not reset and hence no
> table / partition renames can be done.
> the property to be reset is prefixed with key {code}bootstrap.dump.state
> {code}
> and it should be unset. meanwhile the workaround is
> {code}
> describe database extended [db_name];
> {code}
> assuming property is 'bootstrap.dump.state.something'
> {code}
> alter database [db_name] set dbproperties
> ('bootstrap.dump.state.something'='idle');"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20184) Backport HIVE-20085 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546278#comment-16546278 ] Daniel Voros commented on HIVE-20184: - Please ignore test results, first patch had wrong filename and tests were executed against master (see HIVE-20191). > Backport HIVE-20085 to branch-3 > --- > > Key: HIVE-20184 > URL: https://issues.apache.org/jira/browse/HIVE-20184 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Daniel Voros >Assignee: Daniel Voros >Priority: Major > Attachments: HIVE-20184.1.branch-3.patch, HIVE-20184.2-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20184) Backport HIVE-20085 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Voros updated HIVE-20184: Attachment: HIVE-20184.2-branch-3.patch > Backport HIVE-20085 to branch-3 > --- > > Key: HIVE-20184 > URL: https://issues.apache.org/jira/browse/HIVE-20184 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Daniel Voros >Assignee: Daniel Voros >Priority: Major > Attachments: HIVE-20184.1.branch-3.patch, HIVE-20184.2-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20185) Backport HIVE-20111 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-20185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546280#comment-16546280 ] Daniel Voros commented on HIVE-20185: - Please ignore test results, first patch had wrong filename and tests were executed against master (see HIVE-20191). > Backport HIVE-20111 to branch-3 > --- > > Key: HIVE-20185 > URL: https://issues.apache.org/jira/browse/HIVE-20185 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Daniel Voros >Assignee: Daniel Voros >Priority: Major > Attachments: HIVE-20185.1.branch-3.patch, HIVE-20185.2-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20185) Backport HIVE-20111 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-20185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Voros updated HIVE-20185: Attachment: HIVE-20185.2-branch-3.patch > Backport HIVE-20111 to branch-3 > --- > > Key: HIVE-20185 > URL: https://issues.apache.org/jira/browse/HIVE-20185 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Daniel Voros >Assignee: Daniel Voros >Priority: Major > Attachments: HIVE-20185.1.branch-3.patch, HIVE-20185.2-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20191) PreCommit patch application doesn't fail if patch is empty
[
https://issues.apache.org/jira/browse/HIVE-20191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Voros updated HIVE-20191:
Description:
I've created some backport tickets to branch-3 (e.g. HIVE-20181) and made the
mistake of uploading the patch files with wrong filename ({{.}} instead of
{{-}} between version and branch).
These get applied on master, where they're already present, since {{git apply}}
with {{-3}} won't fail if patch is already there. Tests are run on master
instead of failing.
I think the patch application should fail if the patch is empty and branch
selection logic should probably fail too if the patch name is malformed.
was:
I've created some backport tickets to branch-3 (e.g. HIVE-20181) and made the
mistake of uploading the patch files with wrong filename ({{.} instead of {{-}}
between version and branch).
These get applied on master, where they're already present, since {{git apply}}
with {{-3}} won't fail if patch is already there. Tests are run on master
instead of failing.
I think the patch application should fail if the patch is empty and branch
selection logic should probably fail too if the patch name is malformed.
> PreCommit patch application doesn't fail if patch is empty
> --
>
> Key: HIVE-20191
> URL: https://issues.apache.org/jira/browse/HIVE-20191
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
>
> I've created some backport tickets to branch-3 (e.g. HIVE-20181) and made the
> mistake of uploading the patch files with wrong filename ({{.}} instead of
> {{-}} between version and branch).
> These get applied on master, where they're already present, since {{git
> apply}} with {{-3}} won't fail if patch is already there. Tests are run on
> master instead of failing.
> I think the patch application should fail if the patch is empty and branch
> selection logic should probably fail too if the patch name is malformed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20192) HS2 is leaking JDOPersistenceManager objects.
[
https://issues.apache.org/jira/browse/HIVE-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20192:
Description:
Hiveserver2 instances where every 3-4 days they are observing HS2 in an
unresponsive state, we observed that the FGC collection happening regularly
>From JXray report it is seen that pmCache(List of JDOPersistenceManager
>objects) is occupying 84% of the heap and there are around 16,000 references
>of UDFClassLoader.
{code:java}
10,759,230K (84.7%) Object tree for GC root(s) Java Static
org.apache.hadoop.hive.metastore.ObjectStore.pmf
- org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K
(84.6%), 1 reference(s)
- j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
- {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%),
16,872 reference(s)
- org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K (84.5%),
16,872 reference(s)
... 3 more references together retaining 4,933K (< 0.1%)
- java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
... 2 more references together retaining 48b (< 0.1%)
- org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘ 14,810K
(0.1%), 1 reference(s)
... 3 more references together retaining 96b (< 0.1%){code}
When the RawStore object is re-created, it is not allowed to be updated into
the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new RawStore
never gets cleaned-up when the thread exit.
was:
Hiveserver2 instances where every 3-4 days they are observing HS2 in an
unresponsive state, we observed that the FGC collection happening regularly
>From JXray report it is seen that pmCache(List of JDOPersistenceManager
>objects) is occupying 84% of the heap and there are around 16,000 references
>of UDFClassLoader.
When the RawStore object is re-created, it is not allowed to be updated into
the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new RawStore
never gets cleaned-up when the thread exit.
> HS2 is leaking JDOPersistenceManager objects.
> -
>
> Key: HIVE-20192
> URL: https://issues.apache.org/jira/browse/HIVE-20192
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: HiveServer2
>
> Hiveserver2 instances where every 3-4 days they are observing HS2 in an
> unresponsive state, we observed that the FGC collection happening regularly
> From JXray report it is seen that pmCache(List of JDOPersistenceManager
> objects) is occupying 84% of the heap and there are around 16,000 references
> of UDFClassLoader.
> {code:java}
> 10,759,230K (84.7%) Object tree for GC root(s) Java Static
> org.apache.hadoop.hive.metastore.ObjectStore.pmf
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K
> (84.6%), 1 reference(s)
> - j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
> - {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%),
> 16,872 reference(s)
> - org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K
> (84.5%), 16,872 reference(s)
> ... 3 more references together retaining 4,933K (< 0.1%)
> - java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
> ... 2 more references together retaining 48b (< 0.1%)
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘
> 14,810K (0.1%), 1 reference(s)
> ... 3 more references together retaining 96b (< 0.1%){code}
> When the RawStore object is re-created, it is not allowed to be updated into
> the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new
> RawStore never gets cleaned-up when the thread exit.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15131) Change Parquet reader to read metadata on the task side
[
https://issues.apache.org/jira/browse/HIVE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546293#comment-16546293
]
Hive QA commented on HIVE-15131:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931881/HIVE-15131.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14662 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_analyze]
(batchId=24)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=153)
org.apache.hive.minikdc.TestHs2HooksWithMiniKdc.testHookContexts (batchId=263)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12651/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12651/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12651/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931881 - PreCommit-HIVE-Build
> Change Parquet reader to read metadata on the task side
> ---
>
> Key: HIVE-15131
> URL: https://issues.apache.org/jira/browse/HIVE-15131
> Project: Hive
> Issue Type: Bug
> Components: Reader
>Reporter: Chao Sun
>Assignee: Adesh Kumar Rao
>Priority: Major
> Attachments: HIVE-15131.1.patch
>
>
> Currently the {{ParquetRecordReaderWrapper}} still uses the {{readFooter}}
> API without filtering, which means it needs to read metadata about all row
> groups every time. This could some issues when input dataset is particularly
> big and has many columns.
> [Parquet-84|https://issues.apache.org/jira/browse/PARQUET-84] introduced
> another API which allows to do row group filtering on the task side. Hive
> should adopt this API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20192) HS2 is leaking JDOPersistenceManager objects.
[
https://issues.apache.org/jira/browse/HIVE-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20192:
Description:
Hiveserver2 instances where crashing every 3-4 days and observed HS2 in on
unresponsive state. Also, observed that the FGC collection happening regularly
>From JXray report it is seen that pmCache(List of JDOPersistenceManager
>objects) is occupying 84% of the heap and there are around 16,000 references
>of UDFClassLoader.
{code:java}
10,759,230K (84.7%) Object tree for GC root(s) Java Static
org.apache.hadoop.hive.metastore.ObjectStore.pmf
- org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K
(84.6%), 1 reference(s)
- j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
- {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%),
16,872 reference(s)
- org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K (84.5%),
16,872 reference(s)
... 3 more references together retaining 4,933K (< 0.1%)
- java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
... 2 more references together retaining 48b (< 0.1%)
- org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘ 14,810K
(0.1%), 1 reference(s)
... 3 more references together retaining 96b (< 0.1%){code}
When the RawStore object is re-created, it is not allowed to be updated into
the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new RawStore
never gets cleaned-up when the thread exit.
was:
Hiveserver2 instances where every 3-4 days they are observing HS2 in an
unresponsive state, we observed that the FGC collection happening regularly
>From JXray report it is seen that pmCache(List of JDOPersistenceManager
>objects) is occupying 84% of the heap and there are around 16,000 references
>of UDFClassLoader.
{code:java}
10,759,230K (84.7%) Object tree for GC root(s) Java Static
org.apache.hadoop.hive.metastore.ObjectStore.pmf
- org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K
(84.6%), 1 reference(s)
- j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
- {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%),
16,872 reference(s)
- org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K (84.5%),
16,872 reference(s)
... 3 more references together retaining 4,933K (< 0.1%)
- java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
... 2 more references together retaining 48b (< 0.1%)
- org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘ 14,810K
(0.1%), 1 reference(s)
... 3 more references together retaining 96b (< 0.1%){code}
When the RawStore object is re-created, it is not allowed to be updated into
the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new RawStore
never gets cleaned-up when the thread exit.
> HS2 is leaking JDOPersistenceManager objects.
> -
>
> Key: HIVE-20192
> URL: https://issues.apache.org/jira/browse/HIVE-20192
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: HiveServer2
>
> Hiveserver2 instances where crashing every 3-4 days and observed HS2 in on
> unresponsive state. Also, observed that the FGC collection happening regularly
> From JXray report it is seen that pmCache(List of JDOPersistenceManager
> objects) is occupying 84% of the heap and there are around 16,000 references
> of UDFClassLoader.
> {code:java}
> 10,759,230K (84.7%) Object tree for GC root(s) Java Static
> org.apache.hadoop.hive.metastore.ObjectStore.pmf
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K
> (84.6%), 1 reference(s)
> - j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
> - {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%),
> 16,872 reference(s)
> - org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K
> (84.5%), 16,872 reference(s)
> ... 3 more references together retaining 4,933K (< 0.1%)
> - java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
> ... 2 more references together retaining 48b (< 0.1%)
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘
> 14,810K (0.1%), 1 reference(s)
> ... 3 more references together retaining 96b (< 0.1%){code}
> When the RawStore object is re-created, it is not allowed to be updated into
> the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new
> RawStore never gets cleaned-up when the thread exit.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20152) reset db state, when repl dump fails, so rename table can be done
[
https://issues.apache.org/jira/browse/HIVE-20152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anishek updated HIVE-20152:
---
Attachment: HIVE-20152.4.patch
> reset db state, when repl dump fails, so rename table can be done
> -
>
> Key: HIVE-20152
> URL: https://issues.apache.org/jira/browse/HIVE-20152
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20152.1.patch, HIVE-20152.2.patch,
> HIVE-20152.3.patch, HIVE-20152.4.patch
>
>
> If a repl dump command is run and it fails for some reason while doing table
> level dumps, the state set on the db parameters is not reset and hence no
> table / partition renames can be done.
> the property to be reset is prefixed with key {code}bootstrap.dump.state
> {code}
> and it should be unset. meanwhile the workaround is
> {code}
> describe database extended [db_name];
> {code}
> assuming property is 'bootstrap.dump.state.something'
> {code}
> alter database [db_name] set dbproperties
> ('bootstrap.dump.state.something'='idle');"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20152) reset db state, when repl dump fails, so rename table can be done
[
https://issues.apache.org/jira/browse/HIVE-20152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546339#comment-16546339
]
Hive QA commented on HIVE-20152:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
25s{color} | {color:blue} ql in master has 2273 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 3 new + 14 unchanged - 0 fixed
= 17 total (was 14) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
36s{color} | {color:green} ql generated 0 new + 2272 unchanged - 1 fixed = 2272
total (was 2273) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12652/dev-support/hive-personality.sh
|
| git revision | master / dceeefb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12652/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12652/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> reset db state, when repl dump fails, so rename table can be done
> -
>
> Key: HIVE-20152
> URL: https://issues.apache.org/jira/browse/HIVE-20152
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20152.1.patch, HIVE-20152.2.patch,
> HIVE-20152.3.patch, HIVE-20152.4.patch
>
>
> If a repl dump command is run and it fails for some reason while doing table
> level dumps, the state set on the db parameters is not reset and hence no
> table / partition renames can be done.
> the property to be reset is prefixed with key {code}bootstrap.dump.state
> {code}
> and it should be unset. meanwhile the workaround is
> {code}
> describe database extended [db_name];
> {code}
> assuming property is 'bootstrap.dump.state.something'
> {code}
> alter database [db_name] set dbproperties
> ('bootstrap.dump.state.something'='idle');"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20152) reset db state, when repl dump fails, so rename table can be done
[
https://issues.apache.org/jira/browse/HIVE-20152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546417#comment-16546417
]
Hive QA commented on HIVE-20152:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931897/HIVE-20152.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14662 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12652/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12652/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12652/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931897 - PreCommit-HIVE-Build
> reset db state, when repl dump fails, so rename table can be done
> -
>
> Key: HIVE-20152
> URL: https://issues.apache.org/jira/browse/HIVE-20152
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20152.1.patch, HIVE-20152.2.patch,
> HIVE-20152.3.patch, HIVE-20152.4.patch
>
>
> If a repl dump command is run and it fails for some reason while doing table
> level dumps, the state set on the db parameters is not reset and hence no
> table / partition renames can be done.
> the property to be reset is prefixed with key {code}bootstrap.dump.state
> {code}
> and it should be unset. meanwhile the workaround is
> {code}
> describe database extended [db_name];
> {code}
> assuming property is 'bootstrap.dump.state.something'
> {code}
> alter database [db_name] set dbproperties
> ('bootstrap.dump.state.something'='idle');"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[
https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546489#comment-16546489
]
Hive QA commented on HIVE-17896:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} serde in master has 194 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
18s{color} | {color:blue} ql in master has 2273 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 32 new + 430 unchanged - 0
fixed = 462 total (was 430) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
53s{color} | {color:red} serde generated 1 new + 194 unchanged - 0 fixed = 195
total (was 194) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
41s{color} | {color:red} ql generated 8 new + 2273 unchanged - 0 fixed = 2281
total (was 2273) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:serde |
| |
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.compare(Object[],
ObjectInspector[], Object[], ObjectInspector[], boolean[]) negates the return
value of
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.compare(Object,
ObjectInspector, Object, ObjectInspector) At
ObjectInspectorUtils.java:negates the return value of
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.compare(Object,
ObjectInspector, Object, ObjectInspector) At ObjectInspectorUtils.java:[line
956] |
| FindBugs | module:ql |
| | new
org.apache.hadoop.hive.ql.exec.TopNKeyOperator$KeyWrapperComparator(ObjectInspector[],
ObjectInspector[], boolean[]) may expose internal representation by storing an
externally mutable object into
TopNKeyOperator$KeyWrapperComparator.columnSortOrderIsDesc At
TopNKeyOperator.java:expose internal representation by storing an externally
mutable object into TopNKeyOperator$KeyWrapperComparator.columnSortOrderIsDesc
At TopNKeyOperator.java:[line 83] |
| | new
org.apache.hadoop.hive.ql.exec.TopNKeyOperator$KeyWrapperComparator(ObjectInspector[],
ObjectInspector[], boolean[]) may expose internal representation by storing an
externally mutable object into
TopNKeyOperator$KeyWrapperComparator.objectInspectors1 At
TopNKeyOperator.java:expose internal representation by storing an externally
mutable object into TopNKeyOperator$KeyWrapperComparat
[jira] [Updated] (HIVE-20183) Inserting from bucketed table can cause data loss, if the source table contains empty buckets
[
https://issues.apache.org/jira/browse/HIVE-20183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-20183:
--
Attachment: HIVE-20183.2.patch
> Inserting from bucketed table can cause data loss, if the source table
> contains empty buckets
> -
>
> Key: HIVE-20183
> URL: https://issues.apache.org/jira/browse/HIVE-20183
> Project: Hive
> Issue Type: Bug
> Components: Operators
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20183.2.patch, HIVE-20183.patch
>
>
> Could be reproduced by the following:
> {code}
> set hive.enforce.bucketing=true;
> set hive.enforce.sorting=true;
> set hive.optimize.bucketingsorting=true;
> create table bucket1 (id int, val string) clustered by (id) sorted by (id
> ASC) INTO 4 BUCKETS;
> insert into bucket1 values (1, 'abc'), (3, 'abc');
> select * from bucket1;
> +-+--+
> | bucket1.id | bucket1.val |
> +-+--+
> | 3 | abc |
> | 1 | abc |
> +-+--+
> create table bucket2 like bucket1;
> insert overwrite table bucket2 select * from bucket1;
> select * from bucket2;
> +-+--+
> | bucket2.id | bucket2.val |
> +-+--+
> | 1 | abc |
> +-+--+
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20183) Inserting from bucketed table can cause data loss, if the source table contains empty buckets
[
https://issues.apache.org/jira/browse/HIVE-20183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546495#comment-16546495
]
Peter Vary commented on HIVE-20183:
---
Trying from another direction
> Inserting from bucketed table can cause data loss, if the source table
> contains empty buckets
> -
>
> Key: HIVE-20183
> URL: https://issues.apache.org/jira/browse/HIVE-20183
> Project: Hive
> Issue Type: Bug
> Components: Operators
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20183.2.patch, HIVE-20183.patch
>
>
> Could be reproduced by the following:
> {code}
> set hive.enforce.bucketing=true;
> set hive.enforce.sorting=true;
> set hive.optimize.bucketingsorting=true;
> create table bucket1 (id int, val string) clustered by (id) sorted by (id
> ASC) INTO 4 BUCKETS;
> insert into bucket1 values (1, 'abc'), (3, 'abc');
> select * from bucket1;
> +-+--+
> | bucket1.id | bucket1.val |
> +-+--+
> | 3 | abc |
> | 1 | abc |
> +-+--+
> create table bucket2 like bucket1;
> insert overwrite table bucket2 select * from bucket1;
> select * from bucket2;
> +-+--+
> | bucket2.id | bucket2.val |
> +-+--+
> | 1 | abc |
> +-+--+
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[
https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546510#comment-16546510
]
Hive QA commented on HIVE-17896:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931901/HIVE-17896.12.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 48 failed/errored test(s), 14668 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_struct_type_vectorization]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_complex_types_vectorization]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_map_type_vectorization]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_struct_type_vectorization]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_groupby]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[limit_pushdown3]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[limit_pushdown]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_decimal64_reader]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[offset_limit]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[offset_limit_ppd_optimizer]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_cast_constant]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_char_2]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_limit]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_mr_diff_schema_alias]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_decimal]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_string_concat]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_limit]
(batchId=165)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query10]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query15]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query17]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query25]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query26]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query27]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query29]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query35]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query37]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query40]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query43]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query45]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query49]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query50]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query5]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query60]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query66]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query69]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query76]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query77]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query7]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query80]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query82]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query8]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query99]
(
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546588#comment-16546588
]
Hive QA commented on HIVE-17683:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
17s{color} | {color:blue} standalone-metastore/metastore-common in master has
218 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
25s{color} | {color:blue} ql in master has 2273 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 5 new + 276 unchanged - 35
fixed = 281 total (was 311) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12654/dev-support/hive-personality.sh
|
| git revision | master / dceeefb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12654/yetus/diff-checkstyle-ql.txt
|
| modules | C: standalone-metastore/metastore-common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12654/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch,
> HIVE-17683.03.patch, HIVE-17683.04.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546665#comment-16546665
]
Hive QA commented on HIVE-17683:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931907/HIVE-17683.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14648 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=106)
[update_orig_table.q,explainanalyze_3.q,orc_vectorization_ppd.q,vector_non_string_partition.q,multi_count_distinct.q,explainanalyze_1.q,explainuser_3.q,explainanalyze_4.q,hybridgrace_hashjoin_1.q,delete_orig_table.q,hybridgrace_hashjoin_2.q,tez-tag.q,orc_merge12.q,explainanalyze_5.q,vector_join_part_col_char.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12654/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12654/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12654/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931907 - PreCommit-HIVE-Build
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch,
> HIVE-17683.03.patch, HIVE-17683.04.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20152) reset db state, when repl dump fails, so rename table can be done
[
https://issues.apache.org/jira/browse/HIVE-20152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anishek updated HIVE-20152:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch committed to master, Thanks for the review [~sankarh]
> reset db state, when repl dump fails, so rename table can be done
> -
>
> Key: HIVE-20152
> URL: https://issues.apache.org/jira/browse/HIVE-20152
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20152.1.patch, HIVE-20152.2.patch,
> HIVE-20152.3.patch, HIVE-20152.4.patch
>
>
> If a repl dump command is run and it fails for some reason while doing table
> level dumps, the state set on the db parameters is not reset and hence no
> table / partition renames can be done.
> the property to be reset is prefixed with key {code}bootstrap.dump.state
> {code}
> and it should be unset. meanwhile the workaround is
> {code}
> describe database extended [db_name];
> {code}
> assuming property is 'bootstrap.dump.state.something'
> {code}
> alter database [db_name] set dbproperties
> ('bootstrap.dump.state.something'='idle');"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20152) reset db state, when repl dump fails, so rename table can be done
[
https://issues.apache.org/jira/browse/HIVE-20152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anishek updated HIVE-20152:
---
Fix Version/s: 4.0.0
> reset db state, when repl dump fails, so rename table can be done
> -
>
> Key: HIVE-20152
> URL: https://issues.apache.org/jira/browse/HIVE-20152
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20152.1.patch, HIVE-20152.2.patch,
> HIVE-20152.3.patch, HIVE-20152.4.patch
>
>
> If a repl dump command is run and it fails for some reason while doing table
> level dumps, the state set on the db parameters is not reset and hence no
> table / partition renames can be done.
> the property to be reset is prefixed with key {code}bootstrap.dump.state
> {code}
> and it should be unset. meanwhile the workaround is
> {code}
> describe database extended [db_name];
> {code}
> assuming property is 'bootstrap.dump.state.something'
> {code}
> alter database [db_name] set dbproperties
> ('bootstrap.dump.state.something'='idle');"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20193) cboInfo is not present in the explain plan json
[ https://issues.apache.org/jira/browse/HIVE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek reassigned HIVE-20193: -- Assignee: anishek > cboInfo is not present in the explain plan json > --- > > Key: HIVE-20193 > URL: https://issues.apache.org/jira/browse/HIVE-20193 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek >Priority: Major > > *cboInfo* attribute is not present in the explain plan json that is provided > to the pre exec hook in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20180) Backport HIVE-19759 to branch-3
[
https://issues.apache.org/jira/browse/HIVE-20180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546683#comment-16546683
]
Hive QA commented on HIVE-20180:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-12655/patches/PreCommit-HIVE-Build-12655.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12655/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Backport HIVE-19759 to branch-3
> ---
>
> Key: HIVE-20180
> URL: https://issues.apache.org/jira/browse/HIVE-20180
> Project: Hive
> Issue Type: Bug
> Components: Test
>Affects Versions: 3.0.0
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-20180.1.branch-3.patch, HIVE-20180.2-branch-3.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546710#comment-16546710 ] Igor Kryvenko commented on HIVE-17683: -- [~ekoifman] Hi Eugene. I did changes which you requested in your review, please review it when you will have time for it. Seems like failing of tests is not related again. Thanks, Igor. > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch, > HIVE-17683.03.patch, HIVE-17683.04.patch > > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20192) HS2 is leaking JDOPersistenceManager objects.
[
https://issues.apache.org/jira/browse/HIVE-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546802#comment-16546802
]
Vihang Karajgaonkar commented on HIVE-20192:
Does this happen when HMS is embedded in HS2? In case of remote HMS we avoid
leaking {{JDOPersistenceManager}} by issuing a {{cleanupRawStore}} in
{{deleteContext}}. May be we need to the same in HS2 by detecting if HMS is in
embedded mode.
> HS2 is leaking JDOPersistenceManager objects.
> -
>
> Key: HIVE-20192
> URL: https://issues.apache.org/jira/browse/HIVE-20192
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: HiveServer2
>
> Hiveserver2 instances where crashing every 3-4 days and observed HS2 in on
> unresponsive state. Also, observed that the FGC collection happening regularly
> From JXray report it is seen that pmCache(List of JDOPersistenceManager
> objects) is occupying 84% of the heap and there are around 16,000 references
> of UDFClassLoader.
> {code:java}
> 10,759,230K (84.7%) Object tree for GC root(s) Java Static
> org.apache.hadoop.hive.metastore.ObjectStore.pmf
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K
> (84.6%), 1 reference(s)
> - j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
> - {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%),
> 16,872 reference(s)
> - org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K
> (84.5%), 16,872 reference(s)
> ... 3 more references together retaining 4,933K (< 0.1%)
> - java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
> ... 2 more references together retaining 48b (< 0.1%)
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘
> 14,810K (0.1%), 1 reference(s)
> ... 3 more references together retaining 96b (< 0.1%){code}
> When the RawStore object is re-created, it is not allowed to be updated into
> the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new
> RawStore never gets cleaned-up when the thread exit.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19166) TestMiniLlapLocalCliDriver sysdb failure
[ https://issues.apache.org/jira/browse/HIVE-19166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-19166: -- Attachment: HIVE-19166.15.patch > TestMiniLlapLocalCliDriver sysdb failure > > > Key: HIVE-19166 > URL: https://issues.apache.org/jira/browse/HIVE-19166 > Project: Hive > Issue Type: Sub-task > Components: Test >Reporter: Vineet Garg >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19166.04.patch, HIVE-19166.05.patch, > HIVE-19166.06.patch, HIVE-19166.09.patch, HIVE-19166.1.patch, > HIVE-19166.10.patch, HIVE-19166.11.patch, HIVE-19166.12.patch, > HIVE-19166.13.patch, HIVE-19166.14.patch, HIVE-19166.15.patch, > HIVE-19166.2.patch, HIVE-19166.3.patch > > > Broken by HIVE-18715 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20180) Backport HIVE-19759 to branch-3
[
https://issues.apache.org/jira/browse/HIVE-20180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546857#comment-16546857
]
Hive QA commented on HIVE-20180:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931908/HIVE-20180.2-branch-3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 96 failed/errored test(s), 14398 tests
executed
*Failed tests:*
{noformat}
TestAddPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestAddPartitionsFromPartSpec - did not produce a TEST-*.xml file (likely timed
out) (batchId=215)
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=221)
TestAggregateStatsCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=215)
TestAlterPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=215)
TestAppendPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=215)
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=271)
TestCachedStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=221)
TestCatalogCaching - did not produce a TEST-*.xml file (likely timed out)
(batchId=221)
TestCatalogNonDefaultClient - did not produce a TEST-*.xml file (likely timed
out) (batchId=213)
TestCatalogNonDefaultSvr - did not produce a TEST-*.xml file (likely timed out)
(batchId=221)
TestCatalogOldClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestCatalogs - did not produce a TEST-*.xml file (likely timed out)
(batchId=215)
TestCheckConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=223)
TestDatabases - did not produce a TEST-*.xml file (likely timed out)
(batchId=215)
TestDeadline - did not produce a TEST-*.xml file (likely timed out)
(batchId=221)
TestDefaultConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=215)
TestDropPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=271)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=216)
TestExchangePartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=215)
TestFMSketchSerialization - did not produce a TEST-*.xml file (likely timed
out) (batchId=223)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestForeignKey - did not produce a TEST-*.xml file (likely timed out)
(batchId=215)
TestFunctions - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestGetPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestGetTableMeta - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out)
(batchId=223)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out)
(batchId=223)
TestHdfsUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=221)
TestHiveAlterHandler - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestHiveMetaStoreGetMetaConf - did not produce a TEST-*.xml file (likely timed
out) (batchId=221)
TestHiveMetaStorePartitionSpecs - did not produce a TEST-*.xml file (likely
timed out) (batchId=215)
TestHiveMetaStoreSchemaMethods - did not produce a TEST-*.xml file (likely
timed out) (batchId=221)
TestHiveMetaStoreTimeout - did not produce a TEST-*.xml file (likely timed out)
(batchId=223)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=223)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=218)
TestHiveMetastoreCli - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestHyperLogLog - did not produce a TEST-*.xml file (likely timed out)
(batchId=223)
TestHyperLogLogDense - did not produce a TEST-*.xml file (likely timed out)
(batchId=223)
TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out)
(batchId=223)
TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out)
(batchId=223)
TestJSONMessageDeserializer - did not produce a TEST-*.xml file (likely timed
out) (batchId=221)
TestListPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestLockRequestBuilder - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestMarkPartition - did not produce a TEST-*.xml file (likely timed out)
(batchId=221)
TestMarkPartitionRemote - did not produce a TEST-*.xml file (likely timed out)
(batchId=223)
TestMetaStoreConnectionUrlHook - did not produce a TEST
[jira] [Commented] (HIVE-19166) TestMiniLlapLocalCliDriver sysdb failure
[
https://issues.apache.org/jira/browse/HIVE-19166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546868#comment-16546868
]
Zoltan Haindrich commented on HIVE-19166:
-
[~daijy]: Last time I was working on this patch I had a hard time because it
shows tables used by other tests - invoking to load all of them helps with
this.. (and it should probably work with cbo enabled)
{code}
--- ql/src/test/queries/clientpositive/sysdb.q
+++ ql/src/test/queries/clientpositive/sysdb.q
@@ -1,4 +1,5 @@
---! qt:dataset:alltypesorc
+--!
qt:dataset:alltypesorc,alltypesparquet,part,src,src1,srcbucket,srcbucket2,src_cbo,src_json,src_sequencefile,src_thrift,srcpart,cbo_t1,cbo_t2,cbo_t3,lineitem
+
set hive.strict.checks.cartesian.product=false;
set hive.compute.query.using.stats=false;
@@ -6,8 +7,6 @@ set hive.compute.query.using.stats=false;
set hive.support.concurrency=true;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
-set hive.cbo.enable=false;
{code}
> TestMiniLlapLocalCliDriver sysdb failure
>
>
> Key: HIVE-19166
> URL: https://issues.apache.org/jira/browse/HIVE-19166
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Vineet Garg
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19166.04.patch, HIVE-19166.05.patch,
> HIVE-19166.06.patch, HIVE-19166.09.patch, HIVE-19166.1.patch,
> HIVE-19166.10.patch, HIVE-19166.11.patch, HIVE-19166.12.patch,
> HIVE-19166.13.patch, HIVE-19166.14.patch, HIVE-19166.15.patch,
> HIVE-19166.2.patch, HIVE-19166.3.patch
>
>
> Broken by HIVE-18715
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15131) Change Parquet reader to read metadata on the task side
[
https://issues.apache.org/jira/browse/HIVE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546918#comment-16546918
]
Chao Sun commented on HIVE-15131:
-
Looks good [~adeshrao]. Could you also remove the "TODO" comment above the
{{readFooter}} call?
> Change Parquet reader to read metadata on the task side
> ---
>
> Key: HIVE-15131
> URL: https://issues.apache.org/jira/browse/HIVE-15131
> Project: Hive
> Issue Type: Bug
> Components: Reader
>Reporter: Chao Sun
>Assignee: Adesh Kumar Rao
>Priority: Major
> Attachments: HIVE-15131.1.patch
>
>
> Currently the {{ParquetRecordReaderWrapper}} still uses the {{readFooter}}
> API without filtering, which means it needs to read metadata about all row
> groups every time. This could some issues when input dataset is particularly
> big and has many columns.
> [Parquet-84|https://issues.apache.org/jira/browse/PARQUET-84] introduced
> another API which allows to do row group filtering on the task side. Hive
> should adopt this API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19809) Remove Deprecated Code From Utilities Class
[
https://issues.apache.org/jira/browse/HIVE-19809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546924#comment-16546924
]
BELUGA BEHR commented on HIVE-19809:
[~aihuaxu] Patch re-attached and passed all tests. Please consider for
inclusion into the project.
> Remove Deprecated Code From Utilities Class
> ---
>
> Key: HIVE-19809
> URL: https://issues.apache.org/jira/browse/HIVE-19809
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-19809.1.patch, HIVE-19809.2.patch
>
>
> {quote}
> This can go away once hive moves to support only JDK 7 and can use
> Files.createTempDirectory
> {quote}
> Remove the {{createTempDir}} method from the {{Utilities}} class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20184) Backport HIVE-20085 to branch-3
[
https://issues.apache.org/jira/browse/HIVE-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546934#comment-16546934
]
Hive QA commented on HIVE-20184:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931911/HIVE-20184.2-branch-3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12656/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12656/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12656/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-07-17 17:46:23.093
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12656/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z branch-3 ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-07-17 17:46:23.096
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 93e6e36 HIVE-19668 : Over 30% of the heap wasted by duplicate
org.antlr.runtime.CommonToken's and duplicate strings (Misha Dmitriev reviewed
by Aihua Xu and Vihang Karajgaonkar)
+ git clean -f -d
Removing standalone-metastore/metastore-common/
+ git checkout branch-3
Already on 'branch-3'
Your branch is up-to-date with 'origin/branch-3'.
+ git reset --hard origin/branch-3
HEAD is now at 93e6e36 HIVE-19668 : Over 30% of the heap wasted by duplicate
org.antlr.runtime.CommonToken's and duplicate strings (Misha Dmitriev reviewed
by Aihua Xu and Vihang Karajgaonkar)
+ git merge --ff-only origin/branch-3
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-07-17 17:46:24.473
+ rm -rf ../yetus_PreCommit-HIVE-Build-12656
+ mkdir ../yetus_PreCommit-HIVE-Build-12656
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12656
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12656/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc508326912297701017.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc508326912297701017.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
[ERROR] COMPILATION ERROR :
[ERROR]
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:[585,35]
cannot find symbol
symbol: variable EXTERNAL_TABLE_PURGE
location: class org.apache.hadoop.hive.metastore.utils.MetaStoreUtils
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile (default-compile)
on project hive-standalone-metastore: Compilation failure
[ERROR]
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/java/org/apache/hadoo
[jira] [Updated] (HIVE-20116) TezTask is using parent logger
[ https://issues.apache.org/jira/browse/HIVE-20116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20116: - Resolution: Fixed Fix Version/s: 3.2.0 4.0.0 Target Version/s: 4.0.0, 3.2.0 (was: 4.0.0) Status: Resolved (was: Patch Available) Committed to master and branch-3. > TezTask is using parent logger > -- > > Key: HIVE-20116 > URL: https://issues.apache.org/jira/browse/HIVE-20116 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20116.1.patch, HIVE-20116.2.patch, > HIVE-20116.3.patch > > > TezTask is using parent's logger (Task). It should instead use its own class > name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20165) Enable ZLIB for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20165: - Attachment: HIVE-20116.3.patch > Enable ZLIB for streaming ingest > > > Key: HIVE-20165 > URL: https://issues.apache.org/jira/browse/HIVE-20165 > Project: Hive > Issue Type: Bug > Components: Streaming, Transactions >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20165.1.patch, HIVE-20165.2.patch, > HIVE-20165.3.patch > > > Per [~gopalv]'s recommendation tried running streaming ingest with and > without zlib. Following are the numbers > > *Compression: NONE* > Total rows committed: 9380 > Throughput: *156* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *14.1 G* /apps/hive/warehouse/prasanth.db/culvert > > *Compression: ZLIB* > Total rows committed: 9210 > Throughput: *1535000* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *7.4 G* /apps/hive/warehouse/prasanth.db/culvert > > ZLIB is getting us 2x compression and only 2% lesser throughput. We should > enable ZLIB by default for streaming ingest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20165) Enable ZLIB for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20165: - Attachment: (was: HIVE-20116.3.patch) > Enable ZLIB for streaming ingest > > > Key: HIVE-20165 > URL: https://issues.apache.org/jira/browse/HIVE-20165 > Project: Hive > Issue Type: Bug > Components: Streaming, Transactions >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20165.1.patch, HIVE-20165.2.patch, > HIVE-20165.3.patch > > > Per [~gopalv]'s recommendation tried running streaming ingest with and > without zlib. Following are the numbers > > *Compression: NONE* > Total rows committed: 9380 > Throughput: *156* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *14.1 G* /apps/hive/warehouse/prasanth.db/culvert > > *Compression: ZLIB* > Total rows committed: 9210 > Throughput: *1535000* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *7.4 G* /apps/hive/warehouse/prasanth.db/culvert > > ZLIB is getting us 2x compression and only 2% lesser throughput. We should > enable ZLIB by default for streaming ingest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20165) Enable ZLIB for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20165: - Attachment: HIVE-20165.3.patch > Enable ZLIB for streaming ingest > > > Key: HIVE-20165 > URL: https://issues.apache.org/jira/browse/HIVE-20165 > Project: Hive > Issue Type: Bug > Components: Streaming, Transactions >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20165.1.patch, HIVE-20165.2.patch, > HIVE-20165.3.patch > > > Per [~gopalv]'s recommendation tried running streaming ingest with and > without zlib. Following are the numbers > > *Compression: NONE* > Total rows committed: 9380 > Throughput: *156* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *14.1 G* /apps/hive/warehouse/prasanth.db/culvert > > *Compression: ZLIB* > Total rows committed: 9210 > Throughput: *1535000* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *7.4 G* /apps/hive/warehouse/prasanth.db/culvert > > ZLIB is getting us 2x compression and only 2% lesser throughput. We should > enable ZLIB by default for streaming ingest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20181) Backport HIVE-20045 to branch-3
[
https://issues.apache.org/jira/browse/HIVE-20181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546968#comment-16546968
]
Hive QA commented on HIVE-20181:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-12657/patches/PreCommit-HIVE-Build-12657.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12657/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Backport HIVE-20045 to branch-3
> ---
>
> Key: HIVE-20181
> URL: https://issues.apache.org/jira/browse/HIVE-20181
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 3.0.0
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-20181.1.branch-3.patch, HIVE-20181.2-branch-3.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20194) HiveMetastoreClient should use reflection to instantiate embedded HMS instance
[ https://issues.apache.org/jira/browse/HIVE-20194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov reassigned HIVE-20194: - > HiveMetastoreClient should use reflection to instantiate embedded HMS instance > -- > > Key: HIVE-20194 > URL: https://issues.apache.org/jira/browse/HIVE-20194 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > > When HiveMetastoreClient is used in embedded mode, it instantiates metastore > server. Since we want to separate client and server code we can no longer > instantiate the class directly but need to use reflection for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20196) Separate MetastoreConf into common and server parts
[ https://issues.apache.org/jira/browse/HIVE-20196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov reassigned HIVE-20196: - > Separate MetastoreConf into common and server parts > --- > > Key: HIVE-20196 > URL: https://issues.apache.org/jira/browse/HIVE-20196 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > > MetastoreConf has knowledge about some server-specific classes. We need to > separate these into a separate server-specific class. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20181) Backport HIVE-20045 to branch-3
[
https://issues.apache.org/jira/browse/HIVE-20181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547021#comment-16547021
]
Hive QA commented on HIVE-20181:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931909/HIVE-20181.2-branch-3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 14378 tests
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=258)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=258)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=258)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=258)
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=186)
[infer_bucket_sort_num_buckets.q,gen_udf_example_add10.q,spark_explainuser_1.q,spark_use_ts_stats_for_mapjoin.q,orc_merge6.q,orc_merge5.q,bucketmapjoin6.q,import_exported_table.q,spark_opt_shuffle_serde.q,temp_table_external.q,spark_dynamic_partition_pruning_6.q,root_dir_external_table.q,dynamic_rdd_cache.q,auto_sortmerge_join_16.q,vector_outer_join3.q,schemeAuthority.q,parallel_orderby.q,vector_outer_join1.q,spark_dynamic_partition_pruning_recursive_mapjoin.q,spark_dynamic_partition_pruning_mapjoin_only.q]
TestTezPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=258)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_with_masking]
(batchId=174)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testManagedPaths
(batchId=235)
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
(batchId=243)
org.apache.hive.spark.client.rpc.TestRpc.testServerPort (batchId=310)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12657/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12657/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12657/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931909 - PreCommit-HIVE-Build
> Backport HIVE-20045 to branch-3
> ---
>
> Key: HIVE-20181
> URL: https://issues.apache.org/jira/browse/HIVE-20181
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 3.0.0
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-20181.1.branch-3.patch, HIVE-20181.2-branch-3.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20174) Vectorization: Fix NULL / Wrong Results issues in GROUP BY Aggregation Functions
[ https://issues.apache.org/jira/browse/HIVE-20174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20174: Fix Version/s: 4.0.0 > Vectorization: Fix NULL / Wrong Results issues in GROUP BY Aggregation > Functions > > > Key: HIVE-20174 > URL: https://issues.apache.org/jira/browse/HIVE-20174 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 4.0.0 > > Attachments: HIVE-20174.01.patch, HIVE-20174.02.patch, > HIVE-20174.03.patch, HIVE-20174.04.patch, HIVE-20174.05.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized aggregation functions. > > BUGs found: > 1) AVG/VARIANCE (family) in PARTIAL1 mode was returning NULL instead of count > = 0, sum = 0 (All data types). For AVG DECIMAL, only return NULL if there > was an overflow. > 2) AVG/MIN/MAX was not detecting repeated NULL correctly for the TIMESTAMP, > INTERVAL_DAY_TIME, and String Family. Eliminated redundant code. > 3) Fix incorrect calculation for VARIANCE (family) in PARTIAL2 and FINAL > modes (HIVE-18758). > 4) Fix row-mode AVG DECIMAL to enforce output type precision and scale in > COMPLETE and FINAL modes. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20197) Vectorization: Add DECIMAL_64 testing, add Date/Interval/Timestamp arithmetic, and fix more NULL / Wrong Results issues in GROUP BY Aggregation Functions
[ https://issues.apache.org/jira/browse/HIVE-20197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-20197: --- > Vectorization: Add DECIMAL_64 testing, add Date/Interval/Timestamp > arithmetic, and fix more NULL / Wrong Results issues in GROUP BY Aggregation > Functions > - > > Key: HIVE-20197 > URL: https://issues.apache.org/jira/browse/HIVE-20197 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > > Add DECIMAL_64 testing to TestVectorArithmetic and TestVectorAggregation. > And, add a few more aggregation tests to TestVectorAggregation. > Add + and - Date/Interval/Timestamp arithmetic tests to TestVectorArithmetic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20197) Vectorization: Add DECIMAL_64 testing, add Date/Interval/Timestamp arithmetic, and add more GROUP BY Aggregation tests
[ https://issues.apache.org/jira/browse/HIVE-20197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20197: Summary: Vectorization: Add DECIMAL_64 testing, add Date/Interval/Timestamp arithmetic, and add more GROUP BY Aggregation tests (was: Vectorization: Add DECIMAL_64 testing, add Date/Interval/Timestamp arithmetic, and fix more NULL / Wrong Results issues in GROUP BY Aggregation Functions) > Vectorization: Add DECIMAL_64 testing, add Date/Interval/Timestamp > arithmetic, and add more GROUP BY Aggregation tests > -- > > Key: HIVE-20197 > URL: https://issues.apache.org/jira/browse/HIVE-20197 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > > Add DECIMAL_64 testing to TestVectorArithmetic and TestVectorAggregation. > And, add a few more aggregation tests to TestVectorAggregation. > Add + and - Date/Interval/Timestamp arithmetic tests to TestVectorArithmetic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20185) Backport HIVE-20111 to branch-3
[
https://issues.apache.org/jira/browse/HIVE-20185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547043#comment-16547043
]
Hive QA commented on HIVE-20185:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931912/HIVE-20185.2-branch-3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12658/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12658/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12658/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-07-17 19:54:09.655
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12658/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z branch-3 ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-07-17 19:54:09.659
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 530a68a HIVE-20116: TezTask is using parent logger (Prasanth
Jayachandran reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout branch-3
Already on 'branch-3'
Your branch is up-to-date with 'origin/branch-3'.
+ git reset --hard origin/branch-3
HEAD is now at 530a68a HIVE-20116: TezTask is using parent logger (Prasanth
Jayachandran reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/branch-3
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-07-17 19:54:10.952
+ rm -rf ../yetus_PreCommit-HIVE-Build-12658
+ mkdir ../yetus_PreCommit-HIVE-Build-12658
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12658
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12658/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:507: trailing whitespace.
CREATE EXTERNAL TABLE users(key int, userid int, username string, created int)
/data/hiveptest/working/scratch/build.patch:550: trailing whitespace.
CREATE EXTERNAL TABLE hive1_tbl_data_hbase1 (COLUMID string,COLUMN_FN
string,COLUMN_LN string,EMAIL string,COL_UPDATED_DATE timestamp, PK_COLUM
string)
/data/hiveptest/working/scratch/build.patch:564: trailing whitespace.
CREATE EXTERNAL TABLE hive1_tbl_data_hbase2 (COLUMID string,COLUMN_FN
string,COLUMN_LN string,EMAIL string,COL_UPDATED_DATE timestamp, PK_COLUM
string)
/data/hiveptest/working/scratch/build.patch:579: trailing whitespace.
CREATE EXTERNAL TABLE hbase_pushdown(key string, value string)
/data/hiveptest/working/scratch/build.patch:593: trailing whitespace.
CREATE EXTERNAL TABLE hbase_pushdown(key int, value string)
warning: squelched 58 whitespace errors
warning: 63 lines add whitespace errors.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc1863207643358360550.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc1863207643358360550.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source
[jira] [Updated] (HIVE-20197) Vectorization: Add DECIMAL_64 testing, add Date/Interval/Timestamp arithmetic, and add more GROUP BY Aggregation tests
[ https://issues.apache.org/jira/browse/HIVE-20197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20197: Attachment: HIVE-20197.01.patch > Vectorization: Add DECIMAL_64 testing, add Date/Interval/Timestamp > arithmetic, and add more GROUP BY Aggregation tests > -- > > Key: HIVE-20197 > URL: https://issues.apache.org/jira/browse/HIVE-20197 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20197.01.patch > > > Add DECIMAL_64 testing to TestVectorArithmetic and TestVectorAggregation. > And, add a few more aggregation tests to TestVectorAggregation. > Add + and - Date/Interval/Timestamp arithmetic tests to TestVectorArithmetic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20197) Vectorization: Add DECIMAL_64 testing, add Date/Interval/Timestamp arithmetic, and add more GROUP BY Aggregation tests
[ https://issues.apache.org/jira/browse/HIVE-20197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20197: Status: Patch Available (was: Open) > Vectorization: Add DECIMAL_64 testing, add Date/Interval/Timestamp > arithmetic, and add more GROUP BY Aggregation tests > -- > > Key: HIVE-20197 > URL: https://issues.apache.org/jira/browse/HIVE-20197 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20197.01.patch > > > Add DECIMAL_64 testing to TestVectorArithmetic and TestVectorAggregation. > And, add a few more aggregation tests to TestVectorAggregation. > Add + and - Date/Interval/Timestamp arithmetic tests to TestVectorArithmetic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20188) Split server-specific code outside of standalone metastore-common
[ https://issues.apache.org/jira/browse/HIVE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-20188: -- Attachment: HIVE-20188.01.patch > Split server-specific code outside of standalone metastore-common > - > > Key: HIVE-20188 > URL: https://issues.apache.org/jira/browse/HIVE-20188 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20188.01.patch > > > The goal of this JIRA is to split metastore-common and separate the server > code into a separate module. This is still a pom-only change so all consumers > will have access to both. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-20188) Split server-specific code outside of standalone metastore-common
[ https://issues.apache.org/jira/browse/HIVE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-20188 started by Alexander Kolbasov. - > Split server-specific code outside of standalone metastore-common > - > > Key: HIVE-20188 > URL: https://issues.apache.org/jira/browse/HIVE-20188 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20188.01.patch > > > The goal of this JIRA is to split metastore-common and separate the server > code into a separate module. This is still a pom-only change so all consumers > will have access to both. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20188) Split server-specific code outside of standalone metastore-common
[
https://issues.apache.org/jira/browse/HIVE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547051#comment-16547051
]
Alexander Kolbasov commented on HIVE-20188:
---
Patch 01 is merged with
{code}
* commit 4fcf3d720855bef3939ac537ff4a4e5616b493fb (origin/master, origin/HEAD)
| Author: Prasanth Jayachandran
| Date: Tue Jul 17 11:11:56 2018 -0700
|
| HIVE-20116: TezTask is using parent logger (Prasanth Jayachandran
reviewed by Sergey Shelukhin)
{code}
> Split server-specific code outside of standalone metastore-common
> -
>
> Key: HIVE-20188
> URL: https://issues.apache.org/jira/browse/HIVE-20188
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20188.01.patch
>
>
> The goal of this JIRA is to split metastore-common and separate the server
> code into a separate module. This is still a pom-only change so all consumers
> will have access to both.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20188) Split server-specific code outside of standalone metastore-common
[ https://issues.apache.org/jira/browse/HIVE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-20188: -- Status: Patch Available (was: In Progress) > Split server-specific code outside of standalone metastore-common > - > > Key: HIVE-20188 > URL: https://issues.apache.org/jira/browse/HIVE-20188 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20188.01.patch > > > The goal of this JIRA is to split metastore-common and separate the server > code into a separate module. This is still a pom-only change so all consumers > will have access to both. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20182) Backport HIVE-20067 to branch-3
[
https://issues.apache.org/jira/browse/HIVE-20182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547056#comment-16547056
]
Hive QA commented on HIVE-20182:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 23s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-12659/patches/PreCommit-HIVE-Build-12659.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12659/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Backport HIVE-20067 to branch-3
> ---
>
> Key: HIVE-20182
> URL: https://issues.apache.org/jira/browse/HIVE-20182
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-20182.1.branch-3.patch, HIVE-20182.2-branch-3.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17852) remove support for list bucketing "stored as directories" in 3.0
[ https://issues.apache.org/jira/browse/HIVE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547110#comment-16547110 ] Laszlo Bodor commented on HIVE-17852: - orc_llap_counters failure is not related, fails on master too > remove support for list bucketing "stored as directories" in 3.0 > > > Key: HIVE-17852 > URL: https://issues.apache.org/jira/browse/HIVE-17852 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-17852.01.patch, HIVE-17852.02.patch, > HIVE-17852.03.patch, HIVE-17852.04.patch, HIVE-17852.05.patch, > HIVE-17852.06.patch, HIVE-17852.07.patch, HIVE-17852.08.patch, > HIVE-17852.09.patch, HIVE-17852.10.patch, HIVE-17852.11.patch, > HIVE-17852.12.patch, HIVE-17852.13.patch, HIVE-17852.14.patch, > HIVE-17852.15.patch, HIVE-17852.16.patch, HIVE-17852.17.patch, > HIVE-17852.18.patch, HIVE-17852.19.patch, HIVE-17852.20.patch > > > From the email thread: > 1) LB, when stored as directories, adds a lot of low-level complexity to Hive > tables that has to be accounted for in many places in the code where the > files are written or modified - from FSOP to ACID/replication/export. > 2) While working on some FSOP code I noticed that some of that logic is > broken - e.g. the duplicate file removal from tasks, a pretty fundamental > correctness feature in Hive, may be broken. LB also doesn’t appear to be > compatible with e.g. regular bucketing. > 3) The feature hasn’t seen development activity in a while; it also doesn’t > appear to be used a lot. > Keeping with the theme of cleaning up “legacy” code for 3.0, I was proposing > we remove it. > (2) also suggested that, if needed, it might be easier to implement similar > functionality by adding some flexibility to partitions (which LB directories > look like anyway); that would also keep the logic on a higher level of > abstraction (split generation, partition pruning) as opposed to many > low-level places like FSOP, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20182) Backport HIVE-20067 to branch-3
[
https://issues.apache.org/jira/browse/HIVE-20182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547126#comment-16547126
]
Hive QA commented on HIVE-20182:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931910/HIVE-20182.2-branch-3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 14398 tests
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=258)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=258)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=258)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=258)
TestTezPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=258)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_with_masking]
(batchId=174)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testManagedPaths
(batchId=235)
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
(batchId=243)
org.apache.hive.spark.client.rpc.TestRpc.testServerPort (batchId=310)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12659/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12659/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12659/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931910 - PreCommit-HIVE-Build
> Backport HIVE-20067 to branch-3
> ---
>
> Key: HIVE-20182
> URL: https://issues.apache.org/jira/browse/HIVE-20182
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-20182.1.branch-3.patch, HIVE-20182.2-branch-3.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20183) Inserting from bucketed table can cause data loss, if the source table contains empty buckets
[
https://issues.apache.org/jira/browse/HIVE-20183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547188#comment-16547188
]
Hive QA commented on HIVE-20183:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 5m
12s{color} | {color:blue} ql in master has 2274 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12661/dev-support/hive-personality.sh
|
| git revision | master / 4fcf3d7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12661/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Inserting from bucketed table can cause data loss, if the source table
> contains empty buckets
> -
>
> Key: HIVE-20183
> URL: https://issues.apache.org/jira/browse/HIVE-20183
> Project: Hive
> Issue Type: Bug
> Components: Operators
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20183.2.patch, HIVE-20183.patch
>
>
> Could be reproduced by the following:
> {code}
> set hive.enforce.bucketing=true;
> set hive.enforce.sorting=true;
> set hive.optimize.bucketingsorting=true;
> create table bucket1 (id int, val string) clustered by (id) sorted by (id
> ASC) INTO 4 BUCKETS;
> insert into bucket1 values (1, 'abc'), (3, 'abc');
> select * from bucket1;
> +-+--+
> | bucket1.id | bucket1.val |
> +-+--+
> | 3 | abc |
> | 1 | abc |
> +-+--+
> create table bucket2 like bucket1;
> insert overwrite table bucket2 select * from bucket1;
> select * from bucket2;
> +-+--+
> | bucket2.id | bucket2.val |
> +-+--+
> | 1 | abc |
> +-+--+
> {code}
--
This message was sent by
[jira] [Commented] (HIVE-20183) Inserting from bucketed table can cause data loss, if the source table contains empty buckets
[
https://issues.apache.org/jira/browse/HIVE-20183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547223#comment-16547223
]
Hive QA commented on HIVE-20183:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931930/HIVE-20183.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14664 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12661/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12661/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12661/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931930 - PreCommit-HIVE-Build
> Inserting from bucketed table can cause data loss, if the source table
> contains empty buckets
> -
>
> Key: HIVE-20183
> URL: https://issues.apache.org/jira/browse/HIVE-20183
> Project: Hive
> Issue Type: Bug
> Components: Operators
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20183.2.patch, HIVE-20183.patch
>
>
> Could be reproduced by the following:
> {code}
> set hive.enforce.bucketing=true;
> set hive.enforce.sorting=true;
> set hive.optimize.bucketingsorting=true;
> create table bucket1 (id int, val string) clustered by (id) sorted by (id
> ASC) INTO 4 BUCKETS;
> insert into bucket1 values (1, 'abc'), (3, 'abc');
> select * from bucket1;
> +-+--+
> | bucket1.id | bucket1.val |
> +-+--+
> | 3 | abc |
> | 1 | abc |
> +-+--+
> create table bucket2 like bucket1;
> insert overwrite table bucket2 select * from bucket1;
> select * from bucket2;
> +-+--+
> | bucket2.id | bucket2.val |
> +-+--+
> | 1 | abc |
> +-+--+
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20188) Split server-specific code outside of standalone metastore-common
[ https://issues.apache.org/jira/browse/HIVE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547244#comment-16547244 ] Alexander Kolbasov commented on HIVE-20188: --- This change just moves a lot of files from metastore-common to a new metastore-server module. This is not a final state of affairs because there are a lot of other things that can and should be kept in common but it requires some code changes. The goal here is to just move things around without code changes, so some files will later move back to common. > Split server-specific code outside of standalone metastore-common > - > > Key: HIVE-20188 > URL: https://issues.apache.org/jira/browse/HIVE-20188 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20188.01.patch > > > The goal of this JIRA is to split metastore-common and separate the server > code into a separate module. This is still a pom-only change so all consumers > will have access to both. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20194) HiveMetastoreClient should use reflection to instantiate embedded HMS instance
[ https://issues.apache.org/jira/browse/HIVE-20194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-20194: -- Attachment: HIVE-20194.01.patch > HiveMetastoreClient should use reflection to instantiate embedded HMS instance > -- > > Key: HIVE-20194 > URL: https://issues.apache.org/jira/browse/HIVE-20194 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20194.01.patch > > > When HiveMetastoreClient is used in embedded mode, it instantiates metastore > server. Since we want to separate client and server code we can no longer > instantiate the class directly but need to use reflection for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20194) HiveMetastoreClient should use reflection to instantiate embedded HMS instance
[
https://issues.apache.org/jira/browse/HIVE-20194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-20194:
--
Status: Patch Available (was: Open)
Patch 1 is merged with
{code}
* commit 4fcf3d720855bef3939ac537ff4a4e5616b493fb (origin/master, origin/HEAD)
| Author: Prasanth Jayachandran
| Date: Tue Jul 17 11:11:56 2018 -0700
|
| HIVE-20116: TezTask is using parent logger (Prasanth Jayachandran
reviewed by Sergey Shelukhin)
{code}
> HiveMetastoreClient should use reflection to instantiate embedded HMS instance
> --
>
> Key: HIVE-20194
> URL: https://issues.apache.org/jira/browse/HIVE-20194
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20194.01.patch
>
>
> When HiveMetastoreClient is used in embedded mode, it instantiates metastore
> server. Since we want to separate client and server code we can no longer
> instantiate the class directly but need to use reflection for that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20165) Enable ZLIB for streaming ingest
[
https://issues.apache.org/jira/browse/HIVE-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547261#comment-16547261
]
Hive QA commented on HIVE-20165:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
53s{color} | {color:blue} ql in master has 2274 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12662/dev-support/hive-personality.sh
|
| git revision | master / 4fcf3d7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql streaming U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12662/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enable ZLIB for streaming ingest
>
>
> Key: HIVE-20165
> URL: https://issues.apache.org/jira/browse/HIVE-20165
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20165.1.patch, HIVE-20165.2.patch,
> HIVE-20165.3.patch
>
>
> Per [~gopalv]'s recommendation tried running streaming ingest with and
> without zlib. Following are the numbers
>
> *Compression: NONE*
> Total rows committed: 9380
> Throughput: *156* rows/second
> $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert
> *14.1 G* /apps/hive/warehouse/prasanth.db/culvert
>
> *Compression: ZLIB*
> Total rows committed: 9210
> Throughput: *1535000* rows/second
> $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert
> *7.4 G* /apps/hive/warehouse/prasanth.db/culvert
>
> ZLIB is getting us 2x compression and only 2% lesser throughput. We should
> enable ZLIB by default for streaming ingest.
--
This message was sent by Atlassian
[jira] [Updated] (HIVE-20196) Separate MetastoreConf into common and server parts
[ https://issues.apache.org/jira/browse/HIVE-20196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-20196: -- Attachment: HIVE-20196.01.patch > Separate MetastoreConf into common and server parts > --- > > Key: HIVE-20196 > URL: https://issues.apache.org/jira/browse/HIVE-20196 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20196.01.patch > > > MetastoreConf has knowledge about some server-specific classes. We need to > separate these into a separate server-specific class. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20196) Separate MetastoreConf into common and server parts
[
https://issues.apache.org/jira/browse/HIVE-20196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-20196:
--
Status: Patch Available (was: Open)
Patch 1 is on top of
{code}
* commit 4fcf3d720855bef3939ac537ff4a4e5616b493fb (origin/master, origin/HEAD)
| Author: Prasanth Jayachandran
| Date: Tue Jul 17 11:11:56 2018 -0700
|
| HIVE-20116: TezTask is using parent logger (Prasanth Jayachandran
reviewed by Sergey Shelukhin)
{code}
> Separate MetastoreConf into common and server parts
> ---
>
> Key: HIVE-20196
> URL: https://issues.apache.org/jira/browse/HIVE-20196
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20196.01.patch
>
>
> MetastoreConf has knowledge about some server-specific classes. We need to
> separate these into a separate server-specific class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20165) Enable ZLIB for streaming ingest
[
https://issues.apache.org/jira/browse/HIVE-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547276#comment-16547276
]
Hive QA commented on HIVE-20165:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931984/HIVE-20165.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14662 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12662/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12662/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12662/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931984 - PreCommit-HIVE-Build
> Enable ZLIB for streaming ingest
>
>
> Key: HIVE-20165
> URL: https://issues.apache.org/jira/browse/HIVE-20165
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20165.1.patch, HIVE-20165.2.patch,
> HIVE-20165.3.patch
>
>
> Per [~gopalv]'s recommendation tried running streaming ingest with and
> without zlib. Following are the numbers
>
> *Compression: NONE*
> Total rows committed: 9380
> Throughput: *156* rows/second
> $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert
> *14.1 G* /apps/hive/warehouse/prasanth.db/culvert
>
> *Compression: ZLIB*
> Total rows committed: 9210
> Throughput: *1535000* rows/second
> $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert
> *7.4 G* /apps/hive/warehouse/prasanth.db/culvert
>
> ZLIB is getting us 2x compression and only 2% lesser throughput. We should
> enable ZLIB by default for streaming ingest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-14162) Allow disabling of long running job on Hive On Spark On YARN
[ https://issues.apache.org/jira/browse/HIVE-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-14162: Attachment: HIVE-14162.3.patch > Allow disabling of long running job on Hive On Spark On YARN > > > Key: HIVE-14162 > URL: https://issues.apache.org/jira/browse/HIVE-14162 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Thomas Scott >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-14162.1.patch, HIVE-14162.2.patch, > HIVE-14162.3.patch > > > Hive On Spark launches a long running process on the first query to handle > all queries for that user session. In some use cases this is not desired, for > instance when using Hue with large intervals between query executions. > Could we have a property that would cause long running spark jobs to be > terminated after each query execution and started again for the next one? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19166) TestMiniLlapLocalCliDriver sysdb failure
[
https://issues.apache.org/jira/browse/HIVE-19166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547323#comment-16547323
]
Hive QA commented on HIVE-19166:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2274 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
48s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12663/dev-support/hive-personality.sh
|
| git revision | master / 4fcf3d7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: metastore ql itests/hive-unit itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12663/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestMiniLlapLocalCliDriver sysdb failure
>
>
> Key: HIVE-19166
> URL: https://issues.apache.org/jira/browse/HIVE-19166
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Vineet Garg
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19166.04.patch, HIVE-19166.05.patch,
> HIVE-19166.06.patch, HIVE-19166.09.patch, HIVE-19166.1.patch,
> HIVE-19166.10.patch, HIVE-19166.11.patch, HIVE-19166.12.patch,
> HIVE-19166.13.patch, HIVE-19166.14.patch, HIVE-19166.15.patch,
> HIVE-19166.2.patch, HIVE-19166.3.patch
>
>
> Broken by HIVE-18715
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20177) Vectorization: Reduce KeyWrapper allocation in GroupBy Streaming mode
[
https://issues.apache.org/jira/browse/HIVE-20177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-20177:
---
Labels: performance (was: )
> Vectorization: Reduce KeyWrapper allocation in GroupBy Streaming mode
> -
>
> Key: HIVE-20177
> URL: https://issues.apache.org/jira/browse/HIVE-20177
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Labels: performance
> Attachments: HIVE-20177.01.patch, HIVE-20177.WIP.patch
>
>
> The streaming mode for VectorGroupBy allocates a large number of arrays due
> to VectorKeyHashWrapper::duplicateTo()
> Since the vectors can't be mutated in-place while a single batch is being
> processed, this operation can be cut by 1000x by allocating a streaming key
> at the end of the loop, instead of reallocating within the loop.
> {code}
> for(int i = 0; i < batch.size; ++i) {
> if (!batchKeys[i].equals(streamingKey)) {
> // We've encountered a new key, must save current one
> // We can't forward yet, the aggregators have not been evaluated
> rowsToFlush[flushMark] = currentStreamingAggregators;
> if (keysToFlush[flushMark] == null) {
> keysToFlush[flushMark] = (VectorHashKeyWrapper)
> streamingKey.copyKey();
> } else {
> streamingKey.duplicateTo(keysToFlush[flushMark]);
> }
> currentStreamingAggregators =
> streamAggregationBufferRowPool.getFromPool();
> batchKeys[i].duplicateTo(streamingKey);
> ++flushMark;
> }
> {code}
> The duplicateTo can be pushed out of the loop since there only one to truly
> keep a copy of is the last unique key in the VRB.
> The actual byte[] values within the keys are safely copied out by -
> VectorHashKeyWrapperBatch.assignRowColumn() which calls setVal() and not
> setRef().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20165) Enable ZLIB for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-20165: --- Attachment: HIVE-20165.1.branch-3.patch > Enable ZLIB for streaming ingest > > > Key: HIVE-20165 > URL: https://issues.apache.org/jira/browse/HIVE-20165 > Project: Hive > Issue Type: Bug > Components: Streaming, Transactions >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Gopal V >Priority: Major > Attachments: HIVE-20165.1.branch-3.patch, HIVE-20165.1.patch, > HIVE-20165.2.patch, HIVE-20165.3.patch > > > Per [~gopalv]'s recommendation tried running streaming ingest with and > without zlib. Following are the numbers > > *Compression: NONE* > Total rows committed: 9380 > Throughput: *156* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *14.1 G* /apps/hive/warehouse/prasanth.db/culvert > > *Compression: ZLIB* > Total rows committed: 9210 > Throughput: *1535000* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *7.4 G* /apps/hive/warehouse/prasanth.db/culvert > > ZLIB is getting us 2x compression and only 2% lesser throughput. We should > enable ZLIB by default for streaming ingest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20165) Enable ZLIB for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-20165: -- Assignee: Gopal V (was: Prasanth Jayachandran) > Enable ZLIB for streaming ingest > > > Key: HIVE-20165 > URL: https://issues.apache.org/jira/browse/HIVE-20165 > Project: Hive > Issue Type: Bug > Components: Streaming, Transactions >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Gopal V >Priority: Major > Attachments: HIVE-20165.1.branch-3.patch, HIVE-20165.1.patch, > HIVE-20165.2.patch, HIVE-20165.3.patch > > > Per [~gopalv]'s recommendation tried running streaming ingest with and > without zlib. Following are the numbers > > *Compression: NONE* > Total rows committed: 9380 > Throughput: *156* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *14.1 G* /apps/hive/warehouse/prasanth.db/culvert > > *Compression: ZLIB* > Total rows committed: 9210 > Throughput: *1535000* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *7.4 G* /apps/hive/warehouse/prasanth.db/culvert > > ZLIB is getting us 2x compression and only 2% lesser throughput. We should > enable ZLIB by default for streaming ingest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20165) Enable ZLIB for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547328#comment-16547328 ] Gopal V commented on HIVE-20165: Pushed to master, rebased to branch-3 > Enable ZLIB for streaming ingest > > > Key: HIVE-20165 > URL: https://issues.apache.org/jira/browse/HIVE-20165 > Project: Hive > Issue Type: Bug > Components: Streaming, Transactions >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20165.1.branch-3.patch, HIVE-20165.1.patch, > HIVE-20165.2.patch, HIVE-20165.3.patch > > > Per [~gopalv]'s recommendation tried running streaming ingest with and > without zlib. Following are the numbers > > *Compression: NONE* > Total rows committed: 9380 > Throughput: *156* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *14.1 G* /apps/hive/warehouse/prasanth.db/culvert > > *Compression: ZLIB* > Total rows committed: 9210 > Throughput: *1535000* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *7.4 G* /apps/hive/warehouse/prasanth.db/culvert > > ZLIB is getting us 2x compression and only 2% lesser throughput. We should > enable ZLIB by default for streaming ingest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20165) Enable ZLIB for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-20165: -- Assignee: Prasanth Jayachandran (was: Gopal V) > Enable ZLIB for streaming ingest > > > Key: HIVE-20165 > URL: https://issues.apache.org/jira/browse/HIVE-20165 > Project: Hive > Issue Type: Bug > Components: Streaming, Transactions >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20165.1.branch-3.patch, HIVE-20165.1.patch, > HIVE-20165.2.patch, HIVE-20165.3.patch > > > Per [~gopalv]'s recommendation tried running streaming ingest with and > without zlib. Following are the numbers > > *Compression: NONE* > Total rows committed: 9380 > Throughput: *156* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *14.1 G* /apps/hive/warehouse/prasanth.db/culvert > > *Compression: ZLIB* > Total rows committed: 9210 > Throughput: *1535000* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *7.4 G* /apps/hive/warehouse/prasanth.db/culvert > > ZLIB is getting us 2x compression and only 2% lesser throughput. We should > enable ZLIB by default for streaming ingest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19829) Incremental replication load should create tasks in execution phase rather than semantic phase
[ https://issues.apache.org/jira/browse/HIVE-19829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19829: --- Attachment: HIVE-19829.11-branch-3.patch > Incremental replication load should create tasks in execution phase rather > than semantic phase > -- > > Key: HIVE-19829 > URL: https://issues.apache.org/jira/browse/HIVE-19829 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19829.01.patch, HIVE-19829.02.patch, > HIVE-19829.03.patch, HIVE-19829.04.patch, HIVE-19829.06.patch, > HIVE-19829.07.patch, HIVE-19829.07.patch, HIVE-19829.08-branch-3.patch, > HIVE-19829.08.patch, HIVE-19829.09.patch, HIVE-19829.10-branch-3.patch, > HIVE-19829.10.patch, HIVE-19829.11-branch-3.patch > > > Split the incremental load into multiple iterations. In each iteration create > number of tasks equal to the configured value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19829) Incremental replication load should create tasks in execution phase rather than semantic phase
[ https://issues.apache.org/jira/browse/HIVE-19829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19829: --- Attachment: (was: HIVE-19829.11-branch-3.patch) > Incremental replication load should create tasks in execution phase rather > than semantic phase > -- > > Key: HIVE-19829 > URL: https://issues.apache.org/jira/browse/HIVE-19829 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19829.01.patch, HIVE-19829.02.patch, > HIVE-19829.03.patch, HIVE-19829.04.patch, HIVE-19829.06.patch, > HIVE-19829.07.patch, HIVE-19829.07.patch, HIVE-19829.08-branch-3.patch, > HIVE-19829.08.patch, HIVE-19829.09.patch, HIVE-19829.10-branch-3.patch, > HIVE-19829.10.patch, HIVE-19829.11-branch-3.patch > > > Split the incremental load into multiple iterations. In each iteration create > number of tasks equal to the configured value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19166) TestMiniLlapLocalCliDriver sysdb failure
[
https://issues.apache.org/jira/browse/HIVE-19166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547332#comment-16547332
]
Hive QA commented on HIVE-19166:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931961/HIVE-19166.15.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14663 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=165)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12663/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12663/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12663/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931961 - PreCommit-HIVE-Build
> TestMiniLlapLocalCliDriver sysdb failure
>
>
> Key: HIVE-19166
> URL: https://issues.apache.org/jira/browse/HIVE-19166
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Vineet Garg
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19166.04.patch, HIVE-19166.05.patch,
> HIVE-19166.06.patch, HIVE-19166.09.patch, HIVE-19166.1.patch,
> HIVE-19166.10.patch, HIVE-19166.11.patch, HIVE-19166.12.patch,
> HIVE-19166.13.patch, HIVE-19166.14.patch, HIVE-19166.15.patch,
> HIVE-19166.2.patch, HIVE-19166.3.patch
>
>
> Broken by HIVE-18715
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20131) SQL Script changes for creating txn write notification in 3.2.0 files
[ https://issues.apache.org/jira/browse/HIVE-20131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547350#comment-16547350 ] mahesh kumar behera commented on HIVE-20131: [~vgarg] So we dont need sql scripts related to 3.2.0 in apache master ? The changes in 3.0-3.1 upgrade script and 3.1 schema file is to revert the changes done in the previous patch > SQL Script changes for creating txn write notification in 3.2.0 files > --- > > Key: HIVE-20131 > URL: https://issues.apache.org/jira/browse/HIVE-20131 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 4.0.0 > > Attachments: HIVE-20131.01.patch > > > 1. Change partition name size from 1024 to 767 . (mySQL 5.6 and before that > supports max 767 length keys) > 2. Remove the create txn_write_notification_log table creation from 3.1.0 > scripts and add a new scripts for 3.2.0 > 3. Remove the file 3.1.0-to-4.0.0 and instead add file for 3.2.0-to-4.0.0 and > 3.1.0-to-3.2.0 > 4. Change in metastore init schema xml file to take 4.0.0 instead of 3.1.0 > as current version. > h1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19924) Tag distcp jobs run by Repl Load
[ https://issues.apache.org/jira/browse/HIVE-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19924: --- Attachment: HIVE-19924.03.patch > Tag distcp jobs run by Repl Load > > > Key: HIVE-19924 > URL: https://issues.apache.org/jira/browse/HIVE-19924 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19924.01.patch, HIVE-19924.02.patch, > HIVE-19924.03.patch > > > Add tags in jobconf for distcp related jobs started by replication. This will > allow hive to kill these jobs in case beacon retries, or hs2 dies and beacon > issues a kill command. > * one of the tags should definitely be the query_id that starts the job : > With this flow beacon before retrying the bootstrap load, will issue a kill > command to hs2 with the query id of the previous issued command. hs2 will > then kill an running jobs on yarn tagged with the Query_id. > * To get around the additional failure point as mentioned above. The jobs > can be tagged with an additional unique tag_id provided by Beacon in the WITH > clause in repl load command to be used to tag distcp jobs ). Enhance the kill > api to take the tag as input and kill jobs associated with that tag. Problem > here is how do we validate the association of the tag with a hive query id to > make sure this api is not used to kill jobs run by other components, however > we can provide this capability to only admins and should be ok in that case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19441) Use LLAP test Driver for Druid test suites
[
https://issues.apache.org/jira/browse/HIVE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19441:
--
Attachment: HIVE-19441.11.patch
> Use LLAP test Driver for Druid test suites
> --
>
> Key: HIVE-19441
> URL: https://issues.apache.org/jira/browse/HIVE-19441
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19441.10.patch, HIVE-19441.11.patch,
> HIVE-19441.2.patch, HIVE-19441.3.patch, HIVE-19441.4.patch,
> HIVE-19441.5.patch, HIVE-19441.6.patch, HIVE-19441.7.patch,
> HIVE-19441.8.patch, HIVE-19441.8.patch, HIVE-19441.9.patch, HIVE-19441.patch
>
>
> Adding support to the float kind aggregator.
> Use LLAP as test Driver to reduce execution time of tests from about 2 hours
> to 15 min:
> Although this patches unveiling an issue with timezone, maybe it is fixed by
> [~jcamachorodriguez] upcoming set of patches.
>
> Before
> {code}
> [INFO] Executed tasks
> [INFO]
> [INFO] --- maven-compiler-plugin:3.6.1:testCompile (default-testCompile) @
> hive-it-qfile ---
> [INFO] Compiling 21 source files to
> /Users/sbouguerra/Hdev/hive/itests/qtest/target/test-classes
> [INFO]
> [INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ hive-it-qfile
> ---
> [INFO]
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 6,654.117 s - in org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 01:51 h
> [INFO] Finished at: 2018-05-04T12:43:19-07:00
> [INFO]
>
> {code}
> After
> {code}
> INFO] Executed tasks
> [INFO]
> [INFO] --- maven-compiler-plugin:3.6.1:testCompile (default-testCompile) @
> hive-it-qfile ---
> [INFO] Compiling 22 source files to
> /Users/sbouguerra/Hdev/hive/itests/qtest/target/test-classes
> [INFO]
> [INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ hive-it-qfile
> ---
> [INFO]
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 907.167 s - in org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 15:31 min
> [INFO] Finished at: 2018-05-04T13:15:11-07:00
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
