[jira] [Updated] (HIVE-20382) Materialized views: Introduce heuristic to favour incremental rebuild
[ https://issues.apache.org/jira/browse/HIVE-20382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20382: --- Description: Currently, we do not expose stats over ROW\_\_ID.writeId to the optimizer (this should be fixed by HIVE-20313). Even if we did, we always assume uniform distribution of the column values, which can easily lead to overestimations on the number of rows read when we filter on ROW\_\_ID.writeId for materialized views (think about a large transaction for MV creation and then small ones for incremental maintenance). This overestimation can lead to incremental view maintenance not being triggered as cost of the incremental plan is overestimated (we think we will read more rows than we actually do). This could be fixed by introducing histograms that reflect better the column values distribution. Till both fixes are implemented, we will use a config variable that will multiply the estimated cost of the rebuild plan and hence will be able to favour incremental rebuild over full rebuild. was: Currently, we do not expose stats over ROW__ID.writeId to the optimizer (this should be fixed by HIVE-20313). Even if we did, we always assume uniform distribution of the column values, which can easily lead to overestimations on the number of rows read when we filter on ROW__ID.writeId for materialized views (think about a large transaction for MV creation and then small ones for incremental maintenance). This overestimation can lead to incremental view maintenance not being triggered as cost of the incremental plan is overestimated (we think we will read more rows than we actually do). This could be fixed by introducing histograms that reflect better the column values distribution. Till both fixes are implemented, we will use a config variable that will multiply the estimated cost of the rebuild plan and hence will be able to favour incremental rebuild over full rebuild. > Materialized views: Introduce heuristic to favour incremental rebuild > - > > Key: HIVE-20382 > URL: https://issues.apache.org/jira/browse/HIVE-20382 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20382.patch > > > Currently, we do not expose stats over ROW\_\_ID.writeId to the optimizer > (this should be fixed by HIVE-20313). Even if we did, we always assume > uniform distribution of the column values, which can easily lead to > overestimations on the number of rows read when we filter on > ROW\_\_ID.writeId for materialized views (think about a large transaction for > MV creation and then small ones for incremental maintenance). This > overestimation can lead to incremental view maintenance not being triggered > as cost of the incremental plan is overestimated (we think we will read more > rows than we actually do). This could be fixed by introducing histograms that > reflect better the column values distribution. > Till both fixes are implemented, we will use a config variable that will > multiply the estimated cost of the rebuild plan and hence will be able to > favour incremental rebuild over full rebuild. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-20382) Materialized views: Introduce heuristic to favour incremental rebuild
[ https://issues.apache.org/jira/browse/HIVE-20382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-20382 started by Jesus Camacho Rodriguez. -- > Materialized views: Introduce heuristic to favour incremental rebuild > - > > Key: HIVE-20382 > URL: https://issues.apache.org/jira/browse/HIVE-20382 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20382.patch > > > Currently, we do not expose stats over ROW__ID.writeId to the optimizer (this > should be fixed by HIVE-20313). Even if we did, we always assume uniform > distribution of the column values, which can easily lead to overestimations > on the number of rows read when we filter on ROW__ID.writeId for materialized > views (think about a large transaction for MV creation and then small ones > for incremental maintenance). This overestimation can lead to incremental > view maintenance not being triggered as cost of the incremental plan is > overestimated (we think we will read more rows than we actually do). This > could be fixed by introducing histograms that reflect better the column > values distribution. > Till both fixes are implemented, we will use a config variable that will > multiply the estimated cost of the rebuild plan and hence will be able to > favour incremental rebuild over full rebuild. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20382) Materialized views: Introduce heuristic to favour incremental rebuild
[ https://issues.apache.org/jira/browse/HIVE-20382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20382: --- Attachment: HIVE-20382.patch > Materialized views: Introduce heuristic to favour incremental rebuild > - > > Key: HIVE-20382 > URL: https://issues.apache.org/jira/browse/HIVE-20382 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20382.patch > > > Currently, we do not expose stats over ROW__ID.writeId to the optimizer (this > should be fixed by HIVE-20313). Even if we did, we always assume uniform > distribution of the column values, which can easily lead to overestimations > on the number of rows read when we filter on ROW__ID.writeId for materialized > views (think about a large transaction for MV creation and then small ones > for incremental maintenance). This overestimation can lead to incremental > view maintenance not being triggered as cost of the incremental plan is > overestimated (we think we will read more rows than we actually do). This > could be fixed by introducing histograms that reflect better the column > values distribution. > Till both fixes are implemented, we will use a config variable that will > multiply the estimated cost of the rebuild plan and hence will be able to > favour incremental rebuild over full rebuild. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20382) Materialized views: Introduce heuristic to favour incremental rebuild
[ https://issues.apache.org/jira/browse/HIVE-20382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20382: --- Status: Patch Available (was: In Progress) > Materialized views: Introduce heuristic to favour incremental rebuild > - > > Key: HIVE-20382 > URL: https://issues.apache.org/jira/browse/HIVE-20382 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20382.patch > > > Currently, we do not expose stats over ROW__ID.writeId to the optimizer (this > should be fixed by HIVE-20313). Even if we did, we always assume uniform > distribution of the column values, which can easily lead to overestimations > on the number of rows read when we filter on ROW__ID.writeId for materialized > views (think about a large transaction for MV creation and then small ones > for incremental maintenance). This overestimation can lead to incremental > view maintenance not being triggered as cost of the incremental plan is > overestimated (we think we will read more rows than we actually do). This > could be fixed by introducing histograms that reflect better the column > values distribution. > Till both fixes are implemented, we will use a config variable that will > multiply the estimated cost of the rebuild plan and hence will be able to > favour incremental rebuild over full rebuild. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20382) Materialized views: Introduce heuristic to favour incremental rebuild
[ https://issues.apache.org/jira/browse/HIVE-20382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20382: --- Description: Currently, we do not expose stats over ROW__ID.writeId to the optimizer (this should be fixed by HIVE-20313). Even if we did, we always assume uniform distribution of the column values, which can easily lead to overestimations on the number of rows read when we filter on ROW__ID.writeId for materialized views (think about a large transaction for MV creation and then small ones for incremental maintenance). This overestimation can lead to incremental view maintenance not being triggered as cost of the incremental plan is overestimated (we think we will read more rows than we actually do). This could be fixed by introducing histograms that reflect better the column values distribution. Till both fixes are implemented, we will use a config variable that will multiply the estimated cost of the rebuild plan and hence will be able to favour incremental rebuild over full rebuild. > Materialized views: Introduce heuristic to favour incremental rebuild > - > > Key: HIVE-20382 > URL: https://issues.apache.org/jira/browse/HIVE-20382 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Currently, we do not expose stats over ROW__ID.writeId to the optimizer (this > should be fixed by HIVE-20313). Even if we did, we always assume uniform > distribution of the column values, which can easily lead to overestimations > on the number of rows read when we filter on ROW__ID.writeId for materialized > views (think about a large transaction for MV creation and then small ones > for incremental maintenance). This overestimation can lead to incremental > view maintenance not being triggered as cost of the incremental plan is > overestimated (we think we will read more rows than we actually do). This > could be fixed by introducing histograms that reflect better the column > values distribution. > Till both fixes are implemented, we will use a config variable that will > multiply the estimated cost of the rebuild plan and hence will be able to > favour incremental rebuild over full rebuild. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20332) Materialized views: Introduce heuristic on selectivity over ROW__ID to favour incremental rebuild
[
https://issues.apache.org/jira/browse/HIVE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20332:
---
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
We will go with approach in HIVE-20382.
> Materialized views: Introduce heuristic on selectivity over ROW__ID to favour
> incremental rebuild
> -
>
> Key: HIVE-20332
> URL: https://issues.apache.org/jira/browse/HIVE-20332
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20332.01.patch, HIVE-20332.01.patch,
> HIVE-20332.patch
>
>
> Currently, we do not expose stats over {{ROW\_\_ID.writeId}} to the optimizer
> (this should be fixed by HIVE-20313). Even if we did, we always assume
> uniform distribution of the column values, which can easily lead to
> overestimations on the number of rows read when we filter on
> {{ROW\_\_ID.writeId}} for materialized views (think about a large transaction
> for MV creation and then small ones for incremental maintenance). This
> overestimation can lead to incremental view maintenance not being triggered
> as cost of the incremental plan is overestimated (we think we will read more
> rows than we actually do). This could be fixed by introducing histograms that
> reflect better the column values distribution.
> Till both fixes are implemented, we will use a config variable that will set
> the selectivity for filter condition on {{ROW\_\_ID}} during the cost
> calculation. Setting that variable to a low value will favour incremental
> rebuild over full rebuild.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20382) Materialized views: Introduce heuristic to favour incremental rebuild
[ https://issues.apache.org/jira/browse/HIVE-20382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-20382: -- > Materialized views: Introduce heuristic to favour incremental rebuild > - > > Key: HIVE-20382 > URL: https://issues.apache.org/jira/browse/HIVE-20382 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19316) StatsTask fails due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-19316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579273#comment-16579273
]
Hive QA commented on HIVE-19316:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13203/dev-support/hive-personality.sh
|
| git revision | master / 681051f |
| Default Java | 1.8.0_111 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13203/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13203/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13203/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> StatsTask fails due to ClassCastException
> -
>
> Key: HIVE-19316
> URL: https://issues.apache.org/jira/browse/HIVE-19316
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Rui Li
>Assignee: Jaume M
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19316.1.patch, HIVE-19316.2.patch,
> HIVE-19316.3.patch, HIVE-19316.4.patch, HIVE-19316.5.patch
>
>
> The stack trace:
> {noformat}
> 2018-04-26T20:17:37,674 ERROR [pool-7-thread-11]
> metastore.RetryingHMSHandler: java.lang.ClassCastException:
> org.apache.hadoop.hive.metastore.api.LongColumnStatsData cannot be cast to
> org.apache.hadoop.hive.metastore.columnstats.cache.LongColumnStatsDataInspector
> at
> org.apache.hadoop.hive.metastore.columnstats.merge.LongColumnStatsMerger.merge(LongColumnStatsMerger.java:30)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:1052)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
[jira] [Commented] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579270#comment-16579270
]
Hive QA commented on HIVE-20377:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
19s{color} | {color:blue} itests/qtest-druid in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
50s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} llap-server in master has 84 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
16s{color} | {color:blue} ql in master has 2306 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} itests/qtest: The patch generated 1 new + 3 unchanged
- 0 fixed = 4 total (was 3) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} kafka-handler: The patch generated 332 new + 0
unchanged - 0 fixed = 332 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} llap-server: The patch generated 1 new + 29 unchanged
- 1 fixed = 30 total (was 30) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} patch/itests/qtest-druid cannot run setBugDatabaseInfo
from findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
46s{color} | {color:red} patch/itests/util cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} patch/kafka-handler cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
42s{color} | {color:red} patch/llap-server cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
19s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 18m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality
[jira] [Commented] (HIVE-20377) Hive Kafka Storage Handler
[jira] [Commented] (HIVE-18620) Improve error message while dropping a table that is part of a materialized view
[
https://issues.apache.org/jira/browse/HIVE-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579200#comment-16579200
]
Hive QA commented on HIVE-18620:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935431/HIVE-18620.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14873 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=193)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_used_by_mv]
(batchId=97)
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMergeOnTezEdges
(batchId=299)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13201/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13201/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13201/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935431 - PreCommit-HIVE-Build
> Improve error message while dropping a table that is part of a materialized
> view
>
>
> Key: HIVE-18620
> URL: https://issues.apache.org/jira/browse/HIVE-18620
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-18620.patch
>
>
> When we want to drop a table used by a materialized view, we prevent dropping
> that table. However, the message shown is not very meaningful (FK-PK
> violation).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20150) TopNKey pushdown
[ https://issues.apache.org/jira/browse/HIVE-20150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20150: -- Attachment: HIVE-20150.5.patch > TopNKey pushdown > > > Key: HIVE-20150 > URL: https://issues.apache.org/jira/browse/HIVE-20150 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 4.0.0 >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20150.1.patch, HIVE-20150.2.patch, > HIVE-20150.4.patch, HIVE-20150.5.patch > > > TopNKey operator is implemented in HIVE-17896, but it needs more work in > pushdown implementation. So this issue covers TopNKey pushdown implementation > with proper tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18620) Improve error message while dropping a table that is part of a materialized view
[
https://issues.apache.org/jira/browse/HIVE-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579155#comment-16579155
]
Hive QA commented on HIVE-18620:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 2306 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13201/dev-support/hive-personality.sh
|
| git revision | master / 681051f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13201/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve error message while dropping a table that is part of a materialized
> view
>
>
> Key: HIVE-18620
> URL: https://issues.apache.org/jira/browse/HIVE-18620
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-18620.patch
>
>
> When we want to drop a table used by a materialized view, we prevent dropping
> that table. However, the message shown is not very meaningful (FK-PK
> violation).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20304) When hive.optimize.skewjoin and hive.auto.convert.join are both set to true, and the execution engine is mr, same stage may launch twice due to the wrong generated plan
[
https://issues.apache.org/jira/browse/HIVE-20304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianguo Tian updated HIVE-20304:
Description:
`When hive.optimize.skewjoin and hive.auto.convert.join are both set to true,
and the execution engine is set to mr, same stage of a query may launch twice
due to the wrong generated plan. If hive.exec.parallel is also true, the same
stage will launch at the same time and the job will failed due to the first
completed stage clear the map.xml/reduce.xml file stored in the hdfs.
use following sql to reproduce the issue:
{code:java}
CREATE TABLE `tbl1`(
`fence` string);
CREATE TABLE `tbl2`(
`order_id` string,
`phone` string,
`search_id` string
)
PARTITIONED BY (
`dt` string);
CREATE TABLE `tbl3`(
`order_id` string,
`platform` string)
PARTITIONED BY (
`dt` string);
CREATE TABLE `tbl4`(
`groupname` string,
`phone` string)
PARTITIONED BY (
`dt` string);
CREATE TABLE `tbl5`(
`search_id` string,
`fence` string)
PARTITIONED BY (
`dt` string);
SET hive.exec.parallel = TRUE;
SET hive.auto.convert.join = TRUE;
SET hive.optimize.skewjoin = TRUE;
SELECT dt,
platform,
groupname,
count(1) as cnt
FROM
(SELECT dt,
platform,
groupname
FROM
(SELECT fence
FROM tbl1)ta
JOIN
(SELECT a0.dt,
a1.platform,
a2.groupname,
a3.fence
FROM
(SELECT dt,
order_id,
phone,
search_id
FROM tbl2
WHERE dt =20180703 )a0
JOIN
(SELECT order_id,
platform,
dt
FROM tbl3
WHERE dt =20180703 )a1 ON a0.order_id = a1.order_id
INNER JOIN
(SELECT groupname,
phone,
dt
FROM tbl4
WHERE dt =20180703 )a2 ON a0.phone = a2.phone
LEFT JOIN
(SELECT search_id,
fence,
dt
FROM tbl5
WHERE dt =20180703)a3 ON a0.search_id = a3.search_id)t0 ON
ta.fence = t0.fence)t11
GROUP BY dt,
platform,
groupname;
DROP TABLE tbl1;
DROP TABLE tbl2;
DROP TABLE tbl3;
DROP TABLE tbl4;
DROP TABLE tbl5;
{code}
We will get some error message like this:
Examining task ID: task_1531284442065_3637_m_00 (and more) from job
job_1531284442065_3637
Task with the most failures(4):
Task ID:
task_1531284442065_3637_m_00
URL:
[http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1531284442065_3637=task_1531284442065_3637_m_00]
Diagnostic Messages for this Task:
File does not exist:
hdfs://test/tmp/hive-hadoop/hadoop/fe5efa94-abb1-420f-b6ba-ec782e7b79ad/hive_2018-08-03_17-00-17_707_592882314975289971-5/-mr-10045/757eb1f7-7a37-4a7e-abc0-4a3b8b06510c/reduce.xml
java.io.FileNotFoundException: File does not exist:
hdfs://test/tmp/hive-hadoop/hadoop/fe5efa94-abb1-420f-b6ba-ec782e7b79ad/hive_2018-08-03_17-00-17_707_592882314975289971-5/-mr-10045/757eb1f7-7a37-4a7e-abc0-4a3b8b06510c/reduce.xml
Looking into the plan by executing explain, I found that the Stage-4 and
Stage-5 can reached from multi root tasks.
{code:java}
Explain
STAGE DEPENDENCIES:
Stage-21 is a root stage , consists of Stage-34, Stage-5
Stage-34 has a backup stage: Stage-5
Stage-20 depends on stages: Stage-34
Stage-17 depends on stages: Stage-5, Stage-18, Stage-20 , consists of
Stage-32, Stage-33, Stage-1
Stage-32 has a backup stage: Stage-1
Stage-15 depends on stages: Stage-32
Stage-10 depends on stages: Stage-1, Stage-15, Stage-16 , consists of
Stage-31, Stage-2
Stage-31
Stage-9 depends on stages: Stage-31
Stage-2 depends on stages: Stage-9
Stage-33 has a backup stage: Stage-1
Stage-16 depends on stages: Stage-33
Stage-1
Stage-5
Stage-27 is a root stage , consists of Stage-37, Stage-38, Stage-4
Stage-37 has a backup stage: Stage-4
Stage-25 depends on stages: Stage-37
Stage-12 depends on stages: Stage-4, Stage-22, Stage-23, Stage-25, Stage-26 ,
consists of Stage-36, Stage-5
Stage-36
Stage-11 depends on stages: Stage-36
Stage-19 depends on stages: Stage-11 , consists of Stage-35, Stage-5
Stage-35 has a backup stage: Stage-5
Stage-18 depends on stages: Stage-35
Stage-38 has a backup stage: Stage-4
Stage-26 depends on stages: Stage-38
Stage-4
Stage-30 is a root stage , consists of Stage-42, Stage-43, Stage-3
Stage-42 has a backup stage: Stage-3
Stage-28 depends on stages: Stage-42
Stage-14 depends on stages: Stage-3, Stage-28, Stage-29 , consists of
Stage-41, Stage-4
Stage-41
Stage-13 depends on stages: Stage-41
Stage-24 depends on stages: Stage-13 ,
[jira] [Commented] (HIVE-20311) add txn stats checks to some more paths
[ https://issues.apache.org/jira/browse/HIVE-20311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579149#comment-16579149 ] Sergey Shelukhin commented on HIVE-20311: - Some trivial out file changes... jira ate my comment > add txn stats checks to some more paths > --- > > Key: HIVE-20311 > URL: https://issues.apache.org/jira/browse/HIVE-20311 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20311.01.patch, HIVE-20311.patch > > > These were set to false in the original patch for no reason as far as I see. > I later added notes but not TODOs to switch them over, so they remained as > non-txn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-20343:
Status: Patch Available (was: Open)
> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20343.patch
>
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc TBLPROPERTIES
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected
> "insert_only"
> {code}
> describe formatted ctasexampleinsertonly
>
> +---++---+
> | col_name| data_type
>|comment|
> +---++---+
> | # col_name| data_type
>| comment |
> | name | varchar(8)
>| |
> | time | double
>| |
> | | NULL
>| NULL |
> | # Detailed Table Information | NULL
>| NULL |
> | Database: | default
>| NULL |
> | OwnerType:| USER
>| NULL |
> | Owner:| hive
>| NULL |
> | CreateTime: | Wed Aug 08 21:35:15 UTC 2018
>| NULL |
> | LastAccessTime: | UNKNOWN
>| NULL |
> | Retention:| 0
>| NULL |
> | Location: |
> hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
> | NULL |
> | Table Type: | MANAGED_TABLE
>| NULL |
> | Table Parameters: | NULL
>| NULL |
> | | COLUMN_STATS_ACCURATE
>| {}|
> | | bucketing_version
>| 2 |
> | | numFiles
>| 1 |
> | | numRows
>| 1 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 754 |
> | | transactional
>| true |
> | | transactional_properties
>| default |
> | | transient_lastDdlTime
>| 1533764115|
> | | NULL
>| NULL |
> | # Storage Information | NULL
>| NULL |
> | SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde
>| NULL |
> | InputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL |
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL |
> | Compressed: | No
>| NULL |
> | Num Buckets: | -1
>| NULL |
> | Bucket Columns: | []
[jira] [Commented] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579147#comment-16579147
]
Sergey Shelukhin commented on HIVE-20343:
-
[~ekoifman] can you take a look? a small patch
> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20343.patch
>
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc TBLPROPERTIES
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected
> "insert_only"
> {code}
> describe formatted ctasexampleinsertonly
>
> +---++---+
> | col_name| data_type
>|comment|
> +---++---+
> | # col_name| data_type
>| comment |
> | name | varchar(8)
>| |
> | time | double
>| |
> | | NULL
>| NULL |
> | # Detailed Table Information | NULL
>| NULL |
> | Database: | default
>| NULL |
> | OwnerType:| USER
>| NULL |
> | Owner:| hive
>| NULL |
> | CreateTime: | Wed Aug 08 21:35:15 UTC 2018
>| NULL |
> | LastAccessTime: | UNKNOWN
>| NULL |
> | Retention:| 0
>| NULL |
> | Location: |
> hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
> | NULL |
> | Table Type: | MANAGED_TABLE
>| NULL |
> | Table Parameters: | NULL
>| NULL |
> | | COLUMN_STATS_ACCURATE
>| {}|
> | | bucketing_version
>| 2 |
> | | numFiles
>| 1 |
> | | numRows
>| 1 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 754 |
> | | transactional
>| true |
> | | transactional_properties
>| default |
> | | transient_lastDdlTime
>| 1533764115|
> | | NULL
>| NULL |
> | # Storage Information | NULL
>| NULL |
> | SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde
>| NULL |
> | InputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL |
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL |
> | Compressed: | No
>| NULL |
> | Num Buckets: | -1
>| NULL
[jira] [Updated] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-20343:
Attachment: HIVE-20343.patch
> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20343.patch
>
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc TBLPROPERTIES
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected
> "insert_only"
> {code}
> describe formatted ctasexampleinsertonly
>
> +---++---+
> | col_name| data_type
>|comment|
> +---++---+
> | # col_name| data_type
>| comment |
> | name | varchar(8)
>| |
> | time | double
>| |
> | | NULL
>| NULL |
> | # Detailed Table Information | NULL
>| NULL |
> | Database: | default
>| NULL |
> | OwnerType:| USER
>| NULL |
> | Owner:| hive
>| NULL |
> | CreateTime: | Wed Aug 08 21:35:15 UTC 2018
>| NULL |
> | LastAccessTime: | UNKNOWN
>| NULL |
> | Retention:| 0
>| NULL |
> | Location: |
> hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
> | NULL |
> | Table Type: | MANAGED_TABLE
>| NULL |
> | Table Parameters: | NULL
>| NULL |
> | | COLUMN_STATS_ACCURATE
>| {}|
> | | bucketing_version
>| 2 |
> | | numFiles
>| 1 |
> | | numRows
>| 1 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 754 |
> | | transactional
>| true |
> | | transactional_properties
>| default |
> | | transient_lastDdlTime
>| 1533764115|
> | | NULL
>| NULL |
> | # Storage Information | NULL
>| NULL |
> | SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde
>| NULL |
> | InputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL |
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL |
> | Compressed: | No
>| NULL |
> | Num Buckets: | -1
>| NULL |
> | Bucket Columns: | []
[jira] [Updated] (HIVE-20378) don't update stats during alter for txn table conversion
[ https://issues.apache.org/jira/browse/HIVE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20378: Attachment: HIVE-20378.01.patch > don't update stats during alter for txn table conversion > > > Key: HIVE-20378 > URL: https://issues.apache.org/jira/browse/HIVE-20378 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20378.01.patch, HIVE-20378.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16116) Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
[
https://issues.apache.org/jira/browse/HIVE-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579139#comment-16579139
]
Hive QA commented on HIVE-16116:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856276/HIVE-16116.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14879 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13200/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13200/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13200/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12856276 - PreCommit-HIVE-Build
> Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
> --
>
> Key: HIVE-16116
> URL: https://issues.apache.org/jira/browse/HIVE-16116
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16116.1.patch, HIVE-16116.2.patch
>
>
> Env: hive master
> Steps to reproduce:
> 1. clear previous beeline.properties (rm -rf ~/.beeline/beeline.properties)
> 2. Launch beeline, "!save" and exit. This would create new
> "~/.beeline/beeline.properties", which would have
> "beeline.hiveconfvariables={}"
> 3. Launch "beeline --hiveconf hive.tmp.dir=/tmp". This would throw NPE
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.setHiveConfVar(BeeLine.java:885)
> at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:832)
> at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:775)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1009)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579135#comment-16579135
]
Sergey Shelukhin commented on HIVE-20343:
-
Confirmed after some testing... if neither of the default flags is set, the
incorrect table is created with txn props but no txn.
If mm is on by default, everything is fine.
If ACID is on by default, the conversion code treats the table as non-txn
(similar to the first case) and then replaces txn=false, props=insert_only with
full ACID.
I think the solution is to outlaw this syntax.. if properties are specified
explicitly, the flag should also be specified, it's not good for us to guess
what user wanted based on partial specification, given that there are 4
combinations of default options.
Location is a separate issue.
> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Assignee: Sergey Shelukhin
>Priority: Major
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc TBLPROPERTIES
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected
> "insert_only"
> {code}
> describe formatted ctasexampleinsertonly
>
> +---++---+
> | col_name| data_type
>|comment|
> +---++---+
> | # col_name| data_type
>| comment |
> | name | varchar(8)
>| |
> | time | double
>| |
> | | NULL
>| NULL |
> | # Detailed Table Information | NULL
>| NULL |
> | Database: | default
>| NULL |
> | OwnerType:| USER
>| NULL |
> | Owner:| hive
>| NULL |
> | CreateTime: | Wed Aug 08 21:35:15 UTC 2018
>| NULL |
> | LastAccessTime: | UNKNOWN
>| NULL |
> | Retention:| 0
>| NULL |
> | Location: |
> hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
> | NULL |
> | Table Type: | MANAGED_TABLE
>| NULL |
> | Table Parameters: | NULL
>| NULL |
> | | COLUMN_STATS_ACCURATE
>| {}|
> | | bucketing_version
>| 2 |
> | | numFiles
>| 1 |
> | | numRows
>| 1 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 754 |
> | | transactional
>| true |
> | | transactional_properties
>| default |
> | | transient_lastDdlTime
>| 1533764115|
> | | NULL
>| NULL |
> | # Storage Information | NULL
>| NULL |
> | SerDe Library:
[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-20246:
-
Status: Open (was: Patch Available)
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.5.patch
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
> key2=val2);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-20246:
-
Attachment: (was: HIVE-20246.5.patch)
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.5.patch
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
> key2=val2);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-20246:
-
Attachment: HIVE-20246.5.patch
Status: Patch Available (was: Open)
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.5.patch
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
> key2=val2);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20311) add txn stats checks to some more paths
[ https://issues.apache.org/jira/browse/HIVE-20311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20311: Attachment: HIVE-20311.01.patch > add txn stats checks to some more paths > --- > > Key: HIVE-20311 > URL: https://issues.apache.org/jira/browse/HIVE-20311 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20311.01.patch, HIVE-20311.patch > > > These were set to false in the original patch for no reason as far as I see. > I later added notes but not TODOs to switch them over, so they remained as > non-txn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-20246:
-
Attachment: HIVE-20246.5.patch
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.5.patch
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
> key2=val2);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-20246:
-
Attachment: (was: HIVE-20246.5.patch)
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.5.patch
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
> key2=val2);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16116) Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
[
https://issues.apache.org/jira/browse/HIVE-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579110#comment-16579110
]
Hive QA commented on HIVE-16116:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} beeline in master has 53 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} beeline: The patch generated 1 new + 47 unchanged - 0
fixed = 48 total (was 47) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13200/dev-support/hive-personality.sh
|
| git revision | master / 681051f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13200/yetus/diff-checkstyle-beeline.txt
|
| modules | C: beeline U: beeline |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13200/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
> --
>
> Key: HIVE-16116
> URL: https://issues.apache.org/jira/browse/HIVE-16116
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16116.1.patch, HIVE-16116.2.patch
>
>
> Env: hive master
> Steps to reproduce:
> 1. clear previous beeline.properties (rm -rf ~/.beeline/beeline.properties)
> 2. Launch beeline, "!save" and exit. This would create new
> "~/.beeline/beeline.properties", which would have
> "beeline.hiveconfvariables={}"
> 3. Launch "beeline --hiveconf hive.tmp.dir=/tmp". This would throw NPE
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.setHiveConfVar(BeeLine.java:885)
> at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:832)
> at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:775)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1009)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
[jira] [Commented] (HIVE-20378) don't update stats during alter for txn table conversion
[
https://issues.apache.org/jira/browse/HIVE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579099#comment-16579099
]
Hive QA commented on HIVE-20378:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935430/HIVE-20378.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14877 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestHiveAlterHandler.testAlterTableDelColUpdateStats
(batchId=219)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13199/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13199/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13199/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935430 - PreCommit-HIVE-Build
> don't update stats during alter for txn table conversion
>
>
> Key: HIVE-20378
> URL: https://issues.apache.org/jira/browse/HIVE-20378
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20378.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-20343:
---
Assignee: Sergey Shelukhin
> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Assignee: Sergey Shelukhin
>Priority: Major
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc TBLPROPERTIES
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected
> "insert_only"
> {code}
> describe formatted ctasexampleinsertonly
>
> +---++---+
> | col_name| data_type
>|comment|
> +---++---+
> | # col_name| data_type
>| comment |
> | name | varchar(8)
>| |
> | time | double
>| |
> | | NULL
>| NULL |
> | # Detailed Table Information | NULL
>| NULL |
> | Database: | default
>| NULL |
> | OwnerType:| USER
>| NULL |
> | Owner:| hive
>| NULL |
> | CreateTime: | Wed Aug 08 21:35:15 UTC 2018
>| NULL |
> | LastAccessTime: | UNKNOWN
>| NULL |
> | Retention:| 0
>| NULL |
> | Location: |
> hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
> | NULL |
> | Table Type: | MANAGED_TABLE
>| NULL |
> | Table Parameters: | NULL
>| NULL |
> | | COLUMN_STATS_ACCURATE
>| {}|
> | | bucketing_version
>| 2 |
> | | numFiles
>| 1 |
> | | numRows
>| 1 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 754 |
> | | transactional
>| true |
> | | transactional_properties
>| default |
> | | transient_lastDdlTime
>| 1533764115|
> | | NULL
>| NULL |
> | # Storage Information | NULL
>| NULL |
> | SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde
>| NULL |
> | InputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL |
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL |
> | Compressed: | No
>| NULL |
> | Num Buckets: | -1
>| NULL |
> | Bucket Columns: | []
>
[jira] [Commented] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579091#comment-16579091
]
Sergey Shelukhin commented on HIVE-20343:
-
Probably default-to-acid thing is messing with this somehow.
> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Priority: Major
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc TBLPROPERTIES
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected
> "insert_only"
> {code}
> describe formatted ctasexampleinsertonly
>
> +---++---+
> | col_name| data_type
>|comment|
> +---++---+
> | # col_name| data_type
>| comment |
> | name | varchar(8)
>| |
> | time | double
>| |
> | | NULL
>| NULL |
> | # Detailed Table Information | NULL
>| NULL |
> | Database: | default
>| NULL |
> | OwnerType:| USER
>| NULL |
> | Owner:| hive
>| NULL |
> | CreateTime: | Wed Aug 08 21:35:15 UTC 2018
>| NULL |
> | LastAccessTime: | UNKNOWN
>| NULL |
> | Retention:| 0
>| NULL |
> | Location: |
> hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
> | NULL |
> | Table Type: | MANAGED_TABLE
>| NULL |
> | Table Parameters: | NULL
>| NULL |
> | | COLUMN_STATS_ACCURATE
>| {}|
> | | bucketing_version
>| 2 |
> | | numFiles
>| 1 |
> | | numRows
>| 1 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 754 |
> | | transactional
>| true |
> | | transactional_properties
>| default |
> | | transient_lastDdlTime
>| 1533764115|
> | | NULL
>| NULL |
> | # Storage Information | NULL
>| NULL |
> | SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde
>| NULL |
> | InputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL |
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL |
> | Compressed: | No
>| NULL |
> | Num Buckets: | -1
>| NULL |
> | Bucket Columns: | []
[jira] [Updated] (HIVE-20298) Illegal null value in column `TBLS`.`WRITE_ID`
[
https://issues.apache.org/jira/browse/HIVE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20298:
--
Attachment: (was: HIVE-20298.1.patch)
> Illegal null value in column `TBLS`.`WRITE_ID`
> --
>
> Key: HIVE-20298
> URL: https://issues.apache.org/jira/browse/HIVE-20298
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20298.patch
>
>
> Manually upgraded my existing local metastore using
> upgrade-3.0.0-to-3.1.0.mysql.sql, upgrade-3.1.0-to-3.2.0.mysql.sql,
> upgrade-3.2.0-to-4.0.0.mysql.sql. When running DESCRIBE EXTENDED of an
> existing table, I was getting the following error in hive.log. It looks like
> the ObjectStore/MTable classes don't seem to be able to support null values
> in the new writeId column that was added to the TBLS table in the metastore.
> cc [~sershe] [~ekoifman]
> {noformat}
> Caused by: javax.jdo.JDODataStoreException: Illegal null value in column
> `TBLS`.`WRITE_ID`
> NestedThrowables:
> org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal null value
> in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:553)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1802)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1838)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1424)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy39.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2950)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:2898)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:2882)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ... 36 more
> Caused by: org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal
> null value in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.store.rdbms.mapping.datastore.BigIntRDBMSMapping.getLong(BigIntRDBMSMapping.java:140)
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getLong(SingleFieldMapping.java:155)
> at
> org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchLongField(ResultSetGetter.java:124)
> at
> org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:1549)
> at
> org.datanucleus.state.StateManagerImpl.replacingLongField(StateManagerImpl.java:120)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceField(MTable.java)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceFields(MTable.java)
> at
> org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
> at
> org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
> at
> org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
> at
> org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
> at
> org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
> at
>
[jira] [Commented] (HIVE-20298) Illegal null value in column `TBLS`.`WRITE_ID`
[
https://issues.apache.org/jira/browse/HIVE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579090#comment-16579090
]
Jason Dere commented on HIVE-20298:
---
+1
> Illegal null value in column `TBLS`.`WRITE_ID`
> --
>
> Key: HIVE-20298
> URL: https://issues.apache.org/jira/browse/HIVE-20298
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20298.patch
>
>
> Manually upgraded my existing local metastore using
> upgrade-3.0.0-to-3.1.0.mysql.sql, upgrade-3.1.0-to-3.2.0.mysql.sql,
> upgrade-3.2.0-to-4.0.0.mysql.sql. When running DESCRIBE EXTENDED of an
> existing table, I was getting the following error in hive.log. It looks like
> the ObjectStore/MTable classes don't seem to be able to support null values
> in the new writeId column that was added to the TBLS table in the metastore.
> cc [~sershe] [~ekoifman]
> {noformat}
> Caused by: javax.jdo.JDODataStoreException: Illegal null value in column
> `TBLS`.`WRITE_ID`
> NestedThrowables:
> org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal null value
> in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:553)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1802)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1838)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1424)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy39.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2950)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:2898)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:2882)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ... 36 more
> Caused by: org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal
> null value in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.store.rdbms.mapping.datastore.BigIntRDBMSMapping.getLong(BigIntRDBMSMapping.java:140)
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getLong(SingleFieldMapping.java:155)
> at
> org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchLongField(ResultSetGetter.java:124)
> at
> org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:1549)
> at
> org.datanucleus.state.StateManagerImpl.replacingLongField(StateManagerImpl.java:120)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceField(MTable.java)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceFields(MTable.java)
> at
> org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
> at
> org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
> at
> org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
> at
> org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
> at
> org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
> at
>
[jira] [Updated] (HIVE-20298) Illegal null value in column `TBLS`.`WRITE_ID`
[
https://issues.apache.org/jira/browse/HIVE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-20298:
Attachment: HIVE-20298.patch
> Illegal null value in column `TBLS`.`WRITE_ID`
> --
>
> Key: HIVE-20298
> URL: https://issues.apache.org/jira/browse/HIVE-20298
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20298.1.patch, HIVE-20298.patch
>
>
> Manually upgraded my existing local metastore using
> upgrade-3.0.0-to-3.1.0.mysql.sql, upgrade-3.1.0-to-3.2.0.mysql.sql,
> upgrade-3.2.0-to-4.0.0.mysql.sql. When running DESCRIBE EXTENDED of an
> existing table, I was getting the following error in hive.log. It looks like
> the ObjectStore/MTable classes don't seem to be able to support null values
> in the new writeId column that was added to the TBLS table in the metastore.
> cc [~sershe] [~ekoifman]
> {noformat}
> Caused by: javax.jdo.JDODataStoreException: Illegal null value in column
> `TBLS`.`WRITE_ID`
> NestedThrowables:
> org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal null value
> in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:553)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1802)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1838)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1424)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy39.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2950)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:2898)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:2882)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ... 36 more
> Caused by: org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal
> null value in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.store.rdbms.mapping.datastore.BigIntRDBMSMapping.getLong(BigIntRDBMSMapping.java:140)
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getLong(SingleFieldMapping.java:155)
> at
> org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchLongField(ResultSetGetter.java:124)
> at
> org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:1549)
> at
> org.datanucleus.state.StateManagerImpl.replacingLongField(StateManagerImpl.java:120)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceField(MTable.java)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceFields(MTable.java)
> at
> org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
> at
> org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
> at
> org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
> at
> org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
> at
> org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
> at
>
[jira] [Updated] (HIVE-20298) Illegal null value in column `TBLS`.`WRITE_ID`
[
https://issues.apache.org/jira/browse/HIVE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-20298:
Status: Patch Available (was: Open)
[~jdere] can you take a look?
> Illegal null value in column `TBLS`.`WRITE_ID`
> --
>
> Key: HIVE-20298
> URL: https://issues.apache.org/jira/browse/HIVE-20298
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20298.1.patch, HIVE-20298.patch
>
>
> Manually upgraded my existing local metastore using
> upgrade-3.0.0-to-3.1.0.mysql.sql, upgrade-3.1.0-to-3.2.0.mysql.sql,
> upgrade-3.2.0-to-4.0.0.mysql.sql. When running DESCRIBE EXTENDED of an
> existing table, I was getting the following error in hive.log. It looks like
> the ObjectStore/MTable classes don't seem to be able to support null values
> in the new writeId column that was added to the TBLS table in the metastore.
> cc [~sershe] [~ekoifman]
> {noformat}
> Caused by: javax.jdo.JDODataStoreException: Illegal null value in column
> `TBLS`.`WRITE_ID`
> NestedThrowables:
> org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal null value
> in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:553)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1802)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1838)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1424)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy39.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2950)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:2898)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:2882)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ... 36 more
> Caused by: org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal
> null value in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.store.rdbms.mapping.datastore.BigIntRDBMSMapping.getLong(BigIntRDBMSMapping.java:140)
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getLong(SingleFieldMapping.java:155)
> at
> org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchLongField(ResultSetGetter.java:124)
> at
> org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:1549)
> at
> org.datanucleus.state.StateManagerImpl.replacingLongField(StateManagerImpl.java:120)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceField(MTable.java)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceFields(MTable.java)
> at
> org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
> at
> org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
> at
> org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
> at
> org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
> at
> org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
>
[jira] [Commented] (HIVE-19902) Provide Metastore micro-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-19902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579084#comment-16579084 ] Alexander Kolbasov commented on HIVE-19902: --- [~owen.omalley] I moved everything under standalone metastore. I will continue investigating JMH - would it be reasonable to do this as a follow-up work? > Provide Metastore micro-benchmarks > -- > > Key: HIVE-19902 > URL: https://issues.apache.org/jira/browse/HIVE-19902 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-19902.01.patch, HIVE-19902.02.patch, > HIVE-19902.03.patch, HIVE-19902.04.patch, HIVE-19902.05.patch, > HIVE-19902.06.patch > > > It would be very useful to have metastore benchmarks to be able to track perf > issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20379) Rewriting with partitioned materialized views may reference wrong column
[ https://issues.apache.org/jira/browse/HIVE-20379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579076#comment-16579076 ] Jesus Camacho Rodriguez commented on HIVE-20379: [~ashutoshc], could you take a look? https://reviews.apache.org/r/68337/ Thanks > Rewriting with partitioned materialized views may reference wrong column > > > Key: HIVE-20379 > URL: https://issues.apache.org/jira/browse/HIVE-20379 > Project: Hive > Issue Type: Bug > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20379.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20379) Rewriting with partitioned materialized views may reference wrong column
[ https://issues.apache.org/jira/browse/HIVE-20379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20379: --- Attachment: HIVE-20379.patch > Rewriting with partitioned materialized views may reference wrong column > > > Key: HIVE-20379 > URL: https://issues.apache.org/jira/browse/HIVE-20379 > Project: Hive > Issue Type: Bug > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20379.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-20377:
--
Attachment: HIVE-20377.5.patch
> Hive Kafka Storage Handler
> --
>
> Key: HIVE-20377
> URL: https://issues.apache.org/jira/browse/HIVE-20377
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20377.4.patch, HIVE-20377.5.patch, HIVE-20377.patch
>
>
> h1. Goal
> * Read streaming data form Kafka queue as an external table.
> * Allow streaming navigation by pushing down filters on Kafka record
> partition id, offset and timestamp.
> * Insert streaming data form Kafka to an actual Hive internal table, using
> CTAS statement.
> h1. Example
> h2. Create the external table
> {code}
> CREATE EXTERNAL TABLE kafka_table (`timestamp` timestamp, page string, `user`
> string, language string, added int, deleted int, flags string,comment string,
> namespace string)
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wikipedia",
> "kafka.bootstrap.servers"="brokeraddress:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe");
> {code}
> h2. Kafka Metadata
> In order to keep track of Kafka records the storage handler will add
> automatically the Kafka row metadata eg partition id, record offset and
> record timestamp.
> {code}
> DESCRIBE EXTENDED kafka_table
> timestamp timestamp from deserializer
> page string from deserializer
> user string from deserializer
> language string from deserializer
> country string from deserializer
> continent string from deserializer
> namespace string from deserializer
> newpage boolean from deserializer
> unpatrolled boolean from deserializer
> anonymous boolean from deserializer
> robot boolean from deserializer
> added int from deserializer
> deleted int from deserializer
> delta bigint from deserializer
> __partition int from deserializer
> __offset bigint from deserializer
> __timestamp bigint from deserializer
> {code}
> h2. Filter push down.
> Newer Kafka consumers 0.11.0 and higher allow seeking on the stream based on
> a given offset. The proposed storage handler will be able to leverage such
> API by pushing down filters over metadata columns, namely __partition (int),
> __offset(long) and __timestamp(long)
> For instance Query like
> {code}
> select `__offset` from kafka_table where (`__offset` < 10 and `__offset`>3
> and `__partition` = 0) or (`__partition` = 0 and `__offset` < 105 and
> `__offset` > 99) or (`__offset` = 109);
> {code}
> Will result on a scan of partition 0 only then read only records between
> offset 4 and 109.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20379) Rewriting with partitioned materialized views may reference wrong column
[ https://issues.apache.org/jira/browse/HIVE-20379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20379: --- Status: Patch Available (was: In Progress) > Rewriting with partitioned materialized views may reference wrong column > > > Key: HIVE-20379 > URL: https://issues.apache.org/jira/browse/HIVE-20379 > Project: Hive > Issue Type: Bug > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-20379) Rewriting with partitioned materialized views may reference wrong column
[ https://issues.apache.org/jira/browse/HIVE-20379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-20379 started by Jesus Camacho Rodriguez. -- > Rewriting with partitioned materialized views may reference wrong column > > > Key: HIVE-20379 > URL: https://issues.apache.org/jira/browse/HIVE-20379 > Project: Hive > Issue Type: Bug > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20298) Illegal null value in column `TBLS`.`WRITE_ID`
[
https://issues.apache.org/jira/browse/HIVE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-20298:
---
Assignee: Sergey Shelukhin
> Illegal null value in column `TBLS`.`WRITE_ID`
> --
>
> Key: HIVE-20298
> URL: https://issues.apache.org/jira/browse/HIVE-20298
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20298.1.patch
>
>
> Manually upgraded my existing local metastore using
> upgrade-3.0.0-to-3.1.0.mysql.sql, upgrade-3.1.0-to-3.2.0.mysql.sql,
> upgrade-3.2.0-to-4.0.0.mysql.sql. When running DESCRIBE EXTENDED of an
> existing table, I was getting the following error in hive.log. It looks like
> the ObjectStore/MTable classes don't seem to be able to support null values
> in the new writeId column that was added to the TBLS table in the metastore.
> cc [~sershe] [~ekoifman]
> {noformat}
> Caused by: javax.jdo.JDODataStoreException: Illegal null value in column
> `TBLS`.`WRITE_ID`
> NestedThrowables:
> org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal null value
> in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:553)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1802)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1838)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1424)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy39.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2950)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:2898)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:2882)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ... 36 more
> Caused by: org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal
> null value in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.store.rdbms.mapping.datastore.BigIntRDBMSMapping.getLong(BigIntRDBMSMapping.java:140)
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getLong(SingleFieldMapping.java:155)
> at
> org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchLongField(ResultSetGetter.java:124)
> at
> org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:1549)
> at
> org.datanucleus.state.StateManagerImpl.replacingLongField(StateManagerImpl.java:120)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceField(MTable.java)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceFields(MTable.java)
> at
> org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
> at
> org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
> at
> org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
> at
> org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
> at
> org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
> at
>
[jira] [Commented] (HIVE-20378) don't update stats during alter for txn table conversion
[
https://issues.apache.org/jira/browse/HIVE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579063#comment-16579063
]
Hive QA commented on HIVE-20378:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13199/dev-support/hive-personality.sh
|
| git revision | master / 681051f |
| Default Java | 1.8.0_111 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13199/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13199/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13199/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> don't update stats during alter for txn table conversion
>
>
> Key: HIVE-20378
> URL: https://issues.apache.org/jira/browse/HIVE-20378
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20378.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579052#comment-16579052
]
Hive QA commented on HIVE-20377:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935440/HIVE-20377.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13198/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13198/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13198/
Messages:
{noformat}
This message was trimmed, see log for full details
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-auth/3.1.0/hadoop-auth-3.1.0.jar(org/apache/hadoop/security/authentication/client/PseudoAuthenticator.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-auth/3.1.0/hadoop-auth-3.1.0.jar(org/apache/hadoop/security/authentication/server/PseudoAuthenticationHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/security/SecurityUtil.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/util/GenericOptionsParser.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-rewrite/9.3.20.v20170531/jetty-rewrite-9.3.20.v20170531.jar(org/eclipse/jetty/rewrite/handler/RedirectPatternRule.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-rewrite/9.3.20.v20170531/jetty-rewrite-9.3.20.v20170531.jar(org/eclipse/jetty/rewrite/handler/RewriteHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.20.v20170531/jetty-server-9.3.20.v20170531.jar(org/eclipse/jetty/server/Handler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.20.v20170531/jetty-server-9.3.20.v20170531.jar(org/eclipse/jetty/server/Server.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.20.v20170531/jetty-server-9.3.20.v20170531.jar(org/eclipse/jetty/server/ServerConnector.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.20.v20170531/jetty-server-9.3.20.v20170531.jar(org/eclipse/jetty/server/handler/HandlerList.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.20.v20170531/jetty-servlet-9.3.20.v20170531.jar(org/eclipse/jetty/servlet/FilterHolder.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.20.v20170531/jetty-servlet-9.3.20.v20170531.jar(org/eclipse/jetty/servlet/ServletContextHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.20.v20170531/jetty-servlet-9.3.20.v20170531.jar(org/eclipse/jetty/servlet/ServletHolder.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-xml/9.3.20.v20170531/jetty-xml-9.3.20.v20170531.jar(org/eclipse/jetty/xml/XmlConfiguration.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/slf4j/jul-to-slf4j/1.7.10/jul-to-slf4j-1.7.10.jar(org/slf4j/bridge/SLF4JBridgeHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/DispatcherType.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/Filter.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/FilterChain.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/FilterConfig.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/ServletException.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/ServletRequest.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/ServletResponse.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/annotation/WebFilter.class)]]
[loading
[jira] [Commented] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579050#comment-16579050
]
Hive QA commented on HIVE-20246:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935421/HIVE-20246.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14882 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=194)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13197/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13197/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13197/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935421 - PreCommit-HIVE-Build
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.5.patch
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
> key2=val2);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19316) StatsTask fails due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-19316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M updated HIVE-19316:
---
Attachment: HIVE-19316.5.patch
Status: Patch Available (was: Open)
> StatsTask fails due to ClassCastException
> -
>
> Key: HIVE-19316
> URL: https://issues.apache.org/jira/browse/HIVE-19316
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Rui Li
>Assignee: Jaume M
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19316.1.patch, HIVE-19316.2.patch,
> HIVE-19316.3.patch, HIVE-19316.4.patch, HIVE-19316.5.patch
>
>
> The stack trace:
> {noformat}
> 2018-04-26T20:17:37,674 ERROR [pool-7-thread-11]
> metastore.RetryingHMSHandler: java.lang.ClassCastException:
> org.apache.hadoop.hive.metastore.api.LongColumnStatsData cannot be cast to
> org.apache.hadoop.hive.metastore.columnstats.cache.LongColumnStatsDataInspector
> at
> org.apache.hadoop.hive.metastore.columnstats.merge.LongColumnStatsMerger.merge(LongColumnStatsMerger.java:30)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:1052)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy26.set_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:16795)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:16779)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19316) StatsTask fails due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-19316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M updated HIVE-19316:
---
Status: Open (was: Patch Available)
> StatsTask fails due to ClassCastException
> -
>
> Key: HIVE-19316
> URL: https://issues.apache.org/jira/browse/HIVE-19316
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Rui Li
>Assignee: Jaume M
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19316.1.patch, HIVE-19316.2.patch,
> HIVE-19316.3.patch, HIVE-19316.4.patch, HIVE-19316.5.patch
>
>
> The stack trace:
> {noformat}
> 2018-04-26T20:17:37,674 ERROR [pool-7-thread-11]
> metastore.RetryingHMSHandler: java.lang.ClassCastException:
> org.apache.hadoop.hive.metastore.api.LongColumnStatsData cannot be cast to
> org.apache.hadoop.hive.metastore.columnstats.cache.LongColumnStatsDataInspector
> at
> org.apache.hadoop.hive.metastore.columnstats.merge.LongColumnStatsMerger.merge(LongColumnStatsMerger.java:30)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:1052)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy26.set_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:16795)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:16779)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579023#comment-16579023
]
Hive QA commented on HIVE-20246:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13197/dev-support/hive-personality.sh
|
| git revision | master / 681051f |
| Default Java | 1.8.0_111 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13197/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13197/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13197/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.5.patch
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
>
[jira] [Commented] (HIVE-19316) StatsTask fails due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-19316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579010#comment-16579010
]
Hive QA commented on HIVE-19316:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935210/HIVE-19316.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14877 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=194)
org.apache.hive.spark.client.rpc.TestRpc.testClientTimeout (batchId=318)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13196/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13196/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13196/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935210 - PreCommit-HIVE-Build
> StatsTask fails due to ClassCastException
> -
>
> Key: HIVE-19316
> URL: https://issues.apache.org/jira/browse/HIVE-19316
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Rui Li
>Assignee: Jaume M
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19316.1.patch, HIVE-19316.2.patch,
> HIVE-19316.3.patch, HIVE-19316.4.patch
>
>
> The stack trace:
> {noformat}
> 2018-04-26T20:17:37,674 ERROR [pool-7-thread-11]
> metastore.RetryingHMSHandler: java.lang.ClassCastException:
> org.apache.hadoop.hive.metastore.api.LongColumnStatsData cannot be cast to
> org.apache.hadoop.hive.metastore.columnstats.cache.LongColumnStatsDataInspector
> at
> org.apache.hadoop.hive.metastore.columnstats.merge.LongColumnStatsMerger.merge(LongColumnStatsMerger.java:30)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:1052)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy26.set_aggr_stats_for(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:16795)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:16779)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-20377:
--
Attachment: HIVE-20377.4.patch
> Hive Kafka Storage Handler
> --
>
> Key: HIVE-20377
> URL: https://issues.apache.org/jira/browse/HIVE-20377
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20377.4.patch, HIVE-20377.patch
>
>
> h1. Goal
> * Read streaming data form Kafka queue as an external table.
> * Allow streaming navigation by pushing down filters on Kafka record
> partition id, offset and timestamp.
> * Insert streaming data form Kafka to an actual Hive internal table, using
> CTAS statement.
> h1. Example
> h2. Create the external table
> {code}
> CREATE EXTERNAL TABLE kafka_table (`timestamp` timestamp, page string, `user`
> string, language string, added int, deleted int, flags string,comment string,
> namespace string)
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wikipedia",
> "kafka.bootstrap.servers"="brokeraddress:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe");
> {code}
> h2. Kafka Metadata
> In order to keep track of Kafka records the storage handler will add
> automatically the Kafka row metadata eg partition id, record offset and
> record timestamp.
> {code}
> DESCRIBE EXTENDED kafka_table
> timestamp timestamp from deserializer
> page string from deserializer
> user string from deserializer
> language string from deserializer
> country string from deserializer
> continent string from deserializer
> namespace string from deserializer
> newpage boolean from deserializer
> unpatrolled boolean from deserializer
> anonymous boolean from deserializer
> robot boolean from deserializer
> added int from deserializer
> deleted int from deserializer
> delta bigint from deserializer
> __partition int from deserializer
> __offset bigint from deserializer
> __timestamp bigint from deserializer
> {code}
> h2. Filter push down.
> Newer Kafka consumers 0.11.0 and higher allow seeking on the stream based on
> a given offset. The proposed storage handler will be able to leverage such
> API by pushing down filters over metadata columns, namely __partition (int),
> __offset(long) and __timestamp(long)
> For instance Query like
> {code}
> select `__offset` from kafka_table where (`__offset` < 10 and `__offset`>3
> and `__partition` = 0) or (`__partition` = 0 and `__offset` < 105 and
> `__offset` > 99) or (`__offset` = 109);
> {code}
> Will result on a scan of partition 0 only then read only records between
> offset 4 and 109.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20379) Rewriting with partitioned materialized views may reference wrong column
[ https://issues.apache.org/jira/browse/HIVE-20379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-20379: -- > Rewriting with partitioned materialized views may reference wrong column > > > Key: HIVE-20379 > URL: https://issues.apache.org/jira/browse/HIVE-20379 > Project: Hive > Issue Type: Bug > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19316) StatsTask fails due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-19316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578965#comment-16578965
]
Hive QA commented on HIVE-19316:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13196/dev-support/hive-personality.sh
|
| git revision | master / 681051f |
| Default Java | 1.8.0_111 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13196/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13196/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13196/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> StatsTask fails due to ClassCastException
> -
>
> Key: HIVE-19316
> URL: https://issues.apache.org/jira/browse/HIVE-19316
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Rui Li
>Assignee: Jaume M
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19316.1.patch, HIVE-19316.2.patch,
> HIVE-19316.3.patch, HIVE-19316.4.patch
>
>
> The stack trace:
> {noformat}
> 2018-04-26T20:17:37,674 ERROR [pool-7-thread-11]
> metastore.RetryingHMSHandler: java.lang.ClassCastException:
> org.apache.hadoop.hive.metastore.api.LongColumnStatsData cannot be cast to
> org.apache.hadoop.hive.metastore.columnstats.cache.LongColumnStatsDataInspector
> at
> org.apache.hadoop.hive.metastore.columnstats.merge.LongColumnStatsMerger.merge(LongColumnStatsMerger.java:30)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:1052)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
[jira] [Commented] (HIVE-20371) Queries failing with Internal error processing add_write_notification_log
[
https://issues.apache.org/jira/browse/HIVE-20371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578942#comment-16578942
]
Hive QA commented on HIVE-20371:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935396/HIVE-20371.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14877 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=194)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13195/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13195/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13195/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935396 - PreCommit-HIVE-Build
> Queries failing with Internal error processing add_write_notification_log
> -
>
> Key: HIVE-20371
> URL: https://issues.apache.org/jira/browse/HIVE-20371
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl, Standalone Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-20371.01.patch
>
>
> Queries failing with following error:
> {noformat}
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.thrift.TApplicationException: Internal error processing
> add_write_notification_log
> INFO : Completed executing
> command(queryId=hive_20180806072916_a9ae37a9-869f-4218-8357-a96ba713db69);
> Time taken: 878.604 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.thrift.TApplicationException: Internal error processing
> add_write_notification_log (state=08S01,code=1)
> {noformat}
> From hiveserver log:
> {noformat}
> 2018-08-06T07:59:33,656 ERROR [HiveServer2-Background-Pool: Thread-1551]:
> operation.Operation (:()) - Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.thrift.TApplicationException: Internal error processing
> add_write_notification_log
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
> ~[hive-service-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226)
> ~[hive-service-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
> ~[hive-service-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)
> ~[hive-service-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> ~[hadoop-common-3.1.0.3.0.1.0-59.jar:?]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:329)
> ~[hive-service-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_112]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_112]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: Internal error processing
> add_write_notification_log
> at
>
[jira] [Commented] (HIVE-20371) Queries failing with Internal error processing add_write_notification_log
[
https://issues.apache.org/jira/browse/HIVE-20371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578929#comment-16578929
]
Hive QA commented on HIVE-20371:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
27s{color} | {color:blue} standalone-metastore/metastore-common in master has 9
extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
14s{color} | {color:blue} ql in master has 2306 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} itests/hive-unit: The patch generated 16 new + 65
unchanged - 1 fixed = 81 total (was 66) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13195/dev-support/hive-personality.sh
|
| git revision | master / 681051f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13195/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13195/yetus/diff-checkstyle-itests_hive-unit.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13195/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-common itests/hive-unit ql
standalone-metastore/metastore-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13195/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Queries failing with Internal error processing add_write_notification_log
> -
>
> Key: HIVE-20371
> URL: https://issues.apache.org/jira/browse/HIVE-20371
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl,
[jira] [Commented] (HIVE-18620) Improve error message while dropping a table that is part of a materialized view
[ https://issues.apache.org/jira/browse/HIVE-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578920#comment-16578920 ] Ashutosh Chauhan commented on HIVE-18620: - +1 > Improve error message while dropping a table that is part of a materialized > view > > > Key: HIVE-18620 > URL: https://issues.apache.org/jira/browse/HIVE-18620 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-18620.patch > > > When we want to drop a table used by a materialized view, we prevent dropping > that table. However, the message shown is not very meaningful (FK-PK > violation). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-14500) Support masking of columns for materialized views
[ https://issues.apache.org/jira/browse/HIVE-14500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-14500. Resolution: Duplicate > Support masking of columns for materialized views > - > > Key: HIVE-14500 > URL: https://issues.apache.org/jira/browse/HIVE-14500 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Priority: Major > > Verify that column masking is working for materialized views and provided > necessary extensions. Add test cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-14484) Extensions for initial materialized views implementation
[ https://issues.apache.org/jira/browse/HIVE-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-14484. Resolution: Fixed Fix Version/s: 3.2.0 > Extensions for initial materialized views implementation > > > Key: HIVE-14484 > URL: https://issues.apache.org/jira/browse/HIVE-14484 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 3.2.0 > > > Follow-up of HIVE-14249. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14484) Extensions for initial materialized views implementation
[ https://issues.apache.org/jira/browse/HIVE-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578909#comment-16578909 ] Jesus Camacho Rodriguez commented on HIVE-14484: Closing as resolved, all subtasks have been implemented. More information can be found in: https://cwiki.apache.org/confluence/display/Hive/Materialized+views (still being updated). All the rest of materialized views related issues are tagged with 'Materialized views' component. > Extensions for initial materialized views implementation > > > Key: HIVE-14484 > URL: https://issues.apache.org/jira/browse/HIVE-14484 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 3.2.0 > > > Follow-up of HIVE-14249. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18842) CLUSTER BY/DISTRIBUTE BY/SORT BY support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-18842:
---
Issue Type: New Feature (was: Sub-task)
Parent: (was: HIVE-14484)
> CLUSTER BY/DISTRIBUTE BY/SORT BY support for materialized views
> ---
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> We should support defining a {{CLUSTER BY/DISTRIBUTE BY/SORT BY}}
> specification for materialized views.
> The syntax should be extended as follows:
> {code:sql}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
> [COMMENT materialized_view_comment]
> [CLUSTER BY (col_name, ...) | ( [DISTRIBUTE BY (col_name, ...)] [SORT BY
> (col_name, ...)] ) ] -- NEW!
> [
>[ROW FORMAT row_format]
>[STORED AS file_format]
> | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> ]
> [LOCATION hdfs_path]
> [TBLPROPERTIES (property_name=property_value, ...)]
> AS select_statement;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19407) Only support materialized views stored either as ACID or in selected custom storage handlers
[ https://issues.apache.org/jira/browse/HIVE-19407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-19407: --- Issue Type: Improvement (was: Sub-task) Parent: (was: HIVE-14484) > Only support materialized views stored either as ACID or in selected custom > storage handlers > > > Key: HIVE-19407 > URL: https://issues.apache.org/jira/browse/HIVE-19407 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > We should not support, e.g., external HDFS tables. Storage handlers such as > Druid should be fine. We will limit the support for sources that are actually > not handled by Hive, which will in turn produce more predictable behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18620) Improve error message while dropping a table that is part of a materialized view
[ https://issues.apache.org/jira/browse/HIVE-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18620: --- Attachment: HIVE-18620.patch > Improve error message while dropping a table that is part of a materialized > view > > > Key: HIVE-18620 > URL: https://issues.apache.org/jira/browse/HIVE-18620 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-18620.patch > > > When we want to drop a table used by a materialized view, we prevent dropping > that table. However, the message shown is not very meaningful (FK-PK > violation). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18620) Improve error message while dropping a table that is part of a materialized view
[ https://issues.apache.org/jira/browse/HIVE-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578896#comment-16578896 ] Jesus Camacho Rodriguez commented on HIVE-18620: [~ashutoshc], can you take a look? Thanks > Improve error message while dropping a table that is part of a materialized > view > > > Key: HIVE-18620 > URL: https://issues.apache.org/jira/browse/HIVE-18620 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-18620.patch > > > When we want to drop a table used by a materialized view, we prevent dropping > that table. However, the message shown is not very meaningful (FK-PK > violation). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20378) don't update stats during alter for txn table conversion
[ https://issues.apache.org/jira/browse/HIVE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578893#comment-16578893 ] Sergey Shelukhin commented on HIVE-20378: - [~ekoifman] can you take a look? > don't update stats during alter for txn table conversion > > > Key: HIVE-20378 > URL: https://issues.apache.org/jira/browse/HIVE-20378 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20378.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20378) don't update stats during alter for txn table conversion
[ https://issues.apache.org/jira/browse/HIVE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20378: Assignee: Sergey Shelukhin Status: Patch Available (was: Open) > don't update stats during alter for txn table conversion > > > Key: HIVE-20378 > URL: https://issues.apache.org/jira/browse/HIVE-20378 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20378.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18620) Improve error message while dropping a table that is part of a materialized view
[ https://issues.apache.org/jira/browse/HIVE-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18620: --- Status: Patch Available (was: In Progress) > Improve error message while dropping a table that is part of a materialized > view > > > Key: HIVE-18620 > URL: https://issues.apache.org/jira/browse/HIVE-18620 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > > When we want to drop a table used by a materialized view, we prevent dropping > that table. However, the message shown is not very meaningful (FK-PK > violation). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-18620) Improve error message while dropping a table that is part of a materialized view
[ https://issues.apache.org/jira/browse/HIVE-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-18620 started by Jesus Camacho Rodriguez. -- > Improve error message while dropping a table that is part of a materialized > view > > > Key: HIVE-18620 > URL: https://issues.apache.org/jira/browse/HIVE-18620 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > > When we want to drop a table used by a materialized view, we prevent dropping > that table. However, the message shown is not very meaningful (FK-PK > violation). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18620) Improve error message while dropping a table that is part of a materialized view
[ https://issues.apache.org/jira/browse/HIVE-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-18620: -- Assignee: Jesus Camacho Rodriguez (was: Miklos Gergely) > Improve error message while dropping a table that is part of a materialized > view > > > Key: HIVE-18620 > URL: https://issues.apache.org/jira/browse/HIVE-18620 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > > When we want to drop a table used by a materialized view, we prevent dropping > that table. However, the message shown is not very meaningful (FK-PK > violation). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20378) don't update stats during alter for txn table conversion
[ https://issues.apache.org/jira/browse/HIVE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20378: Attachment: HIVE-20378.patch > don't update stats during alter for txn table conversion > > > Key: HIVE-20378 > URL: https://issues.apache.org/jira/browse/HIVE-20378 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20378.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18620) Update materialized views when dropping its source tables
[ https://issues.apache.org/jira/browse/HIVE-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18620: --- Description: When we want to drop a table used by a materialized view, we prevent dropping that table. However, the message shown is not very meaningful (FK-PK violation). (was: When we want to drop a table used by a materialized view, we should prevent dropping that table unless explicitly specified that materialized views based on that table can be dropped too.) > Update materialized views when dropping its source tables > - > > Key: HIVE-18620 > URL: https://issues.apache.org/jira/browse/HIVE-18620 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Miklos Gergely >Priority: Critical > > When we want to drop a table used by a materialized view, we prevent dropping > that table. However, the message shown is not very meaningful (FK-PK > violation). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18620) Improve error message while dropping a table that is part of a materialized view
[ https://issues.apache.org/jira/browse/HIVE-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18620: --- Summary: Improve error message while dropping a table that is part of a materialized view (was: Update materialized views when dropping its source tables) > Improve error message while dropping a table that is part of a materialized > view > > > Key: HIVE-18620 > URL: https://issues.apache.org/jira/browse/HIVE-18620 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Miklos Gergely >Priority: Critical > > When we want to drop a table used by a materialized view, we prevent dropping > that table. However, the message shown is not very meaningful (FK-PK > violation). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19767) HiveServer2 should take hiveconf for non Hive properties
[
https://issues.apache.org/jira/browse/HIVE-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578880#comment-16578880
]
Hive QA commented on HIVE-19767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935395/HIVE-19767.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14877 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13194/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13194/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13194/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935395 - PreCommit-HIVE-Build
> HiveServer2 should take hiveconf for non Hive properties
>
>
> Key: HIVE-19767
> URL: https://issues.apache.org/jira/browse/HIVE-19767
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 1.2.2, 3.0.0, 2.3.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Attachments: HIVE-19767.2.patch, HIVE-19767.3.patch,
> HIVE-19767.4.patch, HIVE-19767.5.patch, HIVE-19767.patch
>
>
> The -hiveconf command line option works in HiveServer2 with properties in
> HiveConf.java, but not so well with other properties (like mapred properties
> or spark properties to control underlying execution engine, or custom
> properties understood by custom listeners)
> It is inconsistent with HiveCLI.
> HiveCLI behavior:
> {noformat}
> ./bin/hive --hiveconf a=b
> hive> set a;
> a=b {noformat}
> HiveServer2 behavior:
> {noformat}
> ./bin/hiveserver2 --hiveconf a=b
> beeline> set a;
> +-+
> | set |
> +-+
> | a is undefined |
> +-+{noformat}
> Although it is possible to set up hive-site.xml or even mapred-site.xml to
> fill in the relevant properties, it is more convenient when testing HS2 with
> different configuration to be able to use --hiveconf to change on the fly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16116) Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
[
https://issues.apache.org/jira/browse/HIVE-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578878#comment-16578878
]
Shawn Weeks commented on HIVE-16116:
In case anyone else runs across this. This error will also occur if this line
is set in beeline.properties "beeline.hiveconfvariables=" it doesn't have to
have the brackets and it might NPE on line 679. I've been dealing with an issue
in Oozie due to this for a week.
> Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
> --
>
> Key: HIVE-16116
> URL: https://issues.apache.org/jira/browse/HIVE-16116
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16116.1.patch, HIVE-16116.2.patch
>
>
> Env: hive master
> Steps to reproduce:
> 1. clear previous beeline.properties (rm -rf ~/.beeline/beeline.properties)
> 2. Launch beeline, "!save" and exit. This would create new
> "~/.beeline/beeline.properties", which would have
> "beeline.hiveconfvariables={}"
> 3. Launch "beeline --hiveconf hive.tmp.dir=/tmp". This would throw NPE
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.setHiveConfVar(BeeLine.java:885)
> at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:832)
> at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:775)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1009)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-20377:
--
Attachment: HIVE-20377.patch
> Hive Kafka Storage Handler
> --
>
> Key: HIVE-20377
> URL: https://issues.apache.org/jira/browse/HIVE-20377
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20377.patch
>
>
> h1. Goal
> * Read streaming data form Kafka queue as an external table.
> * Allow streaming navigation by pushing down filters on Kafka record
> partition id, offset and timestamp.
> * Insert streaming data form Kafka to an actual Hive internal table, using
> CTAS statement.
> h1. Example
> h2. Create the external table
> {code}
> CREATE EXTERNAL TABLE kafka_table (`timestamp` timestamp, page string, `user`
> string, language string, added int, deleted int, flags string,comment string,
> namespace string)
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wikipedia",
> "kafka.bootstrap.servers"="brokeraddress:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe");
> {code}
> h2. Kafka Metadata
> In order to keep track of Kafka records the storage handler will add
> automatically the Kafka row metadata eg partition id, record offset and
> record timestamp.
> {code}
> DESCRIBE EXTENDED kafka_table
> timestamp timestamp from deserializer
> page string from deserializer
> user string from deserializer
> language string from deserializer
> country string from deserializer
> continent string from deserializer
> namespace string from deserializer
> newpage boolean from deserializer
> unpatrolled boolean from deserializer
> anonymous boolean from deserializer
> robot boolean from deserializer
> added int from deserializer
> deleted int from deserializer
> delta bigint from deserializer
> __partition int from deserializer
> __offset bigint from deserializer
> __timestamp bigint from deserializer
> {code}
> h2. Filter push down.
> Newer Kafka consumers 0.11.0 and higher allow seeking on the stream based on
> a given offset. The proposed storage handler will be able to leverage such
> API by pushing down filters over metadata columns, namely __partition (int),
> __offset(long) and __timestamp(long)
> For instance Query like
> {code}
> select `__offset` from kafka_table where (`__offset` < 10 and `__offset`>3
> and `__partition` = 0) or (`__partition` = 0 and `__offset` < 105 and
> `__offset` > 99) or (`__offset` = 109);
> {code}
> Will result on a scan of partition 0 only then read only records between
> offset 4 and 109.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-20377:
--
Status: Patch Available (was: Open)
> Hive Kafka Storage Handler
> --
>
> Key: HIVE-20377
> URL: https://issues.apache.org/jira/browse/HIVE-20377
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20377.patch
>
>
> h1. Goal
> * Read streaming data form Kafka queue as an external table.
> * Allow streaming navigation by pushing down filters on Kafka record
> partition id, offset and timestamp.
> * Insert streaming data form Kafka to an actual Hive internal table, using
> CTAS statement.
> h1. Example
> h2. Create the external table
> {code}
> CREATE EXTERNAL TABLE kafka_table (`timestamp` timestamp, page string, `user`
> string, language string, added int, deleted int, flags string,comment string,
> namespace string)
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wikipedia",
> "kafka.bootstrap.servers"="brokeraddress:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe");
> {code}
> h2. Kafka Metadata
> In order to keep track of Kafka records the storage handler will add
> automatically the Kafka row metadata eg partition id, record offset and
> record timestamp.
> {code}
> DESCRIBE EXTENDED kafka_table
> timestamp timestamp from deserializer
> page string from deserializer
> user string from deserializer
> language string from deserializer
> country string from deserializer
> continent string from deserializer
> namespace string from deserializer
> newpage boolean from deserializer
> unpatrolled boolean from deserializer
> anonymous boolean from deserializer
> robot boolean from deserializer
> added int from deserializer
> deleted int from deserializer
> delta bigint from deserializer
> __partition int from deserializer
> __offset bigint from deserializer
> __timestamp bigint from deserializer
> {code}
> h2. Filter push down.
> Newer Kafka consumers 0.11.0 and higher allow seeking on the stream based on
> a given offset. The proposed storage handler will be able to leverage such
> API by pushing down filters over metadata columns, namely __partition (int),
> __offset(long) and __timestamp(long)
> For instance Query like
> {code}
> select `__offset` from kafka_table where (`__offset` < 10 and `__offset`>3
> and `__partition` = 0) or (`__partition` = 0 and `__offset` < 105 and
> `__offset` > 99) or (`__offset` = 109);
> {code}
> Will result on a scan of partition 0 only then read only records between
> offset 4 and 109.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19767) HiveServer2 should take hiveconf for non Hive properties
[
https://issues.apache.org/jira/browse/HIVE-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578843#comment-16578843
]
Hive QA commented on HIVE-19767:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
24s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
23s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 23s{color}
| {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} common: The patch generated 1 new + 424 unchanged - 0
fixed = 425 total (was 424) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
25s{color} | {color:red} service in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13194/dev-support/hive-personality.sh
|
| git revision | master / 681051f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13194/yetus/patch-mvninstall-service.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13194/yetus/patch-compile-service.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13194/yetus/patch-compile-service.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13194/yetus/diff-checkstyle-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13194/yetus/patch-findbugs-service.txt
|
| modules | C: common service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13194/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveServer2 should take hiveconf for non Hive properties
>
>
> Key: HIVE-19767
> URL: https://issues.apache.org/jira/browse/HIVE-19767
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 1.2.2, 3.0.0, 2.3.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Attachments: HIVE-19767.2.patch, HIVE-19767.3.patch,
> HIVE-19767.4.patch, HIVE-19767.5.patch, HIVE-19767.patch
>
>
> The -hiveconf command line option works in HiveServer2 with properties in
> HiveConf.java, but not
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[
https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578841#comment-16578841
]
Gopal V commented on HIVE-20321:
And query51 has "group by ss_item_sk, d_date"
> Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1
> CacheLine
> -
>
> Key: HIVE-20321
> URL: https://issues.apache.org/jira/browse/HIVE-20321
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Priority: Major
>
> With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper
> is bigger than the low Xmx JVMs.
> {code}
> * 64-bit VM: **
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals:
> OFFSET SIZE
> TYPE DESCRIPTION VALUE
> 016
> (object header) N/A
> 16 4
> int VectorHashKeyWrapper.hashcodeN/A
> 20 4
> (alignment/padding gap)
> 24 8
> long[] VectorHashKeyWrapper.longValues N/A
> 32 8
> double[] VectorHashKeyWrapper.doubleValuesN/A
> 40 8
> byte[][] VectorHashKeyWrapper.byteValues N/A
> 48 8
>int[] VectorHashKeyWrapper.byteStarts N/A
> 56 8
>int[] VectorHashKeyWrapper.byteLengths N/A
> 64 8
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[]
> VectorHashKeyWrapper.decimalValues N/A
> 72 8
> java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A
> 80 8
> org.apache.hadoop.hive.common.type.HiveIntervalDayTime[]
> VectorHashKeyWrapper.intervalDayTimeValues N/A
> 88 8
> boolean[] VectorHashKeyWrapper.isNull N/A
> 96 8
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext
> VectorHashKeyWrapper.hashCtx N/A
> Instance size: 104 bytes
> Space losses: 4 bytes internal + 0 bytes external = 4 bytes total
> {code}
> Pulling this up to a parent class allows for this to be cut down to 32 bytes
> for the single column case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[
https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578834#comment-16578834
]
Eric Wohlstadter commented on HIVE-20321:
-
"specific case was query23 and query65"
"query23 has group by ss_customer_sk across store_sales"
"query65 has group by ss_store_sk, ss_item_sk"
> Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1
> CacheLine
> -
>
> Key: HIVE-20321
> URL: https://issues.apache.org/jira/browse/HIVE-20321
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Priority: Major
>
> With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper
> is bigger than the low Xmx JVMs.
> {code}
> * 64-bit VM: **
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals:
> OFFSET SIZE
> TYPE DESCRIPTION VALUE
> 016
> (object header) N/A
> 16 4
> int VectorHashKeyWrapper.hashcodeN/A
> 20 4
> (alignment/padding gap)
> 24 8
> long[] VectorHashKeyWrapper.longValues N/A
> 32 8
> double[] VectorHashKeyWrapper.doubleValuesN/A
> 40 8
> byte[][] VectorHashKeyWrapper.byteValues N/A
> 48 8
>int[] VectorHashKeyWrapper.byteStarts N/A
> 56 8
>int[] VectorHashKeyWrapper.byteLengths N/A
> 64 8
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[]
> VectorHashKeyWrapper.decimalValues N/A
> 72 8
> java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A
> 80 8
> org.apache.hadoop.hive.common.type.HiveIntervalDayTime[]
> VectorHashKeyWrapper.intervalDayTimeValues N/A
> 88 8
> boolean[] VectorHashKeyWrapper.isNull N/A
> 96 8
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext
> VectorHashKeyWrapper.hashCtx N/A
> Instance size: 104 bytes
> Space losses: 4 bytes internal + 0 bytes external = 4 bytes total
> {code}
> Pulling this up to a parent class allows for this to be cut down to 32 bytes
> for the single column case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20033) Backport HIVE-19432 to branch-2, branch-3
[
https://issues.apache.org/jira/browse/HIVE-20033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578824#comment-16578824
]
Hive QA commented on HIVE-20033:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935392/HIVE-20033.02-branch-3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 103 failed/errored test(s), 14426 tests
executed
*Failed tests:*
{noformat}
TestAddPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestAddPartitionsFromPartSpec - did not produce a TEST-*.xml file (likely timed
out) (batchId=230)
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestAggregateStatsCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestAlterPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestAppendPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=273)
TestCachedStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestCatalogCaching - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestCatalogNonDefaultClient - did not produce a TEST-*.xml file (likely timed
out) (batchId=228)
TestCatalogNonDefaultSvr - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestCatalogOldClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestCatalogs - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestCheckConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestDatabases - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDeadline - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestDefaultConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDropPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=273)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=231)
TestExchangePartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestFMSketchSerialization - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestForeignKey - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestFunctions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestGetPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestGetTableMeta - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHdfsUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestHiveAlterHandler - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestHiveMetaStoreGetMetaConf - did not produce a TEST-*.xml file (likely timed
out) (batchId=236)
TestHiveMetaStorePartitionSpecs - did not produce a TEST-*.xml file (likely
timed out) (batchId=230)
TestHiveMetaStoreSchemaMethods - did not produce a TEST-*.xml file (likely
timed out) (batchId=236)
TestHiveMetaStoreTimeout - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=233)
TestHiveMetastoreCli - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestHyperLogLog - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHyperLogLogDense - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestJSONMessageDeserializer - did not produce a TEST-*.xml file (likely timed
out) (batchId=236)
TestListPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestLockRequestBuilder - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestMarkPartition - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestMarkPartitionRemote - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestMetaStoreConnectionUrlHook - did not produce a TEST-*.xml file
[jira] [Resolved] (HIVE-19115) Merge: Semijoin hints are dropped by the merge
[
https://issues.apache.org/jira/browse/HIVE-19115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepak Jaiswal resolved HIVE-19115.
---
Resolution: Duplicate
Duplicate of HIVE-20354.
> Merge: Semijoin hints are dropped by the merge
> --
>
> Key: HIVE-19115
> URL: https://issues.apache.org/jira/browse/HIVE-19115
> Project: Hive
> Issue Type: Bug
> Components: Query Planning, Transactions
>Reporter: Gopal V
>Assignee: Deepak Jaiswal
>Priority: Major
>
> {code}
> create table target stored as orc as select ss_ticket_number, ss_item_sk,
> current_timestamp as `ts` from tpcds_bin_partitioned_orc_1000.store_sales;
> create table source stored as orc as select sr_ticket_number, sr_item_sk,
> d_date from tpcds_bin_partitioned_orc_1000.store_returns join
> tpcds_bin_partitioned_orc_1000.date_dim where d_date_sk = sr_returned_date_sk;
> merge /* +semi(T, sr_ticket_number, S, 1) */ into target T using (select
> * from source where year(d_date) = 1998) S ON T.ss_ticket_number =
> S.sr_ticket_number and sr_item_sk = ss_item_sk
> when matched THEN UPDATE SET ts = current_timestamp
> when not matched and sr_item_sk is not null and sr_ticket_number is not null
> THEN INSERT VALUES(S.sr_ticket_number, S.sr_item_sk, current_timestamp);
> {code}
> The semijoin hints are ignored and the code says
> {code}
> todo: do we care to preserve comments in original SQL?
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java#L624
> in this case we do.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20338) LLAP: Force synthetic file-id for filesystems which have HDFS protocol impls with POSIX mutation semantics
[ https://issues.apache.org/jira/browse/HIVE-20338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578809#comment-16578809 ] Gopal V edited comment on HIVE-20338 at 8/13/18 7:22 PM: - [~sershe]: this is true only for HDFS NameNodes (& appends don't change inode-ids - people who want to drive an HDFS developer insane can truncate an HDFS file to a smaller length and then append to it again to overwrite existing blocks). FYI - https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adv_Overviewhdfs.htm was (Author: gopalv): [~sershe]: this is true only for HDFS NameNodes. FYI - https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adv_Overviewhdfs.htm > LLAP: Force synthetic file-id for filesystems which have HDFS protocol impls > with POSIX mutation semantics > -- > > Key: HIVE-20338 > URL: https://issues.apache.org/jira/browse/HIVE-20338 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Priority: Major > Attachments: HIVE-20338.WIP.patch > > > HDFS client protocol is not a guarantee of the immutability of files - the > synthetic file-id includes the mtime of the file as well, which is a > fail-safe for filesystems which implement the client wire protocol without > offering the same storage side restrictions on immutability (i.e allow NFS > read-write-modify on the backend). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20338) LLAP: Force synthetic file-id for filesystems which have HDFS protocol impls with POSIX mutation semantics
[ https://issues.apache.org/jira/browse/HIVE-20338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-20338: --- Description: HDFS client protocol is not a guarantee of the immutability of files - the synthetic file-id includes the mtime of the file as well, which is a fail-safe for filesystems which implement the client wire protocol without offering the same storage side restrictions on immutability (i.e allow NFS read-write-modify on the backend). (was: HDFS protocol does not guarantee immutability of files - the synthetic file-id includes the mtime of the file as well, which is a fail-safe for such filesystems. ) > LLAP: Force synthetic file-id for filesystems which have HDFS protocol impls > with POSIX mutation semantics > -- > > Key: HIVE-20338 > URL: https://issues.apache.org/jira/browse/HIVE-20338 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Priority: Major > Attachments: HIVE-20338.WIP.patch > > > HDFS client protocol is not a guarantee of the immutability of files - the > synthetic file-id includes the mtime of the file as well, which is a > fail-safe for filesystems which implement the client wire protocol without > offering the same storage side restrictions on immutability (i.e allow NFS > read-write-modify on the backend). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20338) LLAP: Force synthetic file-id for filesystems which have HDFS protocol impls with POSIX mutation semantics
[ https://issues.apache.org/jira/browse/HIVE-20338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578809#comment-16578809 ] Gopal V commented on HIVE-20338: [~sershe]: this is true only for HDFS NameNodes. FYI - https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adv_Overviewhdfs.htm > LLAP: Force synthetic file-id for filesystems which have HDFS protocol impls > with POSIX mutation semantics > -- > > Key: HIVE-20338 > URL: https://issues.apache.org/jira/browse/HIVE-20338 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Priority: Major > Attachments: HIVE-20338.WIP.patch > > > HDFS protocol does not guarantee immutability of files - the synthetic > file-id includes the mtime of the file as well, which is a fail-safe for such > filesystems. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20338) LLAP: Force synthetic file-id for filesystems which have HDFS protocol impls with POSIX mutation semantics
[ https://issues.apache.org/jira/browse/HIVE-20338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-20338: --- Attachment: HIVE-20338.WIP.patch > LLAP: Force synthetic file-id for filesystems which have HDFS protocol impls > with POSIX mutation semantics > -- > > Key: HIVE-20338 > URL: https://issues.apache.org/jira/browse/HIVE-20338 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Priority: Major > Attachments: HIVE-20338.WIP.patch > > > HDFS protocol does not guarantee immutability of files - the synthetic > file-id includes the mtime of the file as well, which is a fail-safe for such > filesystems. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-20246:
-
Status: Open (was: Patch Available)
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.5.patch
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
> key2=val2);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-20246:
-
Attachment: HIVE-20246.5.patch
Status: Patch Available (was: Open)
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.5.patch
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
> key2=val2);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-20246:
-
Attachment: (was: HIVE-20246.4.patch)
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
> key2=val2);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20338) LLAP: Force synthetic file-id for filesystems which have HDFS protocol impls with POSIX mutation semantics
[ https://issues.apache.org/jira/browse/HIVE-20338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578782#comment-16578782 ] Sergey Shelukhin commented on HIVE-20338: - [~gopalv] doesn't inode id guarantee immutability? I remember discussing with someone that it would change on append. > LLAP: Force synthetic file-id for filesystems which have HDFS protocol impls > with POSIX mutation semantics > -- > > Key: HIVE-20338 > URL: https://issues.apache.org/jira/browse/HIVE-20338 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Priority: Major > > HDFS protocol does not guarantee immutability of files - the synthetic > file-id includes the mtime of the file as well, which is a fail-safe for such > filesystems. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20033) Backport HIVE-19432 to branch-2, branch-3
[
https://issues.apache.org/jira/browse/HIVE-20033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578749#comment-16578749
]
Hive QA commented on HIVE-20033:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 16s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-13193/patches/PreCommit-HIVE-Build-13193.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13193/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Backport HIVE-19432 to branch-2, branch-3
> -
>
> Key: HIVE-20033
> URL: https://issues.apache.org/jira/browse/HIVE-20033
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20033.02-branch-3.patch,
> HIVE-20033.1.branch-2.patch, HIVE-20033.1.branch-3.patch
>
>
> Backport HIVE-19432 to branch-2, branch-3
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20345) Drop database may hang if the tables get deleted from a different call
[
https://issues.apache.org/jira/browse/HIVE-20345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578743#comment-16578743
]
Naveen Gangam commented on HIVE-20345:
--
Looks good to me .. +1
> Drop database may hang if the tables get deleted from a different call
> --
>
> Key: HIVE-20345
> URL: https://issues.apache.org/jira/browse/HIVE-20345
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-20345.1.patch
>
>
> In HiveMetaStore.java drop_database_core function, HIVE-11258 updates the
> startIndex from endIndex incorrectly inside {{if (tables != null &&
> !tables.isEmpty())}} statement. If the tables get deleted before
> getTableObjectsByName() call, then returned table list is empty and
> startIndex won't get updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20195) Split MetastoreUtils into common and server-specific parts
[
https://issues.apache.org/jira/browse/HIVE-20195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578740#comment-16578740
]
Hive QA commented on HIVE-20195:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935388/HIVE-20195.03.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14877 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13192/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13192/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13192/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935388 - PreCommit-HIVE-Build
> Split MetastoreUtils into common and server-specific parts
> --
>
> Key: HIVE-20195
> URL: https://issues.apache.org/jira/browse/HIVE-20195
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20195.01.patch, HIVE-20195.02.patch,
> HIVE-20195.03.patch
>
>
> Parts of MetastoreUtils are used by clients and the server, parts are used by
> server only. We need to separate server-only parts in a separate class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20372) WRTIE_SET typo in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-20372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20372: -- Component/s: Transactions > WRTIE_SET typo in TxnHandler > > > Key: HIVE-20372 > URL: https://issues.apache.org/jira/browse/HIVE-20372 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore, Transactions >Affects Versions: 3.1.0 >Reporter: Laszlo Bodor >Priority: Trivial > Labels: Newbie, newbie, newbie++, newbiee > Fix For: 4.0.0 > > > [https://github.com/prongs/apache-hive/blob/deabe59371e98a21f4c3a58a9d8da51e4632fca5/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L765] > minor typo -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20195) Split MetastoreUtils into common and server-specific parts
[
https://issues.apache.org/jira/browse/HIVE-20195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578621#comment-16578621
]
Hive QA commented on HIVE-20195:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2306 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
21s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
34s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hcatalog-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 3
unchanged - 2 fixed = 3 total (was 5) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore_metastore-server generated 1 new
+ 61 unchanged - 1 fixed = 62 total (was 62) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13192/dev-support/hive-personality.sh
|
| git revision | master / 681051f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13192/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13192/yetus/patch-mvninstall-itests_hcatalog-unit.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13192/yetus/patch-mvninstall-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13192/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| javadoc |
[jira] [Updated] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-20377:
---
Description:
h1. Goal
* Read streaming data form Kafka queue as an external table.
* Allow streaming navigation by pushing down filters on Kafka record partition
id, offset and timestamp.
* Insert streaming data form Kafka to an actual Hive internal table, using CTAS
statement.
h1. Example
h2. Create the external table
{code}
CREATE EXTERNAL TABLE kafka_table (`timestamp` timestamp, page string, `user`
string, language string, added int, deleted int, flags string,comment string,
namespace string)
STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
TBLPROPERTIES
("kafka.topic" = "wikipedia",
"kafka.bootstrap.servers"="brokeraddress:9092",
"kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe");
{code}
h2. Kafka Metadata
In order to keep track of Kafka records the storage handler will add
automatically the Kafka row metadata eg partition id, record offset and record
timestamp.
{code}
DESCRIBE EXTENDED kafka_table
timestamp timestamp from deserializer
pagestring from deserializer
userstring from deserializer
languagestring from deserializer
country string from deserializer
continent string from deserializer
namespace string from deserializer
newpage boolean from deserializer
unpatrolled boolean from deserializer
anonymous boolean from deserializer
robot boolean from deserializer
added int from deserializer
deleted int from deserializer
delta bigint from deserializer
__partition int from deserializer
__offsetbigint from deserializer
__timestamp bigint from deserializer
{code}
h2. Filter push down.
Newer Kafka consumers 0.11.0 and higher allow seeking on the stream based on a
given offset. The proposed storage handler will be able to leverage such API by
pushing down filters over metadata columns, namely __partition (int),
__offset(long) and __timestamp(long)
For instance Query like
{code}
select `__offset` from kafka_table where (`__offset` < 10 and `__offset`>3 and
`__partition` = 0) or (`__partition` = 0 and `__offset` < 105 and `__offset` >
99) or (`__offset` = 109);
{code}
Will result on a scan of partition 0 only then read only records between offset
4 and 109.
was:
h1. Goal
* Read streaming data form Kafka queue as an external table.
* Allow streaming navigation by pushing down filters on Kafka record partition
id, offset and timestamp.
* Insert streaming data form Kafka to an actual Hive internal table, using CTAS
statement.
h1. Example
h2. Create the external table
{code}
CREATE EXTERNAL TABLE kafka_table (`timestamp` timestamps, page string, `user`
string, language string, added int, deleted int, flags string,comment string,
namespace string)
STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
TBLPROPERTIES
("kafka.topic" = "wikipedia",
"kafka.bootstrap.servers"="brokeraddress:9092",
"kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe");
{code}
h2. Kafka Metadata
In order to keep track of Kafka records the storage handler will add
automatically the Kafka row metadata eg partition id, record offset and record
timestamp.
{code}
DESCRIBE EXTENDED kafka_table
timestamp timestamp from deserializer
pagestring from deserializer
userstring from deserializer
languagestring from deserializer
country string from deserializer
continent string from deserializer
namespace string from deserializer
newpage boolean from deserializer
unpatrolled boolean from deserializer
anonymous boolean from deserializer
robot boolean from deserializer
added int from deserializer
deleted int from deserializer
delta bigint from deserializer
__partition int from deserializer
__offsetbigint from deserializer
[jira] [Updated] (HIVE-18444) when creating transactional table make sure location has no data
[ https://issues.apache.org/jira/browse/HIVE-18444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18444: -- Target Version/s: 4.0.0 (was: 3.0.0) > when creating transactional table make sure location has no data > > > Key: HIVE-18444 > URL: https://issues.apache.org/jira/browse/HIVE-18444 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Vaibhav Gumashta >Priority: Major > > if a user creates a new transactional table but sets a location to some place > that already has data any number of things can break. > Data may not be in Acid format, it may have been written by another cluster > and txnids won't make sense in current cluster. Once per table writeIDs > (HIVE-18192) are there, if the data was written by another table, writeIDs > won't match. > This could actually work if the data at the existing location was not written > by an acid write but it would be safer/cleaner to just prevent this (at least > at first). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18444) when creating transactional table make sure location has no data
[ https://issues.apache.org/jira/browse/HIVE-18444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18444: - Assignee: Vaibhav Gumashta > when creating transactional table make sure location has no data > > > Key: HIVE-18444 > URL: https://issues.apache.org/jira/browse/HIVE-18444 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Vaibhav Gumashta >Priority: Major > > if a user creates a new transactional table but sets a location to some place > that already has data any number of things can break. > Data may not be in Acid format, it may have been written by another cluster > and txnids won't make sense in current cluster. Once per table writeIDs > (HIVE-18192) are there, if the data was written by another table, writeIDs > won't match. > This could actually work if the data at the existing location was not written > by an acid write but it would be safer/cleaner to just prevent this (at least > at first). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-19081:
-
Assignee: Vaibhav Gumashta (was: Eugene Koifman)
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20371) Queries failing with Internal error processing add_write_notification_log
[
https://issues.apache.org/jira/browse/HIVE-20371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20371:
Status: Patch Available (was: Open)
Flaky test failure. Re-attaching same patch.
> Queries failing with Internal error processing add_write_notification_log
> -
>
> Key: HIVE-20371
> URL: https://issues.apache.org/jira/browse/HIVE-20371
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl, Standalone Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-20371.01.patch
>
>
> Queries failing with following error:
> {noformat}
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.thrift.TApplicationException: Internal error processing
> add_write_notification_log
> INFO : Completed executing
> command(queryId=hive_20180806072916_a9ae37a9-869f-4218-8357-a96ba713db69);
> Time taken: 878.604 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.thrift.TApplicationException: Internal error processing
> add_write_notification_log (state=08S01,code=1)
> {noformat}
> From hiveserver log:
> {noformat}
> 2018-08-06T07:59:33,656 ERROR [HiveServer2-Background-Pool: Thread-1551]:
> operation.Operation (:()) - Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.thrift.TApplicationException: Internal error processing
> add_write_notification_log
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
> ~[hive-service-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226)
> ~[hive-service-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
> ~[hive-service-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)
> ~[hive-service-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> ~[hadoop-common-3.1.0.3.0.1.0-59.jar:?]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:329)
> ~[hive-service-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_112]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_112]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: Internal error processing
> add_write_notification_log
> at
> org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:2879)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2035)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:477)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:397)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2679)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2350)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2026)
>
[jira] [Updated] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-19081:
--
Target Version/s: 3.1.0 (was: 3.0.0)
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19115) Merge: Semijoin hints are dropped by the merge
[
https://issues.apache.org/jira/browse/HIVE-19115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578608#comment-16578608
]
Eugene Koifman commented on HIVE-19115:
---
[~djaiswal] is this a dup?
> Merge: Semijoin hints are dropped by the merge
> --
>
> Key: HIVE-19115
> URL: https://issues.apache.org/jira/browse/HIVE-19115
> Project: Hive
> Issue Type: Bug
> Components: Query Planning, Transactions
>Reporter: Gopal V
>Assignee: Deepak Jaiswal
>Priority: Major
>
> {code}
> create table target stored as orc as select ss_ticket_number, ss_item_sk,
> current_timestamp as `ts` from tpcds_bin_partitioned_orc_1000.store_sales;
> create table source stored as orc as select sr_ticket_number, sr_item_sk,
> d_date from tpcds_bin_partitioned_orc_1000.store_returns join
> tpcds_bin_partitioned_orc_1000.date_dim where d_date_sk = sr_returned_date_sk;
> merge /* +semi(T, sr_ticket_number, S, 1) */ into target T using (select
> * from source where year(d_date) = 1998) S ON T.ss_ticket_number =
> S.sr_ticket_number and sr_item_sk = ss_item_sk
> when matched THEN UPDATE SET ts = current_timestamp
> when not matched and sr_item_sk is not null and sr_ticket_number is not null
> THEN INSERT VALUES(S.sr_ticket_number, S.sr_item_sk, current_timestamp);
> {code}
> The semijoin hints are ignored and the code says
> {code}
> todo: do we care to preserve comments in original SQL?
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java#L624
> in this case we do.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20371) Queries failing with Internal error processing add_write_notification_log
[
https://issues.apache.org/jira/browse/HIVE-20371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20371:
Attachment: HIVE-20371.01.patch
> Queries failing with Internal error processing add_write_notification_log
> -
>
> Key: HIVE-20371
> URL: https://issues.apache.org/jira/browse/HIVE-20371
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl, Standalone Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-20371.01.patch
>
>
> Queries failing with following error:
> {noformat}
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.thrift.TApplicationException: Internal error processing
> add_write_notification_log
> INFO : Completed executing
> command(queryId=hive_20180806072916_a9ae37a9-869f-4218-8357-a96ba713db69);
> Time taken: 878.604 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.thrift.TApplicationException: Internal error processing
> add_write_notification_log (state=08S01,code=1)
> {noformat}
> From hiveserver log:
> {noformat}
> 2018-08-06T07:59:33,656 ERROR [HiveServer2-Background-Pool: Thread-1551]:
> operation.Operation (:()) - Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.thrift.TApplicationException: Internal error processing
> add_write_notification_log
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
> ~[hive-service-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226)
> ~[hive-service-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
> ~[hive-service-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)
> ~[hive-service-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> ~[hadoop-common-3.1.0.3.0.1.0-59.jar:?]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:329)
> ~[hive-service-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_112]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_112]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: Internal error processing
> add_write_notification_log
> at
> org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:2879)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2035)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:477)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:397)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2679)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2350)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2026)
> ~[hive-exec-3.1.0.3.0.1.0-59.jar:3.1.0.3.0.1.0-59]
> at
