[jira] [Commented] (HIVE-14747) Remove JAVA paths from profiles by sending them from ptest-client
[
https://issues.apache.org/jira/browse/HIVE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118361#comment-16118361
]
Hive QA commented on HIVE-14747:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880822/HIVE-14747.04.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10993 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_decimal]
(batchId=9)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6298/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6298/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6298/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880822 - PreCommit-HIVE-Build
> Remove JAVA paths from profiles by sending them from ptest-client
> -
>
> Key: HIVE-14747
> URL: https://issues.apache.org/jira/browse/HIVE-14747
> Project: Hive
> Issue Type: Sub-task
> Components: Hive, Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-14747.01.patch, HIVE-14747.02.patch,
> HIVE-14747.03.patch, HIVE-14747.04.patch
>
>
> Hive ptest uses some properties files per branch that contain information
> about how to execute the tests.
> This profile includes JAVA paths to build and execute the tests. We should
> get rid of these by passing such information from Jenkins to the
> ptest-server. In case a profile needs a different java version, then we can
> create a specific Jenkins job for that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17251) Remove usage of org.apache.pig.ResourceStatistics#setmBytes method in HCatLoader
[ https://issues.apache.org/jira/browse/HIVE-17251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated HIVE-17251: -- Status: Patch Available (was: In Progress) > Remove usage of org.apache.pig.ResourceStatistics#setmBytes method in > HCatLoader > > > Key: HIVE-17251 > URL: https://issues.apache.org/jira/browse/HIVE-17251 > Project: Hive > Issue Type: Improvement > Components: HCatalog >Reporter: Nandor Kollar >Assignee: Adam Szita >Priority: Minor > Attachments: HIVE-17251.0.patch > > > org.apache.pig.ResourceStatistics#setmBytes is marked as deprecated, and is > going to be removed from Pig. Is it possible to use use the the proper > replacement method (ResourceStatistics#setSizeInBytes) instead? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17251) Remove usage of org.apache.pig.ResourceStatistics#setmBytes method in HCatLoader
[ https://issues.apache.org/jira/browse/HIVE-17251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated HIVE-17251: -- Attachment: HIVE-17251.0.patch > Remove usage of org.apache.pig.ResourceStatistics#setmBytes method in > HCatLoader > > > Key: HIVE-17251 > URL: https://issues.apache.org/jira/browse/HIVE-17251 > Project: Hive > Issue Type: Improvement > Components: HCatalog >Reporter: Nandor Kollar >Assignee: Adam Szita >Priority: Minor > Attachments: HIVE-17251.0.patch > > > org.apache.pig.ResourceStatistics#setmBytes is marked as deprecated, and is > going to be removed from Pig. Is it possible to use use the the proper > replacement method (ResourceStatistics#setSizeInBytes) instead? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17268) WebUI / QueryPlan: query plan is sometimes null when explain output conf is on
[
https://issues.apache.org/jira/browse/HIVE-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118342#comment-16118342
]
Peter Vary commented on HIVE-17268:
---
I think it is acceptable, that the {{hive.log.explain.output}} could not be set
per session basis, so I agree with your solution.
However this patch fixes changes made by HIVE-16982. I would prefer to revert
to the original code since the expected output is the same in case of {{null}}
value. By this, I mean we should revert the changes in the getExplainPlan
method to {{returnStringOrUnknown(explainPlan)}}, and add the config check
directly to the jamon file as you did in this patch.
If it would be a green field code, I would prefer to do the check on the jamon
side, but several other places we use the returnStringOrUnknown to handle
null-s, and in case of codebase this big I prefer uniformity above all :)
Thanks for the report and the patch!
Peter
> WebUI / QueryPlan: query plan is sometimes null when explain output conf is on
> --
>
> Key: HIVE-17268
> URL: https://issues.apache.org/jira/browse/HIVE-17268
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Attachments: HIVE-17268.patch
>
>
> The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true
> TO VIEW PLAN" even when hive.log.explain.output is set to true, when the
> query cannot be compiled, because the plan is null in this case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Work started] (HIVE-17251) Remove usage of org.apache.pig.ResourceStatistics#setmBytes method in HCatLoader
[ https://issues.apache.org/jira/browse/HIVE-17251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-17251 started by Adam Szita. - > Remove usage of org.apache.pig.ResourceStatistics#setmBytes method in > HCatLoader > > > Key: HIVE-17251 > URL: https://issues.apache.org/jira/browse/HIVE-17251 > Project: Hive > Issue Type: Improvement > Components: HCatalog >Reporter: Nandor Kollar >Assignee: Adam Szita >Priority: Minor > > org.apache.pig.ResourceStatistics#setmBytes is marked as deprecated, and is > going to be removed from Pig. Is it possible to use use the the proper > replacement method (ResourceStatistics#setSizeInBytes) instead? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17268) WebUI / QueryPlan: query plan is sometimes null when explain output conf is on
[ https://issues.apache.org/jira/browse/HIVE-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118333#comment-16118333 ] Karen Coppage commented on HIVE-17268: -- [~pvary] Would you mind checking out this fix please? > WebUI / QueryPlan: query plan is sometimes null when explain output conf is on > -- > > Key: HIVE-17268 > URL: https://issues.apache.org/jira/browse/HIVE-17268 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > Attachments: HIVE-17268.patch > > > The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true > TO VIEW PLAN" even when hive.log.explain.output is set to true, when the > query cannot be compiled, because the plan is null in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17268) WebUI / QueryPlan: query plan is sometimes null when explain output conf is on
[ https://issues.apache.org/jira/browse/HIVE-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-17268: - Description: The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true TO VIEW PLAN" even when hive.log.explain.output is set to true, when the query cannot be compiled, because the plan is null in this case. (was: The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true TO VIEW PLAN" when the statement cannot be compiled (as well as when hive.log.explain.output is off), because the query plan is null in this case.) > WebUI / QueryPlan: query plan is sometimes null when explain output conf is on > -- > > Key: HIVE-17268 > URL: https://issues.apache.org/jira/browse/HIVE-17268 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > Attachments: HIVE-17268.patch > > > The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true > TO VIEW PLAN" even when hive.log.explain.output is set to true, when the > query cannot be compiled, because the plan is null in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17268) WebUI / QueryPlan: query plan is sometimes null when explain output conf is on
[ https://issues.apache.org/jira/browse/HIVE-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-17268: - Status: Patch Available (was: In Progress) > WebUI / QueryPlan: query plan is sometimes null when explain output conf is on > -- > > Key: HIVE-17268 > URL: https://issues.apache.org/jira/browse/HIVE-17268 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > Attachments: HIVE-17268.patch > > > The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true > TO VIEW PLAN" when the statement cannot be compiled (as well as when > hive.log.explain.output is off), because the query plan is null in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17268) WebUI / QueryPlan: query plan is sometimes null when explain output conf is on
[ https://issues.apache.org/jira/browse/HIVE-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-17268: - Attachment: HIVE-17268.patch > WebUI / QueryPlan: query plan is sometimes null when explain output conf is on > -- > > Key: HIVE-17268 > URL: https://issues.apache.org/jira/browse/HIVE-17268 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > Attachments: HIVE-17268.patch > > > The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true > TO VIEW PLAN" when the statement cannot be compiled (as well as when > hive.log.explain.output is off), because the query plan is null in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Work started] (HIVE-17268) WebUI / QueryPlan: query plan is sometimes null when explain output conf is on
[ https://issues.apache.org/jira/browse/HIVE-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-17268 started by Karen Coppage. > WebUI / QueryPlan: query plan is sometimes null when explain output conf is on > -- > > Key: HIVE-17268 > URL: https://issues.apache.org/jira/browse/HIVE-17268 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > > The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true > TO VIEW PLAN" when the statement cannot be compiled (as well as when > hive.log.explain.output is off), because the query plan is null in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17268) WebUI / QueryPlan: query plan is sometimes null when explain output conf is on
[ https://issues.apache.org/jira/browse/HIVE-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-17268: Assignee: Karen Coppage > WebUI / QueryPlan: query plan is sometimes null when explain output conf is on > -- > > Key: HIVE-17268 > URL: https://issues.apache.org/jira/browse/HIVE-17268 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > > The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true > TO VIEW PLAN" when the statement cannot be compiled (as well as when > hive.log.explain.output is off), because the query plan is null in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17268) WebUI / QueryPlan: query plan is sometimes null when explain output conf is on
[ https://issues.apache.org/jira/browse/HIVE-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118329#comment-16118329 ] Karen Coppage commented on HIVE-17268: -- Suggested solution: Give Query Profile access to HiveConf, and check hive.log.explain.output in QueryProfileTmpl. If off, display "Set configuration hive.log.explain.output to true to view future query plans" If on, and plan is null, display "Unknown" Downside: HiveConf cannot be set dynamically, and is equivalent to the configuration provided in hive-site.xml > WebUI / QueryPlan: query plan is sometimes null when explain output conf is on > -- > > Key: HIVE-17268 > URL: https://issues.apache.org/jira/browse/HIVE-17268 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Priority: Minor > > The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true > TO VIEW PLAN" when the statement cannot be compiled (as well as when > hive.log.explain.output is off), because the query plan is null in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17235: Status: Patch Available (was: In Progress) > Add ORC Decimal64 Serialization/Deserialization (Part 1) > > > Key: HIVE-17235 > URL: https://issues.apache.org/jira/browse/HIVE-17235 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch, > HIVE-17235.05.patch, HIVE-17235.06.patch, HIVE-17235.07.patch, > HIVE-17235.patch > > > The storage-api changes for ORC-209. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17235: Attachment: HIVE-17235.07.patch > Add ORC Decimal64 Serialization/Deserialization (Part 1) > > > Key: HIVE-17235 > URL: https://issues.apache.org/jira/browse/HIVE-17235 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch, > HIVE-17235.05.patch, HIVE-17235.06.patch, HIVE-17235.07.patch, > HIVE-17235.patch > > > The storage-api changes for ORC-209. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17235: Status: In Progress (was: Patch Available) > Add ORC Decimal64 Serialization/Deserialization (Part 1) > > > Key: HIVE-17235 > URL: https://issues.apache.org/jira/browse/HIVE-17235 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch, > HIVE-17235.05.patch, HIVE-17235.06.patch, HIVE-17235.patch > > > The storage-api changes for ORC-209. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17268) WebUI / QueryPlan: query plan is sometimes null when explain output conf is on
[ https://issues.apache.org/jira/browse/HIVE-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-17268: - Summary: WebUI / QueryPlan: query plan is sometimes null when explain output conf is on (was: WebUI / QueryPlan: recommends setting a conf when ) > WebUI / QueryPlan: query plan is sometimes null when explain output conf is on > -- > > Key: HIVE-17268 > URL: https://issues.apache.org/jira/browse/HIVE-17268 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Priority: Minor > > The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true > TO VIEW PLAN" when the statement cannot be compiled (as well as when > hive.log.explain.output is off), because the query plan is null in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17268) WebUI / QueryPlan: recommends setting a conf when
[ https://issues.apache.org/jira/browse/HIVE-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-17268: - Description: The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true TO VIEW PLAN" when the statement cannot be compiled (as well as when hive.log.explain.output is off), because the query plan is null in this case. (was: The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true TO VIEW PLAN" when the statement cannot be compiled (as well as when hive.log.explain.output is off), because the query plan is null in this case. ) > WebUI / QueryPlan: recommends setting a conf when > -- > > Key: HIVE-17268 > URL: https://issues.apache.org/jira/browse/HIVE-17268 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Priority: Minor > > The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true > TO VIEW PLAN" when the statement cannot be compiled (as well as when > hive.log.explain.output is off), because the query plan is null in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17268) WebUI / QueryPlan: recommends setting a conf when
[ https://issues.apache.org/jira/browse/HIVE-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-17268: - Summary: WebUI / QueryPlan: recommends setting a conf when (was: WebUI / QueryPlan: query plan is sometimes null when explain output conf is on) > WebUI / QueryPlan: recommends setting a conf when > -- > > Key: HIVE-17268 > URL: https://issues.apache.org/jira/browse/HIVE-17268 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Priority: Minor > > The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true > TO VIEW PLAN" when the statement cannot be compiled (as well as when > hive.log.explain.output is off), because the query plan is null in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17267) Make HMS Notification Listeners typesafe
[
https://issues.apache.org/jira/browse/HIVE-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118301#comment-16118301
]
Hive QA commented on HIVE-17267:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880813/HIVE-17267.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10993 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=240)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6297/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6297/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6297/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880813 - PreCommit-HIVE-Build
> Make HMS Notification Listeners typesafe
>
>
> Key: HIVE-17267
> URL: https://issues.apache.org/jira/browse/HIVE-17267
> Project: Hive
> Issue Type: Bug
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-17267.01.patch
>
>
> Currently in the HMS we support two types of notification listeners,
> transactional and non-transactional ones. Transactional listeners will only
> be invoked if the jdbc transaction finished successfully while
> non-transactional ones are supposed to be resilient and will be invoked in
> any case, even for failures.
> Having the same type for these two is a source of confusion and opens the
> door for misconfigurations. We should try to fix this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16945) Add method to compare Operators
[
https://issues.apache.org/jira/browse/HIVE-16945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118288#comment-16118288
]
Jesus Camacho Rodriguez commented on HIVE-16945:
[~lirui], sorry for taking this long to review.
Patch LGTM. Knitpick: could you add null check for {{other.getConf()}} in
{{logicalEquals}} method? Thanks
> Add method to compare Operators
>
>
> Key: HIVE-16945
> URL: https://issues.apache.org/jira/browse/HIVE-16945
> Project: Hive
> Issue Type: Improvement
> Components: Operators
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Rui Li
> Attachments: HIVE-16945.1.patch, HIVE-16945.2.patch
>
>
> HIVE-10844 introduced a comparator factory class for operators that
> encapsulates all the logic to assess whether two operators are equal:
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java
> The current design might create problems as any change in fields of operators
> will break the comparators. It would be better to do this via inheritance
> from Operator base class, by adding a {{logicalEquals(Operator other)}}
> method.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-14747) Remove JAVA paths from profiles by sending them from ptest-client
[ https://issues.apache.org/jira/browse/HIVE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-14747: --- Attachment: HIVE-14747.04.patch > Remove JAVA paths from profiles by sending them from ptest-client > - > > Key: HIVE-14747 > URL: https://issues.apache.org/jira/browse/HIVE-14747 > Project: Hive > Issue Type: Sub-task > Components: Hive, Testing Infrastructure >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-14747.01.patch, HIVE-14747.02.patch, > HIVE-14747.03.patch, HIVE-14747.04.patch > > > Hive ptest uses some properties files per branch that contain information > about how to execute the tests. > This profile includes JAVA paths to build and execute the tests. We should > get rid of these by passing such information from Jenkins to the > ptest-server. In case a profile needs a different java version, then we can > create a specific Jenkins job for that. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15225) QueryPlan.getJSONValue should code against empty string values
[
https://issues.apache.org/jira/browse/HIVE-15225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118237#comment-16118237
]
Hive QA commented on HIVE-15225:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12839260/HIVE-15225.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10993 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=240)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6296/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6296/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6296/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12839260 - PreCommit-HIVE-Build
> QueryPlan.getJSONValue should code against empty string values
> --
>
> Key: HIVE-15225
> URL: https://issues.apache.org/jira/browse/HIVE-15225
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-15225.1.patch
>
>
> The current {{QueryPlan.getJSONValue}} implementation is as below:
> {code}
> public String getJSONValue(Object value) {
> String v = "null";
> if (value != null) {
> v = value.toString();
> if (v.charAt(0) != '[' && v.charAt(0) != '{') {
> v = "\"" + v + "\"";
> }
> }
> return v;
> }
> {code}
> When {{value.toString()}} returns an empty string, a
> StringIndexOutOfRangeException would be thrown out when "v.charAt(0)" is
> evaluated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17267) Make HMS Notification Listeners typesafe
[ https://issues.apache.org/jira/browse/HIVE-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17267: --- Status: Patch Available (was: Open) > Make HMS Notification Listeners typesafe > > > Key: HIVE-17267 > URL: https://issues.apache.org/jira/browse/HIVE-17267 > Project: Hive > Issue Type: Bug >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17267.01.patch > > > Currently in the HMS we support two types of notification listeners, > transactional and non-transactional ones. Transactional listeners will only > be invoked if the jdbc transaction finished successfully while > non-transactional ones are supposed to be resilient and will be invoked in > any case, even for failures. > Having the same type for these two is a source of confusion and opens the > door for misconfigurations. We should try to fix this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17267) Make HMS Notification Listeners typesafe
[ https://issues.apache.org/jira/browse/HIVE-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17267: --- Attachment: HIVE-17267.01.patch Attaching first draft. > Make HMS Notification Listeners typesafe > > > Key: HIVE-17267 > URL: https://issues.apache.org/jira/browse/HIVE-17267 > Project: Hive > Issue Type: Bug >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17267.01.patch > > > Currently in the HMS we support two types of notification listeners, > transactional and non-transactional ones. Transactional listeners will only > be invoked if the jdbc transaction finished successfully while > non-transactional ones are supposed to be resilient and will be invoked in > any case, even for failures. > Having the same type for these two is a source of confusion and opens the > door for misconfigurations. We should try to fix this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17155) findConfFile() in HiveConf.java has some issues with the conf path
[ https://issues.apache.org/jira/browse/HIVE-17155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaoqiang_song reassigned HIVE-17155: - Assignee: xiaoqiang_song (was: Aihua Xu) > findConfFile() in HiveConf.java has some issues with the conf path > -- > > Key: HIVE-17155 > URL: https://issues.apache.org/jira/browse/HIVE-17155 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Aihua Xu >Assignee: xiaoqiang_song >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-17155.1.patch > > > In findConfFile() function of HiveConf.java, here are some issues. > File.pathSeparator which is ":" is used as the separator rather than "/". new > File(jarUri).getParentFile() will get the "$hive_home/lib" folder, but > actually we want "$hive_home". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16784) Missing lineage information when hive.blobstore.optimizations.enabled is true
[
https://issues.apache.org/jira/browse/HIVE-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118195#comment-16118195
]
Barna Zsombor Klara commented on HIVE-16784:
Thanks for the review [~stakiar]. Do you need more time for the continued
review? If not [~pvary] is eager to commit this.
> Missing lineage information when hive.blobstore.optimizations.enabled is true
> -
>
> Key: HIVE-16784
> URL: https://issues.apache.org/jira/browse/HIVE-16784
> Project: Hive
> Issue Type: Bug
>Reporter: Marta Kuczora
>Assignee: Barna Zsombor Klara
> Fix For: 3.0.0
>
> Attachments: HIVE-16784.01.patch, HIVE-16784.02.patch,
> HIVE-16784.02.patch
>
>
> Running the commands of the add_part_multiple.q test on S3 with
> hive.blobstore.optimizations.enabled=true fails because of missing lineage
> information.
> Running the command on HDFS
> {noformat}
> from src TABLESAMPLE (1 ROWS)
> insert into table add_part_test PARTITION (ds='2010-01-01') select 100,100
> insert into table add_part_test PARTITION (ds='2010-02-01') select 200,200
> insert into table add_part_test PARTITION (ds='2010-03-01') select 400,300
> insert into table add_part_test PARTITION (ds='2010-04-01') select 500,400;
> {noformat}
> results the following posthook outputs
> {noformat}
> POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-01-01).key EXPRESSION []
> POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-01-01).value EXPRESSION []
> POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-02-01).key EXPRESSION []
> POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-02-01).value EXPRESSION []
> POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-03-01).key EXPRESSION []
> POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-03-01).value EXPRESSION []
> POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-04-01).key EXPRESSION []
> POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-04-01).value EXPRESSION []
> {noformat}
> These lines are not printed when running the command on the table located in
> S3.
> If hive.blobstore.optimizations.enabled=false, the lineage information is
> printed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17008) Fix boolean flag switchup in DropTableEvent
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118172#comment-16118172
]
Hive QA commented on HIVE-17008:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880801/HIVE-17008.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 10993 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge_incompat2]
(batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6295/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6295/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6295/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880801 - PreCommit-HIVE-Build
> Fix boolean flag switchup in DropTableEvent
> ---
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Dan Burkert
> Attachments: HIVE-17008.0.patch, HIVE-17008.1.patch,
> HIVE-17008.2.patch, HIVE-17008.3.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16948) Invalid explain when running dynamic partition pruning query in Hive On Spark
[
https://issues.apache.org/jira/browse/HIVE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118112#comment-16118112
]
Hive QA commented on HIVE-16948:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880794/HIVE-16948.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6294/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6294/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6294/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-08-08 09:46:15.376
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-6294/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-08-08 09:46:15.379
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 6b03a9c HIVE-17247: HoS DPP: UDFs on the partition column side
does not evaluate correctly (Sahil Takiar, reviewed by Rui Li)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 6b03a9c HIVE-17247: HoS DPP: UDFs on the partition column side
does not evaluate correctly (Sahil Takiar, reviewed by Rui Li)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-08-08 09:46:21.472
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java: No such
file or directory
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/CombineEquivalentWorkResolver.java:
No such file or directory
error: a/ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestSparkTask.java:
No such file or directory
error:
a/ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning.q.out:
No such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880794 - PreCommit-HIVE-Build
> Invalid explain when running dynamic partition pruning query in Hive On Spark
> -
>
> Key: HIVE-16948
> URL: https://issues.apache.org/jira/browse/HIVE-16948
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16948_1.patch, HIVE-16948.2.patch,
> HIVE-16948.5.patch, HIVE-16948.patch
>
>
> in
> [union_subquery.q|https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/spark_dynamic_partition_pruning.q#L107]
> in spark_dynamic_partition_pruning.q
> {code}
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.spark.dynamic.partition.pruning=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.strict.checks.cartesian.product=false;
> explain select ds from (select distinct(ds) as ds from srcpart union all
> select distinct(ds) as ds from srcpart) s where s.ds in (select
> max(srcpart.ds) from srcpart union all select min(srcpart.ds) from srcpart);
> {code}
> explain
> {code}
> STAGE DEPENDENCIES:
> Stage-2 is a root stage
> Stage-1 depends on stages: Stage-2
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-2
> Spark
> Edges:
> Reducer 11 <- Map 10 (GROUP, 1)
> Reducer 13 <- Map 12 (GROUP, 1)
> DagName: root_20170622231525_20a777e5-e659-4138-b605-65f8395e18e2:2
>
[jira] [Commented] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization (Part 1)
[
https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118111#comment-16118111
]
Hive QA commented on HIVE-17235:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880774/HIVE-17235.06.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 10995 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hadoop.hive.serde2.io.TestHiveDecimalWritableVersion.testVerifyHiveDecimalWritablePublicMethodsAndFieldsVersions
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6293/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6293/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6293/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880774 - PreCommit-HIVE-Build
> Add ORC Decimal64 Serialization/Deserialization (Part 1)
>
>
> Key: HIVE-17235
> URL: https://issues.apache.org/jira/browse/HIVE-17235
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch,
> HIVE-17235.05.patch, HIVE-17235.06.patch, HIVE-17235.patch
>
>
> The storage-api changes for ORC-209.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17267) Make HMS Notification Listeners typesafe
[ https://issues.apache.org/jira/browse/HIVE-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara reassigned HIVE-17267: -- > Make HMS Notification Listeners typesafe > > > Key: HIVE-17267 > URL: https://issues.apache.org/jira/browse/HIVE-17267 > Project: Hive > Issue Type: Bug >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > > Currently in the HMS we support two types of notification listeners, > transactional and non-transactional ones. Transactional listeners will only > be invoked if the jdbc transaction finished successfully while > non-transactional ones are supposed to be resilient and will be invoked in > any case, even for failures. > Having the same type for these two is a source of confusion and opens the > door for misconfigurations. We should try to fix this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15225) QueryPlan.getJSONValue should code against empty string values
[
https://issues.apache.org/jira/browse/HIVE-15225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118090#comment-16118090
]
Peter Vary commented on HIVE-15225:
---
Thanks for the catch!
Is there any easy way to add a test case?
Thanks,
Peter
> QueryPlan.getJSONValue should code against empty string values
> --
>
> Key: HIVE-15225
> URL: https://issues.apache.org/jira/browse/HIVE-15225
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-15225.1.patch
>
>
> The current {{QueryPlan.getJSONValue}} implementation is as below:
> {code}
> public String getJSONValue(Object value) {
> String v = "null";
> if (value != null) {
> v = value.toString();
> if (v.charAt(0) != '[' && v.charAt(0) != '{') {
> v = "\"" + v + "\"";
> }
> }
> return v;
> }
> {code}
> When {{value.toString()}} returns an empty string, a
> StringIndexOutOfRangeException would be thrown out when "v.charAt(0)" is
> evaluated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17008) Fix boolean flag switchup in DropTableEvent
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara reassigned HIVE-17008:
--
Assignee: Dan Burkert (was: Barna Zsombor Klara)
> Fix boolean flag switchup in DropTableEvent
> ---
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Dan Burkert
> Attachments: HIVE-17008.0.patch, HIVE-17008.1.patch,
> HIVE-17008.2.patch, HIVE-17008.3.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17008) Fix boolean flag switchup in DropTableEvent
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara updated HIVE-17008:
---
Attachment: HIVE-17008.3.patch
> Fix boolean flag switchup in DropTableEvent
> ---
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-17008.0.patch, HIVE-17008.1.patch,
> HIVE-17008.2.patch, HIVE-17008.3.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17008) Fix boolean flag switchup in DropTableEvent
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara reassigned HIVE-17008:
--
Assignee: Barna Zsombor Klara
> Fix boolean flag switchup in DropTableEvent
> ---
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-17008.0.patch, HIVE-17008.1.patch,
> HIVE-17008.2.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17008) Fix boolean flag switchup in DropTableEvent
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara updated HIVE-17008:
---
Summary: Fix boolean flag switchup in DropTableEvent (was:
DbNotificationListener should skip failed events)
> Fix boolean flag switchup in DropTableEvent
> ---
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
> Attachments: HIVE-17008.0.patch, HIVE-17008.1.patch,
> HIVE-17008.2.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17265) Cache merged column stats from retrieved partitions
[
https://issues.apache.org/jira/browse/HIVE-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118053#comment-16118053
]
Hive QA commented on HIVE-17265:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880762/HIVE-17265.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10993 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort]
(batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6292/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6292/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6292/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880762 - PreCommit-HIVE-Build
> Cache merged column stats from retrieved partitions
> ---
>
> Key: HIVE-17265
> URL: https://issues.apache.org/jira/browse/HIVE-17265
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-17265.patch
>
>
> Currently when we retrieve stats from the metastore for a column in a
> partitioned table, we will execute the logic to merge the column stats coming
> from each partition multiple times.
> Even though we avoid multiple calls to metastore if the cache for the stats
> in enabled, merging the stats for a given column can take a large amount of
> time if there is a large number of partitions.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16948) Invalid explain when running dynamic partition pruning query in Hive On Spark
[
https://issues.apache.org/jira/browse/HIVE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang_intel updated HIVE-16948:
Attachment: HIVE-16948.5.patch
changes:
1. remove the empty sparkWork and sparkTask
2. add test in TestSparkTask to test removeEmptySparkTask. Directly copy
part code of
CombineEquivalentWorkResolver.EquivalentWorkMatcher#removeEmptySparkTask to
TestSparkTask to test. It is better to mock some situation to test
EquivalentWorkMatcher#removeEmptySparkTask even this situation is rarely
occurred. If this is unnecessary, i can remove them.
> Invalid explain when running dynamic partition pruning query in Hive On Spark
> -
>
> Key: HIVE-16948
> URL: https://issues.apache.org/jira/browse/HIVE-16948
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16948_1.patch, HIVE-16948.2.patch,
> HIVE-16948.5.patch, HIVE-16948.patch
>
>
> in
> [union_subquery.q|https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/spark_dynamic_partition_pruning.q#L107]
> in spark_dynamic_partition_pruning.q
> {code}
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.spark.dynamic.partition.pruning=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.strict.checks.cartesian.product=false;
> explain select ds from (select distinct(ds) as ds from srcpart union all
> select distinct(ds) as ds from srcpart) s where s.ds in (select
> max(srcpart.ds) from srcpart union all select min(srcpart.ds) from srcpart);
> {code}
> explain
> {code}
> STAGE DEPENDENCIES:
> Stage-2 is a root stage
> Stage-1 depends on stages: Stage-2
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-2
> Spark
> Edges:
> Reducer 11 <- Map 10 (GROUP, 1)
> Reducer 13 <- Map 12 (GROUP, 1)
> DagName: root_20170622231525_20a777e5-e659-4138-b605-65f8395e18e2:2
> Vertices:
> Map 10
> Map Operator Tree:
> TableScan
> alias: srcpart
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Group By Operator
> aggregations: max(ds)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string)
> Map 12
> Map Operator Tree:
> TableScan
> alias: srcpart
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats:
> PARTIAL Column stats: NONE
> Group By Operator
> aggregations: min(ds)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string)
> Reducer 11
> Reduce Operator Tree:
> Group By Operator
> aggregations: max(VALUE._col0)
> mode: mergepartial
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: _col0 is not null (type: boolean)
> Statistics: Num rows: 1 Data size: 184 Basic stats:
> COMPLETE Column stats: NONE
> Group By Operator
> keys: _col0 (type: string)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 2 Data size: 368 Basic stats:
>
[jira] [Commented] (HIVE-16811) Estimate statistics in absence of stats
[
https://issues.apache.org/jira/browse/HIVE-16811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118001#comment-16118001
]
Hive QA commented on HIVE-16811:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880753/HIVE-16811.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10994 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[udaf_collect_set_2]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6291/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6291/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6291/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880753 - PreCommit-HIVE-Build
> Estimate statistics in absence of stats
> ---
>
> Key: HIVE-16811
> URL: https://issues.apache.org/jira/browse/HIVE-16811
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16811.1.patch, HIVE-16811.2.patch,
> HIVE-16811.3.patch, HIVE-16811.4.patch, HIVE-16811.5.patch, HIVE-16811.6.patch
>
>
> Currently Join ordering completely bails out in absence of statistics and
> this could lead to bad joins such as cross joins.
> e.g. following select query will produce cross join.
> {code:sql}
> create table supplier (S_SUPPKEY INT, S_NAME STRING, S_ADDRESS STRING,
> S_NATIONKEY INT,
> S_PHONE STRING, S_ACCTBAL DOUBLE, S_COMMENT STRING)
> CREATE TABLE lineitem (L_ORDERKEY INT,
> L_PARTKEY INT,
> L_SUPPKEY INT,
> L_LINENUMBERINT,
> L_QUANTITY DOUBLE,
> L_EXTENDEDPRICE DOUBLE,
> L_DISCOUNT DOUBLE,
> L_TAX DOUBLE,
> L_RETURNFLAGSTRING,
> L_LINESTATUSSTRING,
> l_shipdate STRING,
> L_COMMITDATESTRING,
> L_RECEIPTDATE STRING,
> L_SHIPINSTRUCT STRING,
> L_SHIPMODE STRING,
> L_COMMENT STRING) partitioned by (dl
> int)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '|';
> CREATE TABLE part(
> p_partkey INT,
> p_name STRING,
> p_mfgr STRING,
> p_brand STRING,
> p_type STRING,
> p_size INT,
> p_container STRING,
> p_retailprice DOUBLE,
> p_comment STRING
> );
> explain select count(1) from part,supplier,lineitem where p_partkey =
> l_partkey and s_suppkey = l_suppkey;
> {code}
> Estimating stats will prevent join ordering algorithm to bail out and come up
> with join at least better than cross join
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-9543) MetaException(message:Metastore contains multiple versions)
[
https://issues.apache.org/jira/browse/HIVE-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
guo_tian_qi reassigned HIVE-9543:
-
Assignee: guo_tian_qi
> MetaException(message:Metastore contains multiple versions)
> ---
>
> Key: HIVE-9543
> URL: https://issues.apache.org/jira/browse/HIVE-9543
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.13.1
>Reporter: Junyong Li
>Assignee: guo_tian_qi
>
> When i run bin/hive command, i got the following exception:
> {noformat}
> Logging initialized using configuration in
> jar:file:/home/hadoop/apache-hive-0.13.1-bin/lib/hive-common-0.13.1.jar!/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
> ... 7 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
> ... 12 more
> Caused by: MetaException(message:Metastore contains multiple versions)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6368)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6330)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6289)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6277)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy9.verifySchema(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:476)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:356)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:54)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:171)
> ... 17 more
> {noformat}
> And i have found two record in metastore table VERSION. after reading source
> code, i found following code maybe cause the problem:
> In the org.apache.hadoop.hive.metastore.ObjectStore.java:6289:
> {noformat}
> String schemaVer = getMetaStoreSchemaVersion();
> if (schemaVer == null) {
> // metastore has no schema version information
> if
[jira] [Commented] (HIVE-15767) Hive On Spark is not working on secure clusters from Oozie
[
https://issues.apache.org/jira/browse/HIVE-15767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117956#comment-16117956
]
Peter Cseh commented on HIVE-15767:
---
I don't remember all the details, but here is a longer stack trace:{code}
java.lang.RuntimeException: java.io.IOException: Exception reading
file:/yarn/nm/usercache/yshi/appcache/application_1485271416004_0001/container_1485271416004_0001_01_02/container_tokens
at
org.apache.hadoop.mapreduce.security.TokenCache.mergeBinaryTokens(TokenCache.java:160)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:138)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:243)
{code}
Spark does not refer the {{mapreduce.job.credentials.binary}} directly, it is
hard-coded in Hadoop's TokenCache
[here|https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java#L148].
I think this TokenCache is used by Hadoop's FileSystem implementation too so
when Spark talks to HDFS it does through this class.
> Hive On Spark is not working on secure clusters from Oozie
> --
>
> Key: HIVE-15767
> URL: https://issues.apache.org/jira/browse/HIVE-15767
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.2.1, 2.1.1
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: HIVE-15767-001.patch, HIVE-15767-002.patch,
> HIVE-15767.1.patch
>
>
> When a HiveAction is launched form Oozie with Hive On Spark enabled, we're
> getting errors:
> {noformat}
> Caused by: java.io.IOException: Exception reading
> file:/yarn/nm/usercache/yshi/appcache/application_1485271416004_0022/container_1485271416004_0022_01_02/container_tokens
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:188)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.mergeBinaryTokens(TokenCache.java:155)
> {noformat}
> This is caused by passing the {{mapreduce.job.credentials.binary}} property
> to the Spark configuration in RemoteHiveSparkClient.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization (Part 1)
[
https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117928#comment-16117928
]
Hive QA commented on HIVE-17235:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880774/HIVE-17235.06.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10995 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.serde2.io.TestHiveDecimalWritableVersion.testVerifyHiveDecimalWritablePublicMethodsAndFieldsVersions
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6290/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6290/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6290/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880774 - PreCommit-HIVE-Build
> Add ORC Decimal64 Serialization/Deserialization (Part 1)
>
>
> Key: HIVE-17235
> URL: https://issues.apache.org/jira/browse/HIVE-17235
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch,
> HIVE-17235.05.patch, HIVE-17235.06.patch, HIVE-17235.patch
>
>
> The storage-api changes for ORC-209.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16895) Multi-threaded execution of bootstrap dump of partitions
[ https://issues.apache.org/jira/browse/HIVE-16895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117910#comment-16117910 ] Lefty Leverenz commented on HIVE-16895: --- Doc note: This adds *hive.repl.partitions.dump.parallelism* to HiveConf.java, so it needs to be documented in the wiki. * [Configuration Properties -- Replication | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Replication] Added a TODOC3.0 label. > Multi-threaded execution of bootstrap dump of partitions > - > > Key: HIVE-16895 > URL: https://issues.apache.org/jira/browse/HIVE-16895 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-16895.1.patch, HIVE-16895.2.patch > > > to allow faster execution of bootstrap dump phase we dump multiple partitions > from same table simultaneously. > even though dumping functions is not going to be a blocker, moving to > similar execution modes for all metastore objects will make code more > coherent. > Bootstrap dump at db level does : > * boostrap of all tables > ** boostrap of all partitions in a table. (scope of current jira) > * boostrap of all functions -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16895) Multi-threaded execution of bootstrap dump of partitions
[ https://issues.apache.org/jira/browse/HIVE-16895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-16895: -- Labels: TODOC3.0 (was: ) > Multi-threaded execution of bootstrap dump of partitions > - > > Key: HIVE-16895 > URL: https://issues.apache.org/jira/browse/HIVE-16895 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-16895.1.patch, HIVE-16895.2.patch > > > to allow faster execution of bootstrap dump phase we dump multiple partitions > from same table simultaneously. > even though dumping functions is not going to be a blocker, moving to > similar execution modes for all metastore objects will make code more > coherent. > Bootstrap dump at db level does : > * boostrap of all tables > ** boostrap of all partitions in a table. (scope of current jira) > * boostrap of all functions -- This message was sent by Atlassian JIRA (v6.4.14#64029)
