[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999318#comment-15999318 ] zhihai xu commented on HIVE-16456: -- thanks [~xuefuz] for the review, I replied your comment on RB. > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch, HIVE-16456.001.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16275) Vectorization: Add ReduceSink support for TopN (in specialized native classes)
[ https://issues.apache.org/jira/browse/HIVE-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16275: Status: In Progress (was: Patch Available) > Vectorization: Add ReduceSink support for TopN (in specialized native classes) > -- > > Key: HIVE-16275 > URL: https://issues.apache.org/jira/browse/HIVE-16275 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-16275.01.patch, HIVE-16275.02.patch, > HIVE-16275.03.patch, HIVE-16275.04.patch, HIVE-16275.05.patch, > HIVE-16275.06.patch, HIVE-16275.07.patch, HIVE-16275.08.patch > > > Currently, we don't specialize vectorization of ReduceSink when Top N is > planned. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16275) Vectorization: Add ReduceSink support for TopN (in specialized native classes)
[ https://issues.apache.org/jira/browse/HIVE-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16275: Attachment: HIVE-16275.08.patch > Vectorization: Add ReduceSink support for TopN (in specialized native classes) > -- > > Key: HIVE-16275 > URL: https://issues.apache.org/jira/browse/HIVE-16275 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-16275.01.patch, HIVE-16275.02.patch, > HIVE-16275.03.patch, HIVE-16275.04.patch, HIVE-16275.05.patch, > HIVE-16275.06.patch, HIVE-16275.07.patch, HIVE-16275.08.patch > > > Currently, we don't specialize vectorization of ReduceSink when Top N is > planned. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16557) Vectorization: Specialize ReduceSink empty key case
[ https://issues.apache.org/jira/browse/HIVE-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16557: Status: Patch Available (was: In Progress) > Vectorization: Specialize ReduceSink empty key case > --- > > Key: HIVE-16557 > URL: https://issues.apache.org/jira/browse/HIVE-16557 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16557.01.patch, HIVE-16557.02.patch, > HIVE-16557.03.patch, HIVE-16557.04.patch, HIVE-16557.05.patch, > HIVE-16557.06.patch > > > Gopal pointed out that native Vectorization of ReduceSink is missing the > empty key case. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16557) Vectorization: Specialize ReduceSink empty key case
[ https://issues.apache.org/jira/browse/HIVE-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16557: Attachment: HIVE-16557.06.patch > Vectorization: Specialize ReduceSink empty key case > --- > > Key: HIVE-16557 > URL: https://issues.apache.org/jira/browse/HIVE-16557 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16557.01.patch, HIVE-16557.02.patch, > HIVE-16557.03.patch, HIVE-16557.04.patch, HIVE-16557.05.patch, > HIVE-16557.06.patch > > > Gopal pointed out that native Vectorization of ReduceSink is missing the > empty key case. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-13583) E061-14: Search Conditions
[
https://issues.apache.org/jira/browse/HIVE-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999313#comment-15999313
]
Hive QA commented on HIVE-13583:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866578/HIVE-13583.2.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10653 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5067/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5067/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5067/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866578 - PreCommit-HIVE-Build
> E061-14: Search Conditions
> --
>
> Key: HIVE-13583
> URL: https://issues.apache.org/jira/browse/HIVE-13583
> Project: Hive
> Issue Type: Sub-task
>Reporter: Carter Shanklin
>Assignee: Zoltan Haindrich
> Attachments: HIVE-13583.1.patch, HIVE-13583.2.patch
>
>
> This is a part of the SQL:2011 Analytics Complete Umbrella JIRA HIVE-13554.
> Support for various forms of search conditions are mandatory in the SQL
> standard. For example, " is not true;" Hive should support those
> forms mandated by the standard.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16557) Vectorization: Specialize ReduceSink empty key case
[ https://issues.apache.org/jira/browse/HIVE-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16557: Status: In Progress (was: Patch Available) > Vectorization: Specialize ReduceSink empty key case > --- > > Key: HIVE-16557 > URL: https://issues.apache.org/jira/browse/HIVE-16557 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16557.01.patch, HIVE-16557.02.patch, > HIVE-16557.03.patch, HIVE-16557.04.patch, HIVE-16557.05.patch > > > Gopal pointed out that native Vectorization of ReduceSink is missing the > empty key case. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16541) PTF: Avoid shuffling constant keys for empty OVER()
[
https://issues.apache.org/jira/browse/HIVE-16541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999295#comment-15999295
]
Hive QA commented on HIVE-16541:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866566/HIVE-16541.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10652 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[outer_reference_windowed]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin1]
(batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5066/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5066/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5066/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866566 - PreCommit-HIVE-Build
> PTF: Avoid shuffling constant keys for empty OVER()
> ---
>
> Key: HIVE-16541
> URL: https://issues.apache.org/jira/browse/HIVE-16541
> Project: Hive
> Issue Type: Bug
> Components: PTF-Windowing
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-16541.1.patch, HIVE-16541.2.patch
>
>
> Generating surrogate keys with
> {code}
> select row_number() over() as p_key, * from table;
> {code}
> uses a sorted edge with "0 ASC NULLS FIRST" as the sort order.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-1010) Implement INFORMATION_SCHEMA in Hive
[
https://issues.apache.org/jira/browse/HIVE-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999260#comment-15999260
]
Hive QA commented on HIVE-1010:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866561/HIVE-1010.14.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10652 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5065/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5065/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5065/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866561 - PreCommit-HIVE-Build
> Implement INFORMATION_SCHEMA in Hive
>
>
> Key: HIVE-1010
> URL: https://issues.apache.org/jira/browse/HIVE-1010
> Project: Hive
> Issue Type: New Feature
> Components: Metastore, Query Processor, Server Infrastructure
>Reporter: Jeff Hammerbacher
>Assignee: Gunther Hagleitner
> Attachments: HIVE-1010.10.patch, HIVE-1010.11.patch,
> HIVE-1010.12.patch, HIVE-1010.13.patch, HIVE-1010.14.patch,
> HIVE-1010.7.patch, HIVE-1010.8.patch, HIVE-1010.9.patch
>
>
> INFORMATION_SCHEMA is part of the SQL92 standard and would be useful to
> implement using our metastore.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16568) Support complex types in external LLAP InputFormat
[
https://issues.apache.org/jira/browse/HIVE-16568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999242#comment-15999242
]
Hive QA commented on HIVE-16568:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866591/HIVE-16568.3.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10653 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1]
(batchId=235)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_buckets]
(batchId=238)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5064/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5064/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5064/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866591 - PreCommit-HIVE-Build
> Support complex types in external LLAP InputFormat
> --
>
> Key: HIVE-16568
> URL: https://issues.apache.org/jira/browse/HIVE-16568
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-16568.1.patch, HIVE-16568.2.patch,
> HIVE-16568.3.patch
>
>
> Currently just supports primitive types
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16590) Make initializing dag names in SparkWork thread safe for parallel compilation (HIVE-13512)
[
https://issues.apache.org/jira/browse/HIVE-16590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999229#comment-15999229
]
Hive QA commented on HIVE-16590:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866510/HIVE-16590.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 10653 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5063/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5063/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5063/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866510 - PreCommit-HIVE-Build
> Make initializing dag names in SparkWork thread safe for parallel compilation
> (HIVE-13512)
> --
>
> Key: HIVE-16590
> URL: https://issues.apache.org/jira/browse/HIVE-16590
> Project: Hive
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16590.1.patch
>
>
> In HIVE-13512 some modifications to {{TezWork}} were made to avoid generating
> duplicate dag ids during parallel compilation. We should do the equivalent
> for {{SparkWork}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16582) HashTableLoader should log info about the input, rows, size etc.
[
https://issues.apache.org/jira/browse/HIVE-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999214#comment-15999214
]
Hive QA commented on HIVE-16582:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866509/HIVE-16582.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10652 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5059/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5059/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5059/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866509 - PreCommit-HIVE-Build
> HashTableLoader should log info about the input, rows, size etc.
>
>
> Key: HIVE-16582
> URL: https://issues.apache.org/jira/browse/HIVE-16582
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Attachments: HIVE-16582.1.patch, HIVE-16582.2.patch
>
>
> Will be useful to log the following info during hash table loading
> - input name
> - number of rows
> - estimated data size (LLAP tracks this)
> - object cache key
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16582) HashTableLoader should log info about the input, rows, size etc.
[ https://issues.apache.org/jira/browse/HIVE-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999188#comment-15999188 ] Sergey Shelukhin commented on HIVE-16582: - +1 w/the update the exception message to include all the data in case of error due to mapjoin going over size > HashTableLoader should log info about the input, rows, size etc. > > > Key: HIVE-16582 > URL: https://issues.apache.org/jira/browse/HIVE-16582 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > Attachments: HIVE-16582.1.patch, HIVE-16582.2.patch > > > Will be useful to log the following info during hash table loading > - input name > - number of rows > - estimated data size (LLAP tracks this) > - object cache key -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16556) Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES table
[
https://issues.apache.org/jira/browse/HIVE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999187#comment-15999187
]
Hive QA commented on HIVE-16556:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866502/HIVE-16556.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10654 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=218)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testConnection (batchId=236)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValid (batchId=236)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValidNeg (batchId=236)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeProxyAuth
(batchId=236)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth
(batchId=236)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testProxyAuth (batchId=236)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=236)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5058/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5058/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5058/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866502 - PreCommit-HIVE-Build
> Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES
> table
>
>
> Key: HIVE-16556
> URL: https://issues.apache.org/jira/browse/HIVE-16556
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16556.01.patch, HIVE-16556.02.patch,
> HIVE-16556.03.patch, HIVE-16556.04.patch, HIVE-16556.05.patch
>
>
> sub-task to modify schema tool and its related changes so that the new table
> is added to the schema when schematool initializes or upgrades the schema.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16601) Display Session Id, Query Name / Id, and Dag Id in Spark UI
[
https://issues.apache.org/jira/browse/HIVE-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16601:
Attachment: HIVE-16601.1.patch
> Display Session Id, Query Name / Id, and Dag Id in Spark UI
> ---
>
> Key: HIVE-16601
> URL: https://issues.apache.org/jira/browse/HIVE-16601
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16601.1.patch, Spark UI Applications List.png,
> Spark UI Jobs List.png
>
>
> We should display the session id for each HoS Application Launched, and the
> Query Name / Id and Dag Id for each Spark job launched. Hive-on-MR does
> something similar via the {{mapred.job.name}} parameter. The query name is
> displayed in the Job Name of the MR app.
> The changes here should also allow us to leverage the config
> {{hive.query.name}} for HoS.
> This should help with debuggability of HoS applications. The Hive-on-Tez UI
> does something similar.
> Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16601) Display Session Id, Query Name / Id, and Dag Id in Spark UI
[
https://issues.apache.org/jira/browse/HIVE-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16601:
Status: Patch Available (was: Open)
> Display Session Id, Query Name / Id, and Dag Id in Spark UI
> ---
>
> Key: HIVE-16601
> URL: https://issues.apache.org/jira/browse/HIVE-16601
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16601.1.patch, Spark UI Applications List.png,
> Spark UI Jobs List.png
>
>
> We should display the session id for each HoS Application Launched, and the
> Query Name / Id and Dag Id for each Spark job launched. Hive-on-MR does
> something similar via the {{mapred.job.name}} parameter. The query name is
> displayed in the Job Name of the MR app.
> The changes here should also allow us to leverage the config
> {{hive.query.name}} for HoS.
> This should help with debuggability of HoS applications. The Hive-on-Tez UI
> does something similar.
> Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16601) Display Session Id, Query Name / Id, and Dag Id in Spark UI
[
https://issues.apache.org/jira/browse/HIVE-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16601:
Description:
We should display the session id for each HoS Application Launched, and the
Query Name / Id and Dag Id for each Spark job launched. Hive-on-MR does
something similar via the {{mapred.job.name}} parameter. The query name is
displayed in the Job Name of the MR app.
The changes here should also allow us to leverage the config
{{hive.query.name}} for HoS.
This should help with debuggability of HoS applications. The Hive-on-Tez UI
does something similar.
Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523
was:
We should display the session id for each HoS Application Launched, and the
Query Name / Id and Dag Id for each Spark job launched.
This should help with debuggability of HoS applications. The Hive-on-Tez UI
does something similar.
Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523
> Display Session Id, Query Name / Id, and Dag Id in Spark UI
> ---
>
> Key: HIVE-16601
> URL: https://issues.apache.org/jira/browse/HIVE-16601
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: Spark UI Applications List.png, Spark UI Jobs List.png
>
>
> We should display the session id for each HoS Application Launched, and the
> Query Name / Id and Dag Id for each Spark job launched. Hive-on-MR does
> something similar via the {{mapred.job.name}} parameter. The query name is
> displayed in the Job Name of the MR app.
> The changes here should also allow us to leverage the config
> {{hive.query.name}} for HoS.
> This should help with debuggability of HoS applications. The Hive-on-Tez UI
> does something similar.
> Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16598) LlapServiceDriver - create directories and warn of errors
[ https://issues.apache.org/jira/browse/HIVE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999155#comment-15999155 ] Prasanth Jayachandran commented on HIVE-16598: -- +1 > LlapServiceDriver - create directories and warn of errors > - > > Key: HIVE-16598 > URL: https://issues.apache.org/jira/browse/HIVE-16598 > Project: Hive > Issue Type: Bug >Reporter: Kavan Suresh >Assignee: Sergey Shelukhin > Attachments: HIVE-16598.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16285) Servlet for dynamically configuring log levels
[ https://issues.apache.org/jira/browse/HIVE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-16285: - Attachment: HIVE-16285.6.patch Rebase + some missing out changes. > Servlet for dynamically configuring log levels > -- > > Key: HIVE-16285 > URL: https://issues.apache.org/jira/browse/HIVE-16285 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16285.1.patch, HIVE-16285.2.patch, > HIVE-16285.3.patch, HIVE-16285.4.patch, HIVE-16285.5.patch, > HIVE-16285.5.patch, HIVE-16285.6.patch > > > Many long running services like HS2, LLAP etc. will benefit from having an > endpoint to dynamically change log levels for various loggers. This will help > greatly with debuggability without requiring a restart of the service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16601) Display Session Id, Query Name / Id, and Dag Id in Spark UI
[ https://issues.apache.org/jira/browse/HIVE-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-16601: Attachment: Spark UI Jobs List.png Spark UI Applications List.png Attaching two screenshots of what the Spark UI looks like with these patches. > Display Session Id, Query Name / Id, and Dag Id in Spark UI > --- > > Key: HIVE-16601 > URL: https://issues.apache.org/jira/browse/HIVE-16601 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: Spark UI Applications List.png, Spark UI Jobs List.png > > > We should display the session id for each HoS Application Launched, and the > Query Name / Id and Dag Id for each Spark job launched. > This should help with debuggability of HoS applications. The Hive-on-Tez UI > does something similar. > Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16601) Display Session Id, Query Name / Id, and Dag Id in Spark UI
[ https://issues.apache.org/jira/browse/HIVE-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-16601: --- > Display Session Id, Query Name / Id, and Dag Id in Spark UI > --- > > Key: HIVE-16601 > URL: https://issues.apache.org/jira/browse/HIVE-16601 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar > > We should display the session id for each HoS Application Launched, and the > Query Name / Id and Dag Id for each Spark job launched. > This should help with debuggability of HoS applications. The Hive-on-Tez UI > does something similar. > Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16268) enable incremental repl dump to handle functions metadata
[ https://issues.apache.org/jira/browse/HIVE-16268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan reassigned HIVE-16268: --- Assignee: anishek (was: Sushanth Sowmyan) > enable incremental repl dump to handle functions metadata > - > > Key: HIVE-16268 > URL: https://issues.apache.org/jira/browse/HIVE-16268 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.2.0 >Reporter: anishek >Assignee: anishek > Fix For: 3.0.0 > > Attachments: HIVE-16268.1.patch, HIVE-16268.2.patch, > HIVE-16268.3.patch, HIVE-16268.4.patch, HIVE-16268.5.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > this is created separately to ensure that any other metadata related to > replication which comes from replication spec, if they are needed as part of > the function dump output when doing incremental update. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16268) enable incremental repl dump to handle functions metadata
[ https://issues.apache.org/jira/browse/HIVE-16268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-16268: Attachment: HIVE-16268.5.patch Attaching .5.patch identical to .4.patch so it'll be picked up by ptest and not ignored. > enable incremental repl dump to handle functions metadata > - > > Key: HIVE-16268 > URL: https://issues.apache.org/jira/browse/HIVE-16268 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.2.0 >Reporter: anishek >Assignee: Sushanth Sowmyan > Fix For: 3.0.0 > > Attachments: HIVE-16268.1.patch, HIVE-16268.2.patch, > HIVE-16268.3.patch, HIVE-16268.4.patch, HIVE-16268.5.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > this is created separately to ensure that any other metadata related to > replication which comes from replication spec, if they are needed as part of > the function dump output when doing incremental update. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16268) enable incremental repl dump to handle functions metadata
[ https://issues.apache.org/jira/browse/HIVE-16268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-16268: Status: Patch Available (was: Open) > enable incremental repl dump to handle functions metadata > - > > Key: HIVE-16268 > URL: https://issues.apache.org/jira/browse/HIVE-16268 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.2.0 >Reporter: anishek >Assignee: Sushanth Sowmyan > Fix For: 3.0.0 > > Attachments: HIVE-16268.1.patch, HIVE-16268.2.patch, > HIVE-16268.3.patch, HIVE-16268.4.patch, HIVE-16268.5.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > this is created separately to ensure that any other metadata related to > replication which comes from replication spec, if they are needed as part of > the function dump output when doing incremental update. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16588) Ressource leak by druid http client
[
https://issues.apache.org/jira/browse/HIVE-16588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999131#comment-15999131
]
Hive QA commented on HIVE-16588:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866503/16588.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5057/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5057/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5057/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-05-05 23:39:37.093
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-5057/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-05-05 23:39:37.096
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at d09f3f8 HIVE-16577: Syntax error in the metastore init scripts
for mssql (Vihang Karajgaonkar, reviewed by Aihua Xu & Thejas M Nair)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at d09f3f8 HIVE-16577: Syntax error in the metastore init scripts
for mssql (Vihang Karajgaonkar, reviewed by Aihua Xu & Thejas M Nair)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-05-05 23:39:37.609
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:56
error:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:
patch does not apply
error: patch failed:
druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidQueryBasedInputFormat.java:216
error:
druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidQueryBasedInputFormat.java:
patch does not apply
error: patch failed:
druid-handler/src/test/org/apache/hadoop/hive/druid/TestDruidStorageHandler.java:86
error:
druid-handler/src/test/org/apache/hadoop/hive/druid/TestDruidStorageHandler.java:
patch does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866503 - PreCommit-HIVE-Build
> Ressource leak by druid http client
> ---
>
> Key: HIVE-16588
> URL: https://issues.apache.org/jira/browse/HIVE-16588
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Nishant Bangarwa
> Labels: druid
> Fix For: 3.0.0
>
> Attachments: 16588.patch
>
>
> Current implementation of druid storage handler does leak some resources if
> the creation of the http client fails due to too many files exception.
> The reason this is leaking is the fact the cleaning hook is registered after
> the client starts.
> In order to fix this will extract the creation of the HTTP client to become
> static and reusable instead of per query creation.
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16268) enable incremental repl dump to handle functions metadata
[ https://issues.apache.org/jira/browse/HIVE-16268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-16268: Status: Open (was: Patch Available) > enable incremental repl dump to handle functions metadata > - > > Key: HIVE-16268 > URL: https://issues.apache.org/jira/browse/HIVE-16268 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.2.0 >Reporter: anishek >Assignee: anishek > Fix For: 3.0.0 > > Attachments: HIVE-16268.1.patch, HIVE-16268.2.patch, > HIVE-16268.3.patch, HIVE-16268.4.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > this is created separately to ensure that any other metadata related to > replication which comes from replication spec, if they are needed as part of > the function dump output when doing incremental update. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16578) Semijoin Hints should use column name, if provided for partition key check
[ https://issues.apache.org/jira/browse/HIVE-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-16578: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master > Semijoin Hints should use column name, if provided for partition key check > -- > > Key: HIVE-16578 > URL: https://issues.apache.org/jira/browse/HIVE-16578 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Fix For: 3.0.0 > > Attachments: HIVE-16578.1.patch, HIVE-16578.2.patch, > HIVE-16578.3.patch, HIVE-16578.4.patch, HIVE-16578.5.patch > > > Current logic does not verify the column name provided in the hint against > the column on which the runtime filtering branch will originate from. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16268) enable incremental repl dump to handle functions metadata
[ https://issues.apache.org/jira/browse/HIVE-16268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan reassigned HIVE-16268: --- Assignee: Sushanth Sowmyan (was: anishek) > enable incremental repl dump to handle functions metadata > - > > Key: HIVE-16268 > URL: https://issues.apache.org/jira/browse/HIVE-16268 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.2.0 >Reporter: anishek >Assignee: Sushanth Sowmyan > Fix For: 3.0.0 > > Attachments: HIVE-16268.1.patch, HIVE-16268.2.patch, > HIVE-16268.3.patch, HIVE-16268.4.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > this is created separately to ensure that any other metadata related to > replication which comes from replication spec, if they are needed as part of > the function dump output when doing incremental update. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16588) Ressource leak by druid http client
[
https://issues.apache.org/jira/browse/HIVE-16588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999127#comment-15999127
]
Hive QA commented on HIVE-16588:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866503/16588.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5056/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5056/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5056/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-05-05 23:39:01.275
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-5056/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-05-05 23:39:01.278
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at d09f3f8 HIVE-16577: Syntax error in the metastore init scripts
for mssql (Vihang Karajgaonkar, reviewed by Aihua Xu & Thejas M Nair)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at d09f3f8 HIVE-16577: Syntax error in the metastore init scripts
for mssql (Vihang Karajgaonkar, reviewed by Aihua Xu & Thejas M Nair)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-05-05 23:39:06.662
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:56
error:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java:
patch does not apply
error: patch failed:
druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidQueryBasedInputFormat.java:216
error:
druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidQueryBasedInputFormat.java:
patch does not apply
error: patch failed:
druid-handler/src/test/org/apache/hadoop/hive/druid/TestDruidStorageHandler.java:86
error:
druid-handler/src/test/org/apache/hadoop/hive/druid/TestDruidStorageHandler.java:
patch does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866503 - PreCommit-HIVE-Build
> Ressource leak by druid http client
> ---
>
> Key: HIVE-16588
> URL: https://issues.apache.org/jira/browse/HIVE-16588
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Nishant Bangarwa
> Labels: druid
> Fix For: 3.0.0
>
> Attachments: 16588.patch
>
>
> Current implementation of druid storage handler does leak some resources if
> the creation of the http client fails due to too many files exception.
> The reason this is leaking is the fact the cleaning hook is registered after
> the client starts.
> In order to fix this will extract the creation of the HTTP client to become
> static and reusable instead of per query creation.
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16343) LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
[ https://issues.apache.org/jira/browse/HIVE-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999132#comment-15999132 ] Prasanth Jayachandran commented on HIVE-16343: -- Removed all other options to find PID. Using JVM_PID approach suggested by [~sseth]. Wrt. SMAP discussed with [~rajesh.balamohan], looks like it won't be in the order of multiple seconds. Also NM metrics dumps meminfo every few seconds (LLAP dump interval is even slower). > LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring > > > Key: HIVE-16343 > URL: https://issues.apache.org/jira/browse/HIVE-16343 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16343.1.patch, HIVE-16343.2.patch > > > Publish MemInfo from ProcfsBasedProcessTree to llap metrics. This will useful > for monitoring and also setting up triggers via JMC. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16578) Semijoin Hints should use column name, if provided for partition key check
[
https://issues.apache.org/jira/browse/HIVE-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999122#comment-15999122
]
Hive QA commented on HIVE-16578:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866485/HIVE-16578.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10652 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=155)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5055/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5055/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5055/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866485 - PreCommit-HIVE-Build
> Semijoin Hints should use column name, if provided for partition key check
> --
>
> Key: HIVE-16578
> URL: https://issues.apache.org/jira/browse/HIVE-16578
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-16578.1.patch, HIVE-16578.2.patch,
> HIVE-16578.3.patch, HIVE-16578.4.patch, HIVE-16578.5.patch
>
>
> Current logic does not verify the column name provided in the hint against
> the column on which the runtime filtering branch will originate from.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16343) LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
[ https://issues.apache.org/jira/browse/HIVE-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-16343: - Attachment: HIVE-16343.2.patch > LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring > > > Key: HIVE-16343 > URL: https://issues.apache.org/jira/browse/HIVE-16343 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16343.1.patch, HIVE-16343.2.patch > > > Publish MemInfo from ProcfsBasedProcessTree to llap metrics. This will useful > for monitoring and also setting up triggers via JMC. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16579) CachedStore: improvements to partition col stats caching and cache column stats for unpartitioned table
[ https://issues.apache.org/jira/browse/HIVE-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-16579: Summary: CachedStore: improvements to partition col stats caching and cache column stats for unpartitioned table (was: CachedStore: improvements to partition col stats caching) > CachedStore: improvements to partition col stats caching and cache column > stats for unpartitioned table > --- > > Key: HIVE-16579 > URL: https://issues.apache.org/jira/browse/HIVE-16579 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-16579.1.patch > > > 1. Update stats cache when partitions/table is dropped. > 2. Update cached partition col stats in the background cache update thread. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HIVE-16580) CachedStore: Cache column stats for unpartitioned tables
[ https://issues.apache.org/jira/browse/HIVE-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta resolved HIVE-16580. - Resolution: Duplicate Including changes in HIVE-16579 > CachedStore: Cache column stats for unpartitioned tables > > > Key: HIVE-16580 > URL: https://issues.apache.org/jira/browse/HIVE-16580 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16539) Add PTF tests for blobstores
[ https://issues.apache.org/jira/browse/HIVE-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Rodríguez Hortalá updated HIVE-16539: -- Attachment: HIVE-16539.3.patch > Add PTF tests for blobstores > > > Key: HIVE-16539 > URL: https://issues.apache.org/jira/browse/HIVE-16539 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Juan Rodríguez Hortalá >Assignee: Juan Rodríguez Hortalá > Fix For: 2.2.0 > > Attachments: HIVE-16539.2.patch, HIVE-16539.3.patch, HIVE-16539.patch > > > This patch covers 6 tests that check the integration of several Partitioned > Table Functions with blobstores -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16600) Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel order by in multi_insert cases
[ https://issues.apache.org/jira/browse/HIVE-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated HIVE-16600: Status: Patch Available (was: Open) > Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel > order by in multi_insert cases > > > Key: HIVE-16600 > URL: https://issues.apache.org/jira/browse/HIVE-16600 > Project: Hive > Issue Type: Sub-task >Reporter: liyunzhang_intel >Assignee: liyunzhang_intel > Attachments: HIVE-16600.patch > > > in multi_insert cases multi_insert_gby2.q, the parallelism of SORT operator > is 1 even we set "hive.optimize.sampling.orderby" = true. This is because > the logic of SetSparkReducerParallelism#needSetParallelism does not support > this case. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16600) Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel order by in multi_insert cases
[ https://issues.apache.org/jira/browse/HIVE-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated HIVE-16600: Attachment: HIVE-16600.patch > Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel > order by in multi_insert cases > > > Key: HIVE-16600 > URL: https://issues.apache.org/jira/browse/HIVE-16600 > Project: Hive > Issue Type: Sub-task >Reporter: liyunzhang_intel >Assignee: liyunzhang_intel > Attachments: HIVE-16600.patch > > > in multi_insert cases multi_insert_gby2.q, the parallelism of SORT operator > is 1 even we set "hive.optimize.sampling.orderby" = true. This is because > the logic of SetSparkReducerParallelism#needSetParallelism does not support > this case. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16578) Semijoin Hints should use column name, if provided for partition key check
[
https://issues.apache.org/jira/browse/HIVE-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999069#comment-15999069
]
Hive QA commented on HIVE-16578:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866485/HIVE-16578.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10652 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5054/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5054/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5054/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866485 - PreCommit-HIVE-Build
> Semijoin Hints should use column name, if provided for partition key check
> --
>
> Key: HIVE-16578
> URL: https://issues.apache.org/jira/browse/HIVE-16578
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-16578.1.patch, HIVE-16578.2.patch,
> HIVE-16578.3.patch, HIVE-16578.4.patch, HIVE-16578.5.patch
>
>
> Current logic does not verify the column name provided in the hint against
> the column on which the runtime filtering branch will originate from.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[
https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999003#comment-15999003
]
Hive QA commented on HIVE-16456:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866483/HIVE-16456.001.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10652 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
(batchId=148)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5053/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5053/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5053/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866483 - PreCommit-HIVE-Build
> Kill spark job when InterruptedException happens or driverContext.isShutdown
> is true.
> -
>
> Key: HIVE-16456
> URL: https://issues.apache.org/jira/browse/HIVE-16456
> Project: Hive
> Issue Type: Improvement
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16456.000.patch, HIVE-16456.001.patch
>
>
> Kill spark job when InterruptedException happens or driverContext.isShutdown
> is true. If the InterruptedException happened in RemoteSparkJobMonitor and
> LocalSparkJobMonitor, it will be better to kill the job. Also there is a race
> condition between submit the spark job and query/operation cancellation, it
> will be better to check driverContext.isShutdown right after submit the spark
> job. This will guarantee the job being killed no matter when shutdown is
> called. It is similar as HIVE-15997.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16600) Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel order by in multi_insert cases
[ https://issues.apache.org/jira/browse/HIVE-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel reassigned HIVE-16600: --- Assignee: liyunzhang_intel > Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel > order by in multi_insert cases > > > Key: HIVE-16600 > URL: https://issues.apache.org/jira/browse/HIVE-16600 > Project: Hive > Issue Type: Sub-task >Reporter: liyunzhang_intel >Assignee: liyunzhang_intel > > in multi_insert cases multi_insert_gby2.q, the parallelism of SORT operator > is 1 even we set "hive.optimize.sampling.orderby" = true. This is because > the logic of SetSparkReducerParallelism#needSetParallelism does not support > this case. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15997) Resource leaks when query is cancelled
[
https://issues.apache.org/jira/browse/HIVE-15997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998961#comment-15998961
]
Yongzhi Chen commented on HIVE-15997:
-
This interrupt() in our code can cause InterruptedException when the code is
doing(or is scheduled to do) hdfs file operations or zookeeper lock operations.
For we do not have many long single file operations or long single lock
operations, but we do have many fast operations, the performance of cancel
operation will not affect by adding checkpoints . I found that the thread
interrupt() has no effect for some running operations: for example I tried to
interrupt HMS client who is waiting for a response of a long running API (for
example ListPartitions), the interrupt can not stop the waiting at all. And the
interrupt has some "delay effect", it causes InterruptedException later (for
example when the cleanup folder operations happen.) So we should not put the
Thread.currentThread().interrupt() in the heavily used method isInterrupted().
if, in the future, we find the place the interrupt() is really needed, we can
just add the code there.
> Resource leaks when query is cancelled
> ---
>
> Key: HIVE-15997
> URL: https://issues.apache.org/jira/browse/HIVE-15997
> Project: Hive
> Issue Type: Bug
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Fix For: 2.2.0
>
> Attachments: HIVE-15997.1.patch
>
>
> There may some resource leaks when query is cancelled.
> We see following stacks in the log:
> Possible files and folder leak:
> {noformat}
> 2017-02-02 06:23:25,410 WARN hive.ql.Context: [HiveServer2-Background-Pool:
> Thread-61]: Error Removing Scratch: java.io.IOException: Failed on local
> exception: java.nio.channels.ClosedByInterruptException; Host Details : local
> host is: "ychencdh511t-1.vpc.cloudera.com/172.26.11.50"; destination host is:
> "ychencdh511t-1.vpc.cloudera.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> at org.apache.hadoop.ipc.Client.call(Client.java:1476)
> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>
> at com.sun.proxy.$Proxy25.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
>
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
>
> at com.sun.proxy.$Proxy26.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
>
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671)
>
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671)
>
> at org.apache.hadoop.hive.ql.Context.removeScratchDir(Context.java:405)
> at org.apache.hadoop.hive.ql.Context.clear(Context.java:541)
> at org.apache.hadoop.hive.ql.Driver.releaseContext(Driver.java:2109)
> at org.apache.hadoop.hive.ql.Driver.closeInProcess(Driver.java:2150)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1472)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1212)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
[jira] [Updated] (HIVE-16586) Fix Unit test failures when CachedStore is enabled
[ https://issues.apache.org/jira/browse/HIVE-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-16586: -- Attachment: HIVE-16586.2.patch Another trial with stats disabled (stats issue will be addressed in HIVE-16579/HIVE-16580). > Fix Unit test failures when CachedStore is enabled > -- > > Key: HIVE-16586 > URL: https://issues.apache.org/jira/browse/HIVE-16586 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: enable_cachedstore.patch, HIVE-16586.1.patch, > HIVE-16586.2.patch > > > Though we don't plan to turn on CachedStore by default, we want to make sure > unit tests pass with CachedStore. I turn on CachedStore in the patch in order > to run unit tests with it, but I will turn off CachedStore when committing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16582) HashTableLoader should log info about the input, rows, size etc.
[
https://issues.apache.org/jira/browse/HIVE-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998935#comment-15998935
]
Hive QA commented on HIVE-16582:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866509/HIVE-16582.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10652 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1]
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5052/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5052/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5052/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866509 - PreCommit-HIVE-Build
> HashTableLoader should log info about the input, rows, size etc.
>
>
> Key: HIVE-16582
> URL: https://issues.apache.org/jira/browse/HIVE-16582
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Attachments: HIVE-16582.1.patch, HIVE-16582.2.patch
>
>
> Will be useful to log the following info during hash table loading
> - input name
> - number of rows
> - estimated data size (LLAP tracks this)
> - object cache key
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16599) NPE in runtime filtering cost when handling SMB Joins
[ https://issues.apache.org/jira/browse/HIVE-16599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal reassigned HIVE-16599: - > NPE in runtime filtering cost when handling SMB Joins > - > > Key: HIVE-16599 > URL: https://issues.apache.org/jira/browse/HIVE-16599 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > > A test with SMB joins failed with NPE in runtime filtering costing logic. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16456) Kill spark job when InterruptedException happens or driverContext.isShutdown is true.
[ https://issues.apache.org/jira/browse/HIVE-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998910#comment-15998910 ] Xuefu Zhang commented on HIVE-16456: [~zxu], I had one comment on RB. Thanks. > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. > - > > Key: HIVE-16456 > URL: https://issues.apache.org/jira/browse/HIVE-16456 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16456.000.patch, HIVE-16456.001.patch > > > Kill spark job when InterruptedException happens or driverContext.isShutdown > is true. If the InterruptedException happened in RemoteSparkJobMonitor and > LocalSparkJobMonitor, it will be better to kill the job. Also there is a race > condition between submit the spark job and query/operation cancellation, it > will be better to check driverContext.isShutdown right after submit the spark > job. This will guarantee the job being killed no matter when shutdown is > called. It is similar as HIVE-15997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-10120) Disallow create table with dot/colon in column name
[
https://issues.apache.org/jira/browse/HIVE-10120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998902#comment-15998902
]
Dudu Markovitz commented on HIVE-10120:
---
[~leftylev]] - It seems the documentation was not updated. Would you like me to
take care of that?
> Disallow create table with dot/colon in column name
> ---
>
> Key: HIVE-10120
> URL: https://issues.apache.org/jira/browse/HIVE-10120
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Labels: TODOC1.2
> Fix For: 1.2.0
>
> Attachments: HIVE-10120.01.patch, HIVE-10120.02.patch
>
>
> Since we don't allow users to query column names with dot in the middle such
> as emp.no, don't allow users to create tables with such columns that cannot
> be queried. Fix the documentation to reflect this fix.
> Here is an example. Consider this table:
> {code}
> CREATE TABLE a (`emp.no` string);
> select `emp.no` from a; fails with this message:
> FAILED: RuntimeException java.lang.RuntimeException: cannot find field emp
> from [0:emp.no]
> {code}
> The hive documentation needs to be fixed:
> {code}
> (https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL) seems
> to indicate that any Unicode character can go between the backticks in the
> select statement, but it doesn’t like the dot/colon or even select * when
> there is a column that has a dot/colon.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HIVE-10120) Disallow create table with dot/colon in column name
[
https://issues.apache.org/jira/browse/HIVE-10120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998902#comment-15998902
]
Dudu Markovitz edited comment on HIVE-10120 at 5/5/17 8:20 PM:
---
[~leftylev] - It seems the documentation was not updated. Would you like me to
take care of that?
was (Author: dmarkovitz):
[~leftylev]] - It seems the documentation was not updated. Would you like me to
take care of that?
> Disallow create table with dot/colon in column name
> ---
>
> Key: HIVE-10120
> URL: https://issues.apache.org/jira/browse/HIVE-10120
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Labels: TODOC1.2
> Fix For: 1.2.0
>
> Attachments: HIVE-10120.01.patch, HIVE-10120.02.patch
>
>
> Since we don't allow users to query column names with dot in the middle such
> as emp.no, don't allow users to create tables with such columns that cannot
> be queried. Fix the documentation to reflect this fix.
> Here is an example. Consider this table:
> {code}
> CREATE TABLE a (`emp.no` string);
> select `emp.no` from a; fails with this message:
> FAILED: RuntimeException java.lang.RuntimeException: cannot find field emp
> from [0:emp.no]
> {code}
> The hive documentation needs to be fixed:
> {code}
> (https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL) seems
> to indicate that any Unicode character can go between the backticks in the
> select statement, but it doesn’t like the dot/colon or even select * when
> there is a column that has a dot/colon.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16556) Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES table
[ https://issues.apache.org/jira/browse/HIVE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998896#comment-15998896 ] Vihang Karajgaonkar commented on HIVE-16556: Manually tested against all the databases. The patch is working as expected. Hi [~ngangam] Can you please take a look again? Thanks! > Modify schematool scripts to initialize and create METASTORE_DB_PROPERTIES > table > > > Key: HIVE-16556 > URL: https://issues.apache.org/jira/browse/HIVE-16556 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-16556.01.patch, HIVE-16556.02.patch, > HIVE-16556.03.patch, HIVE-16556.04.patch, HIVE-16556.05.patch > > > sub-task to modify schema tool and its related changes so that the new table > is added to the schema when schematool initializes or upgrades the schema. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16579) CachedStore: improvements to partition col stats caching
[ https://issues.apache.org/jira/browse/HIVE-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998883#comment-15998883 ] Vaibhav Gumashta commented on HIVE-16579: - Need to update the patch. Will submit a v2 in a bit. > CachedStore: improvements to partition col stats caching > > > Key: HIVE-16579 > URL: https://issues.apache.org/jira/browse/HIVE-16579 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-16579.1.patch > > > 1. Update stats cache when partitions/table is dropped. > 2. Update cached partition col stats in the background cache update thread. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16598) LlapServiceDriver - create directories and warn of errors
[ https://issues.apache.org/jira/browse/HIVE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16598: Attachment: HIVE-16598.patch [~prasanth_j] can you take a look? thanks > LlapServiceDriver - create directories and warn of errors > - > > Key: HIVE-16598 > URL: https://issues.apache.org/jira/browse/HIVE-16598 > Project: Hive > Issue Type: Bug >Reporter: Kavan Suresh >Assignee: Sergey Shelukhin > Attachments: HIVE-16598.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16598) LlapServiceDriver - create directories and warn of errors
[ https://issues.apache.org/jira/browse/HIVE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16598: Status: Patch Available (was: Open) > LlapServiceDriver - create directories and warn of errors > - > > Key: HIVE-16598 > URL: https://issues.apache.org/jira/browse/HIVE-16598 > Project: Hive > Issue Type: Bug >Reporter: Kavan Suresh >Assignee: Sergey Shelukhin > Attachments: HIVE-16598.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16598) LlapServiceDriver - create directories and warn of errors
[ https://issues.apache.org/jira/browse/HIVE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-16598: --- > LlapServiceDriver - create directories and warn of errors > - > > Key: HIVE-16598 > URL: https://issues.apache.org/jira/browse/HIVE-16598 > Project: Hive > Issue Type: Bug >Reporter: Kavan Suresh >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998876#comment-15998876
]
Hive QA commented on HIVE-13652:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866633/HIVE-13652.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 10653 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5051/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5051/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5051/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866633 - PreCommit-HIVE-Build
> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
> Issue Type: Bug
> Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Attachments: HIVE-13652.01.patch,
> ReplLoad_PartitionOrder_AfterFix_Log.png,
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using
> "export table" command is imported (using "import table") such a way that
> order of partitions is changed to day, month, year.
> Export DB: Hive 0.14
> Import DB: Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998870#comment-15998870
]
Thejas M Nair commented on HIVE-13652:
--
Created HIVE-16597 to track larger cleanup around datastructure for partSpec.
> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
> Issue Type: Bug
> Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Attachments: HIVE-13652.01.patch,
> ReplLoad_PartitionOrder_AfterFix_Log.png,
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using
> "export table" command is imported (using "import table") such a way that
> order of partitions is changed to day, month, year.
> Export DB: Hive 0.14
> Import DB: Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16597) Replace use of Map for partSpec with List>
[ https://issues.apache.org/jira/browse/HIVE-16597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998869#comment-15998869 ] Thejas M Nair commented on HIVE-16597: -- I am not working on this right now, so folks can free to take on this. > Replace use of Map for partSpec with List String>> > --- > > Key: HIVE-16597 > URL: https://issues.apache.org/jira/browse/HIVE-16597 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair > > As discussed in [HIVE-13652 comment > |https://issues.apache.org/jira/browse/HIVE-13652?focusedCommentId=15998857&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15998857] > the use of Map for partSpec in AddPartitionDesc makes it > vulnerable to similar mistakes like what happened with issue in HIVE-13652. > We should cleanup the code to use List> . -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15997) Resource leaks when query is cancelled
[
https://issues.apache.org/jira/browse/HIVE-15997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998861#comment-15998861
]
Xuefu Zhang commented on HIVE-15997:
Thanks for the explanation, [~ychena]. One more question: which part of the
code was handling the InterruptedException that
Thread.currentThread().interrupt() throws? I was wondering if there is any
side-effect after the thrower is removed. Thanks.
> Resource leaks when query is cancelled
> ---
>
> Key: HIVE-15997
> URL: https://issues.apache.org/jira/browse/HIVE-15997
> Project: Hive
> Issue Type: Bug
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Fix For: 2.2.0
>
> Attachments: HIVE-15997.1.patch
>
>
> There may some resource leaks when query is cancelled.
> We see following stacks in the log:
> Possible files and folder leak:
> {noformat}
> 2017-02-02 06:23:25,410 WARN hive.ql.Context: [HiveServer2-Background-Pool:
> Thread-61]: Error Removing Scratch: java.io.IOException: Failed on local
> exception: java.nio.channels.ClosedByInterruptException; Host Details : local
> host is: "ychencdh511t-1.vpc.cloudera.com/172.26.11.50"; destination host is:
> "ychencdh511t-1.vpc.cloudera.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> at org.apache.hadoop.ipc.Client.call(Client.java:1476)
> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>
> at com.sun.proxy.$Proxy25.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
>
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
>
> at com.sun.proxy.$Proxy26.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
>
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671)
>
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671)
>
> at org.apache.hadoop.hive.ql.Context.removeScratchDir(Context.java:405)
> at org.apache.hadoop.hive.ql.Context.clear(Context.java:541)
> at org.apache.hadoop.hive.ql.Driver.releaseContext(Driver.java:2109)
> at org.apache.hadoop.hive.ql.Driver.closeInProcess(Driver.java:2150)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1472)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1212)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:681)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
>
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
> at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:714)
> at org.apache.hadoop.ipc.Client$Connection.access$2900(
[jira] [Commented] (HIVE-16047) Shouldn't try to get KeyProvider unless encryption is enabled
[
https://issues.apache.org/jira/browse/HIVE-16047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998856#comment-15998856
]
Sergio Peña commented on HIVE-16047:
Thanks [~eddyxu].
> Shouldn't try to get KeyProvider unless encryption is enabled
> -
>
> Key: HIVE-16047
> URL: https://issues.apache.org/jira/browse/HIVE-16047
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-16047.1.patch, HIVE-16047.2.patch
>
>
> Found lots of following errors in HS2 log:
> {noformat}
> hdfs.KeyProviderCache: Could not find uri with key
> [dfs.encryption.key.provider.uri] to create a keyProvider !!
> {noformat}
> Similar to HDFS-7931
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998857#comment-15998857
]
Thejas M Nair commented on HIVE-13652:
--
[~luky]
LinkedHashMap has iteration order that is same as the insertion order, so this
would work. So the ordering is defined (it uses a doubly-linked list to
remember the order).
However, I agree that the use of Map for partSpec object in the
AddPartitionDesc methods makes it vulnerable to such bugs again in future. We
should do a cleanup to replace use of Map with something like
List> which would have prevented this bug in first place.
I will create a follow up bug for that.
> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
> Issue Type: Bug
> Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Attachments: HIVE-13652.01.patch,
> ReplLoad_PartitionOrder_AfterFix_Log.png,
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using
> "export table" command is imported (using "import table") such a way that
> order of partitions is changed to day, month, year.
> Export DB: Hive 0.14
> Import DB: Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16539) Add PTF tests for blobstores
[
https://issues.apache.org/jira/browse/HIVE-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998792#comment-15998792
]
Hive QA commented on HIVE-16539:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866466/HIVE-16539.2.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10658 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_orcfile]
(batchId=238)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5050/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5050/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5050/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866466 - PreCommit-HIVE-Build
> Add PTF tests for blobstores
>
>
> Key: HIVE-16539
> URL: https://issues.apache.org/jira/browse/HIVE-16539
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: Juan Rodríguez Hortalá
>Assignee: Juan Rodríguez Hortalá
> Fix For: 2.2.0
>
> Attachments: HIVE-16539.2.patch, HIVE-16539.patch
>
>
> This patch covers 6 tests that check the integration of several Partitioned
> Table Functions with blobstores
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16577) Syntax error in the metastore init scripts for mssql
[
https://issues.apache.org/jira/browse/HIVE-16577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998786#comment-15998786
]
Sushanth Sowmyan commented on HIVE-16577:
-
Thanks for the catch!
> Syntax error in the metastore init scripts for mssql
>
>
> Key: HIVE-16577
> URL: https://issues.apache.org/jira/browse/HIVE-16577
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.2.0, 2.3.0, 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Blocker
> Fix For: 2.3.0, 3.0.0
>
> Attachments: HIVE-16577.01.patch
>
>
> HIVE-10562 introduced a new column to {{NOTIFICATION_LOG}} table. The mssql
> init scripts which were modified have a syntax error and they fail to
> initialize metastore schema from 2.2.0 onwards.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16596) CrossProductCheck failed to detect cross product between two unions
[
https://issues.apache.org/jira/browse/HIVE-16596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang reassigned HIVE-16596:
---
> CrossProductCheck failed to detect cross product between two unions
> ---
>
> Key: HIVE-16596
> URL: https://issues.apache.org/jira/browse/HIVE-16596
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
>
> To reproduce:
> {code}
> create table f (a int, b string);
> set hive.auto.convert.join=false;
> explain select * from (select * from f union all select * from f) a join
> (select * from f union all select * from f) b;
> {code}
> No cross product warning is given.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16485) Enable outputName for RS operator in explain formatted
[ https://issues.apache.org/jira/browse/HIVE-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16485: --- Status: Patch Available (was: Open) > Enable outputName for RS operator in explain formatted > -- > > Key: HIVE-16485 > URL: https://issues.apache.org/jira/browse/HIVE-16485 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16485.01.patch, HIVE-16485.02.patch, > HIVE-16485.03.patch, HIVE-16485.04.patch, HIVE-16485-disableMasking, plan, > query > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16485) Enable outputName for RS operator in explain formatted
[ https://issues.apache.org/jira/browse/HIVE-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16485: --- Attachment: HIVE-16485.04.patch > Enable outputName for RS operator in explain formatted > -- > > Key: HIVE-16485 > URL: https://issues.apache.org/jira/browse/HIVE-16485 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16485.01.patch, HIVE-16485.02.patch, > HIVE-16485.03.patch, HIVE-16485.04.patch, HIVE-16485-disableMasking, plan, > query > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16485) Enable outputName for RS operator in explain formatted
[ https://issues.apache.org/jira/browse/HIVE-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16485: --- Status: Open (was: Patch Available) > Enable outputName for RS operator in explain formatted > -- > > Key: HIVE-16485 > URL: https://issues.apache.org/jira/browse/HIVE-16485 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16485.01.patch, HIVE-16485.02.patch, > HIVE-16485.03.patch, HIVE-16485.04.patch, HIVE-16485-disableMasking, plan, > query > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16485) Enable outputName for RS operator in explain formatted
[ https://issues.apache.org/jira/browse/HIVE-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16485: --- Attachment: (was: HIVE-16485.04.patch) > Enable outputName for RS operator in explain formatted > -- > > Key: HIVE-16485 > URL: https://issues.apache.org/jira/browse/HIVE-16485 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16485.01.patch, HIVE-16485.02.patch, > HIVE-16485.03.patch, HIVE-16485-disableMasking, plan, query > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16562) Issues with nullif / fetch task
[
https://issues.apache.org/jira/browse/HIVE-16562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998709#comment-15998709
]
Ashutosh Chauhan commented on HIVE-16562:
-
+1 pending tests
> Issues with nullif / fetch task
> ---
>
> Key: HIVE-16562
> URL: https://issues.apache.org/jira/browse/HIVE-16562
> Project: Hive
> Issue Type: Bug
>Reporter: Carter Shanklin
>Assignee: Zoltan Haindrich
> Attachments: HIVE-16562.1.patch, HIVE-16562.2.patch
>
>
> HIVE-13555 adds support for nullif. I'm encountering issues with nullif on
> master (3.0.0-SNAPSHOT rdac3786d86462e4d08d62d23115e6b7a3e534f5d)
> Cluster side jobs work fine but client side don't.
> Consider these two tables:
> e011_02:
> Columns c1 = float, c2 = double
> 1.0 1.0
> 1.5 1.5
> 2.0 2.0
> test:
> Columns c1 = int, c2 = int
> Data:
> 1 1
> 2 2
> And this query:
> select nullif(c1, c2) from e011_02;
> With e011_02 I get:
> {code}
> java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: Error
> evaluating NULLIF(c1,c2)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:165)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2177)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating
> NULLIF(c1,c2)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:93)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:442)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:434)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
> ... 13 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.lazy.LazyFloat cannot be cast to
> org.apache.hadoop.io.FloatWritable
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector.get(WritableFloatObjectInspector.java:36)
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.comparePrimitiveObjects(PrimitiveObjectInspectorUtils.java:412)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFNullif.evaluate(GenericUDFNullif.java:93)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
> ... 18 more
> {code}
> With
> select nullif(c1, c2) from test;
> I get:
> {code}
> 2017-05-01T03:32:19,905 ERROR [cbaf5380-5b06-4531-aeb9-524c62314a46 main]
> CliDriver: Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: Error
> evaluating NULLIF(c1,c2)
> java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: Error
> evaluating NULLIF(c1,c2)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:165)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2177)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
> at sun.reflect.NativeMethodAccessorI
[jira] [Commented] (HIVE-16577) Syntax error in the metastore init scripts for mssql
[
https://issues.apache.org/jira/browse/HIVE-16577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998696#comment-15998696
]
Vihang Karajgaonkar commented on HIVE-16577:
Thanks [~aihuaxu] Updated the fixed version to 2.3.0
> Syntax error in the metastore init scripts for mssql
>
>
> Key: HIVE-16577
> URL: https://issues.apache.org/jira/browse/HIVE-16577
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.2.0, 2.3.0, 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Blocker
> Fix For: 2.3.0, 3.0.0
>
> Attachments: HIVE-16577.01.patch
>
>
> HIVE-10562 introduced a new column to {{NOTIFICATION_LOG}} table. The mssql
> init scripts which were modified have a syntax error and they fail to
> initialize metastore schema from 2.2.0 onwards.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16577) Syntax error in the metastore init scripts for mssql
[
https://issues.apache.org/jira/browse/HIVE-16577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-16577:
---
Fix Version/s: (was: 2.4.0)
2.3.0
> Syntax error in the metastore init scripts for mssql
>
>
> Key: HIVE-16577
> URL: https://issues.apache.org/jira/browse/HIVE-16577
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.2.0, 2.3.0, 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Blocker
> Fix For: 2.3.0, 3.0.0
>
> Attachments: HIVE-16577.01.patch
>
>
> HIVE-10562 introduced a new column to {{NOTIFICATION_LOG}} table. The mssql
> init scripts which were modified have a syntax error and they fail to
> initialize metastore schema from 2.2.0 onwards.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16586) Fix Unit test failures when CachedStore is enabled
[
https://issues.apache.org/jira/browse/HIVE-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998692#comment-15998692
]
Hive QA commented on HIVE-16586:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866455/enable_cachedstore.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 69 failed/errored test(s), 10652 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[select_dummy_source]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_3]
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_change_col]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_cascade]
(batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_part]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_1]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_2]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative3]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_join0]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_partlvl]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[describe_table]
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynamic_partition_skip_default]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_logical]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[extrapolate_part_stats_date]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[extrapolate_part_stats_full]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[extrapolate_part_stats_partial]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join26] (batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join28] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join32] (batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join33] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join45] (batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_parse] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part14]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part15]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_mapjoin]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_subquery]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_date2]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_timestamp2]
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[repl_1_drop] (batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[repl_2_exim_basic]
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[repl_3_exim_metadata]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_create_table_db_table]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_partitions]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_ppr_all]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[tez_join_hash]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[truncate_table]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[tunable_ndv] (batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union36] (batchId=83)
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[url_hook]
(batchId=229)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnStatsUpdateForStatsOptimizer_1]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[escape1]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[escape2]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join32_lessSize]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part1]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part2]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
(batchId=143)
org.apache.hadoo
[jira] [Updated] (HIVE-16577) Syntax error in the metastore init scripts for mssql
[
https://issues.apache.org/jira/browse/HIVE-16577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-16577:
Resolution: Fixed
Fix Version/s: 2.4.0
3.0.0
Status: Resolved (was: Patch Available)
Pushed to branch-2 and master. Thanks Vihang for the work and Thejas to review.
> Syntax error in the metastore init scripts for mssql
>
>
> Key: HIVE-16577
> URL: https://issues.apache.org/jira/browse/HIVE-16577
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.2.0, 2.3.0, 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Blocker
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-16577.01.patch
>
>
> HIVE-10562 introduced a new column to {{NOTIFICATION_LOG}} table. The mssql
> init scripts which were modified have a syntax error and they fail to
> initialize metastore schema from 2.2.0 onwards.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16568) Support complex types in external LLAP InputFormat
[ https://issues.apache.org/jira/browse/HIVE-16568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998615#comment-15998615 ] Prasanth Jayachandran commented on HIVE-16568: -- +1 > Support complex types in external LLAP InputFormat > -- > > Key: HIVE-16568 > URL: https://issues.apache.org/jira/browse/HIVE-16568 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-16568.1.patch, HIVE-16568.2.patch, > HIVE-16568.3.patch > > > Currently just supports primitive types -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16539) Add PTF tests for blobstores
[
https://issues.apache.org/jira/browse/HIVE-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998590#comment-15998590
]
Hive QA commented on HIVE-16539:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866466/HIVE-16539.2.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10658 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_orcfile]
(batchId=238)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5048/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5048/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5048/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866466 - PreCommit-HIVE-Build
> Add PTF tests for blobstores
>
>
> Key: HIVE-16539
> URL: https://issues.apache.org/jira/browse/HIVE-16539
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: Juan Rodríguez Hortalá
>Assignee: Juan Rodríguez Hortalá
> Fix For: 2.2.0
>
> Attachments: HIVE-16539.2.patch, HIVE-16539.patch
>
>
> This patch covers 6 tests that check the integration of several Partitioned
> Table Functions with blobstores
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16389) Allow HookContext to access SQLOperationDisplay
[
https://issues.apache.org/jira/browse/HIVE-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16389:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Allow HookContext to access SQLOperationDisplay
> ---
>
> Key: HIVE-16389
> URL: https://issues.apache.org/jira/browse/HIVE-16389
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Fix For: 3.0.0
>
> Attachments: HIVE-16389.1.patch, HIVE-16389.2.patch,
> HIVE-16389.3.patch, HIVE-16389.4.patch, HIVE-16389.5.patch
>
>
> There is a lot of useful information in {{SQLOperationDisplay}} that users of
> Hive Hooks may be interested in.
> We should allow Hive Hooks to access this info by adding the
> {{SQLOperationDisplay}} to {{HookContext}}.
> This will allow hooks to have access to all information available in the HS2
> Web UI.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16389) Allow HookContext to access SQLOperationDisplay
[
https://issues.apache.org/jira/browse/HIVE-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998587#comment-15998587
]
Sahil Takiar commented on HIVE-16389:
-
Thanks [~spena], we probably can just stick to master. Will close this.
> Allow HookContext to access SQLOperationDisplay
> ---
>
> Key: HIVE-16389
> URL: https://issues.apache.org/jira/browse/HIVE-16389
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Fix For: 3.0.0
>
> Attachments: HIVE-16389.1.patch, HIVE-16389.2.patch,
> HIVE-16389.3.patch, HIVE-16389.4.patch, HIVE-16389.5.patch
>
>
> There is a lot of useful information in {{SQLOperationDisplay}} that users of
> Hive Hooks may be interested in.
> We should allow Hive Hooks to access this info by adding the
> {{SQLOperationDisplay}} to {{HookContext}}.
> This will allow hooks to have access to all information available in the HS2
> Web UI.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-13583) E061-14: Search Conditions
[ https://issues.apache.org/jira/browse/HIVE-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998585#comment-15998585 ] Ashutosh Chauhan commented on HIVE-13583: - +1 pending tests > E061-14: Search Conditions > -- > > Key: HIVE-13583 > URL: https://issues.apache.org/jira/browse/HIVE-13583 > Project: Hive > Issue Type: Sub-task >Reporter: Carter Shanklin >Assignee: Zoltan Haindrich > Attachments: HIVE-13583.1.patch, HIVE-13583.2.patch > > > This is a part of the SQL:2011 Analytics Complete Umbrella JIRA HIVE-13554. > Support for various forms of search conditions are mandatory in the SQL > standard. For example, " is not true;" Hive should support those > forms mandated by the standard. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16530) Improve execution logs for REPL commands
[
https://issues.apache.org/jira/browse/HIVE-16530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998516#comment-15998516
]
Hive QA commented on HIVE-16530:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866620/HIVE-16530.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10652 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_decimal]
(batchId=9)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5047/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5047/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5047/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866620 - PreCommit-HIVE-Build
> Improve execution logs for REPL commands
>
>
> Key: HIVE-16530
> URL: https://issues.apache.org/jira/browse/HIVE-16530
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Affects Versions: 2.2.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Attachments: Bootstrap_ReplDump_Console_Log.png,
> Bootstrap_ReplLoad_Console_Log.png, HIVE-16530.01.patch, HIVE-16530.02.patch,
> Incremental_ReplDump_Console_Log.png, Incremental_ReplLoad_Console_Log.png
>
>
> This is the log format that is being proposed for Hive Repl query logs
> For bootstrap case:
> Hive will log a message for each object as it is being bootstrapped and it
> will be in the following sequence
> - Tables first (views are tables for this purpose) at time including
> partitions (depth first), followed by functions, constraints
> - The ordering is based on the ordering of listStatus API of HDFS
> - For each object, a message at the beginning of the replication will be
> logged
> - Every partition bootstrapped will be followed by a message saying the
> number of partitions bootstrapped so far (for the table) and the partition
> name
> - And a message at the end of bootstrap of an object
> Incremental case:
> - We will have DB Name, event id and event type will be part of the log
> header (for debugging/troubleshooting)
> - We will have information of current event ID and total number of events to
> replicate for every event replicated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-8838) Support Parquet through HCatalog
[ https://issues.apache.org/jira/browse/HIVE-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998512#comment-15998512 ] Aihua Xu commented on HIVE-8838: The patch looks good to me. [~spena] Can you also take a look since you are more familiar with parquet? > Support Parquet through HCatalog > > > Key: HIVE-8838 > URL: https://issues.apache.org/jira/browse/HIVE-8838 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Assignee: Adam Szita > Attachments: HIVE-8838.0.patch, HIVE-8838.1.patch > > > Similar to HIVE-8687 for Avro we need to fix Parquet with HCatalog. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998489#comment-15998489
]
Lukas Waldmann commented on HIVE-13652:
---
I just wonder, if by using LinkedHashMap we don't introduce just another
vulnerability there as order of items in the HashMap is not defined and if ever
hashing algo change than again we may get completely different order?
> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
> Issue Type: Bug
> Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Attachments: HIVE-13652.01.patch,
> ReplLoad_PartitionOrder_AfterFix_Log.png,
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using
> "export table" command is imported (using "import table") such a way that
> order of partitions is changed to day, month, year.
> Export DB: Hive 0.14
> Import DB: Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-13652) Import table change order of dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-13652:
Status: Patch Available (was: Open)
> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
> Issue Type: Bug
> Components: Import/Export, repl
>Affects Versions: 1.2.1, 1.2.0
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Attachments: HIVE-13652.01.patch,
> ReplLoad_PartitionOrder_AfterFix_Log.png,
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using
> "export table" command is imported (using "import table") such a way that
> order of partitions is changed to day, month, year.
> Export DB: Hive 0.14
> Import DB: Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998464#comment-15998464
]
Sankar Hariappan commented on HIVE-13652:
-
Updated patch for additional verification of partition location.
[~thejas], Please review the patch.
> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
> Issue Type: Bug
> Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Attachments: HIVE-13652.01.patch,
> ReplLoad_PartitionOrder_AfterFix_Log.png,
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using
> "export table" command is imported (using "import table") such a way that
> order of partitions is changed to day, month, year.
> Export DB: Hive 0.14
> Import DB: Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-13652) Import table change order of dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-13652:
Attachment: HIVE-13652.01.patch
> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
> Issue Type: Bug
> Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Attachments: HIVE-13652.01.patch,
> ReplLoad_PartitionOrder_AfterFix_Log.png,
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using
> "export table" command is imported (using "import table") such a way that
> order of partitions is changed to day, month, year.
> Export DB: Hive 0.14
> Import DB: Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Work stopped] (HIVE-13652) Import table change order of dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13652 stopped by Sankar Hariappan.
---
> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
> Issue Type: Bug
> Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Attachments: ReplLoad_PartitionOrder_AfterFix_Log.png,
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using
> "export table" command is imported (using "import table") such a way that
> order of partitions is changed to day, month, year.
> Export DB: Hive 0.14
> Import DB: Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-13652) Import table change order of dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-13652:
Attachment: (was: HIVE-13652.01.patch)
> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
> Issue Type: Bug
> Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Attachments: ReplLoad_PartitionOrder_AfterFix_Log.png,
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using
> "export table" command is imported (using "import table") such a way that
> order of partitions is changed to day, month, year.
> Export DB: Hive 0.14
> Import DB: Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15104) Hive on Spark generate more shuffle data than hive on mr
[ https://issues.apache.org/jira/browse/HIVE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998437#comment-15998437 ] Rui Li commented on HIVE-15104: --- Tried disabling relocation locally. It does solve the AbstractMethodError. However, seems Spark still needs the hashCode on reducer side. Will dig more ... > Hive on Spark generate more shuffle data than hive on mr > > > Key: HIVE-15104 > URL: https://issues.apache.org/jira/browse/HIVE-15104 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.2.1 >Reporter: wangwenli >Assignee: Rui Li > > the same sql, running on spark and mr engine, will generate different size > of shuffle data. > i think it is because of hive on mr just serialize part of HiveKey, but hive > on spark which using kryo will serialize full of Hivekey object. > what is your opionion? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16389) Allow HookContext to access SQLOperationDisplay
[
https://issues.apache.org/jira/browse/HIVE-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998429#comment-15998429
]
Sergio Peña commented on HIVE-16389:
[~stakiar] I committed to master now. Do you want this patch to branch-2 as
well? If so, could you attach a new one to run tests on branch-2? If not, then
you can close it.
> Allow HookContext to access SQLOperationDisplay
> ---
>
> Key: HIVE-16389
> URL: https://issues.apache.org/jira/browse/HIVE-16389
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Fix For: 3.0.0
>
> Attachments: HIVE-16389.1.patch, HIVE-16389.2.patch,
> HIVE-16389.3.patch, HIVE-16389.4.patch, HIVE-16389.5.patch
>
>
> There is a lot of useful information in {{SQLOperationDisplay}} that users of
> Hive Hooks may be interested in.
> We should allow Hive Hooks to access this info by adding the
> {{SQLOperationDisplay}} to {{HookContext}}.
> This will allow hooks to have access to all information available in the HS2
> Web UI.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16389) Allow HookContext to access SQLOperationDisplay
[
https://issues.apache.org/jira/browse/HIVE-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-16389:
---
Fix Version/s: 3.0.0
> Allow HookContext to access SQLOperationDisplay
> ---
>
> Key: HIVE-16389
> URL: https://issues.apache.org/jira/browse/HIVE-16389
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Fix For: 3.0.0
>
> Attachments: HIVE-16389.1.patch, HIVE-16389.2.patch,
> HIVE-16389.3.patch, HIVE-16389.4.patch, HIVE-16389.5.patch
>
>
> There is a lot of useful information in {{SQLOperationDisplay}} that users of
> Hive Hooks may be interested in.
> We should allow Hive Hooks to access this info by adding the
> {{SQLOperationDisplay}} to {{HookContext}}.
> This will allow hooks to have access to all information available in the HS2
> Web UI.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15834) Add unit tests for org.json usage on master
[ https://issues.apache.org/jira/browse/HIVE-15834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Voros updated HIVE-15834: Fix Version/s: (was: 2.2.0) Status: Patch Available (was: Open) > Add unit tests for org.json usage on master > --- > > Key: HIVE-15834 > URL: https://issues.apache.org/jira/browse/HIVE-15834 > Project: Hive > Issue Type: Sub-task >Reporter: Daniel Voros >Assignee: Daniel Voros > Attachments: HIVE-15834.1.patch > > > Before switching implementation, we should add some tests that capture the > current behavior. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15834) Add unit tests for org.json usage on master
[ https://issues.apache.org/jira/browse/HIVE-15834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Voros updated HIVE-15834: Attachment: HIVE-15834.1.patch Attaching patch #1. This adds test to all non-trivial usages of the org.json library. This is the port of HIVE-15833 with a few additions to cover new paths. > Add unit tests for org.json usage on master > --- > > Key: HIVE-15834 > URL: https://issues.apache.org/jira/browse/HIVE-15834 > Project: Hive > Issue Type: Sub-task >Reporter: Daniel Voros >Assignee: Daniel Voros > Fix For: 2.2.0 > > Attachments: HIVE-15834.1.patch > > > Before switching implementation, we should add some tests that capture the > current behavior. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16450) Some metastore operations are not retried even with desired underlining exceptions
[
https://issues.apache.org/jira/browse/HIVE-16450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-16450:
Status: Patch Available (was: In Progress)
> Some metastore operations are not retried even with desired underlining
> exceptions
> --
>
> Key: HIVE-16450
> URL: https://issues.apache.org/jira/browse/HIVE-16450
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-16450.1.patch, HIVE-16450.2.patch,
> HIVE-16450.3.patch, HIVE-16450.4.patch
>
>
> In RetryingHMSHandler class, we are expecting the operations should retry
> when the cause of MetaException is JDOException or NucleusException.
> {noformat}
> if (e.getCause() instanceof MetaException && e.getCause().getCause()
> != null) {
> if (e.getCause().getCause() instanceof javax.jdo.JDOException ||
> e.getCause().getCause() instanceof NucleusException) {
> // The JDOException or the Nucleus Exception may be wrapped
> further in a MetaException
> caughtException = e.getCause().getCause();
>}
> {noformat}
> While in ObjectStore, many places we are only throwing new MetaException(msg)
> without the cause, so we are missing retrying for some cases. e.g., with the
> following JDOException, we should retry but it's ignored.
> {noformat}
> 2017-04-04 17:28:21,602 ERROR metastore.ObjectStore
> (ObjectStore.java:getMTableColumnStatistics(6555)) - Error retrieving
> statistics via jdo
> javax.jdo.JDOException: Exception thrown when executing query
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
> at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:6546)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$1200(ObjectStore.java:171)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6606)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6595)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2633)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6594)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6588)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
> at com.sun.proxy.$Proxy0.getTableColumnStatistics(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTableUpdateTableColumnStats(HiveAlterHandler.java:787)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:247)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3809)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3779)
> at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy3.alter_table_with_environment_context(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9617)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9601)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGro
[jira] [Updated] (HIVE-16450) Some metastore operations are not retried even with desired underlining exceptions
[
https://issues.apache.org/jira/browse/HIVE-16450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-16450:
Attachment: HIVE-16450.4.patch
patch-4: 2 LLAP unit tests are failing because we didn't check the return List
is null (NullPointerException). Fixed the code.
> Some metastore operations are not retried even with desired underlining
> exceptions
> --
>
> Key: HIVE-16450
> URL: https://issues.apache.org/jira/browse/HIVE-16450
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-16450.1.patch, HIVE-16450.2.patch,
> HIVE-16450.3.patch, HIVE-16450.4.patch
>
>
> In RetryingHMSHandler class, we are expecting the operations should retry
> when the cause of MetaException is JDOException or NucleusException.
> {noformat}
> if (e.getCause() instanceof MetaException && e.getCause().getCause()
> != null) {
> if (e.getCause().getCause() instanceof javax.jdo.JDOException ||
> e.getCause().getCause() instanceof NucleusException) {
> // The JDOException or the Nucleus Exception may be wrapped
> further in a MetaException
> caughtException = e.getCause().getCause();
>}
> {noformat}
> While in ObjectStore, many places we are only throwing new MetaException(msg)
> without the cause, so we are missing retrying for some cases. e.g., with the
> following JDOException, we should retry but it's ignored.
> {noformat}
> 2017-04-04 17:28:21,602 ERROR metastore.ObjectStore
> (ObjectStore.java:getMTableColumnStatistics(6555)) - Error retrieving
> statistics via jdo
> javax.jdo.JDOException: Exception thrown when executing query
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
> at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:6546)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$1200(ObjectStore.java:171)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6606)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6595)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2633)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6594)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6588)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
> at com.sun.proxy.$Proxy0.getTableColumnStatistics(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTableUpdateTableColumnStats(HiveAlterHandler.java:787)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:247)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3809)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3779)
> at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy3.alter_table_with_environment_context(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9617)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9601)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Metho
[jira] [Updated] (HIVE-16450) Some metastore operations are not retried even with desired underlining exceptions
[
https://issues.apache.org/jira/browse/HIVE-16450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-16450:
Status: In Progress (was: Patch Available)
> Some metastore operations are not retried even with desired underlining
> exceptions
> --
>
> Key: HIVE-16450
> URL: https://issues.apache.org/jira/browse/HIVE-16450
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-16450.1.patch, HIVE-16450.2.patch,
> HIVE-16450.3.patch
>
>
> In RetryingHMSHandler class, we are expecting the operations should retry
> when the cause of MetaException is JDOException or NucleusException.
> {noformat}
> if (e.getCause() instanceof MetaException && e.getCause().getCause()
> != null) {
> if (e.getCause().getCause() instanceof javax.jdo.JDOException ||
> e.getCause().getCause() instanceof NucleusException) {
> // The JDOException or the Nucleus Exception may be wrapped
> further in a MetaException
> caughtException = e.getCause().getCause();
>}
> {noformat}
> While in ObjectStore, many places we are only throwing new MetaException(msg)
> without the cause, so we are missing retrying for some cases. e.g., with the
> following JDOException, we should retry but it's ignored.
> {noformat}
> 2017-04-04 17:28:21,602 ERROR metastore.ObjectStore
> (ObjectStore.java:getMTableColumnStatistics(6555)) - Error retrieving
> statistics via jdo
> javax.jdo.JDOException: Exception thrown when executing query
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
> at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:6546)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$1200(ObjectStore.java:171)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6606)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6595)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2633)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6594)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6588)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
> at com.sun.proxy.$Proxy0.getTableColumnStatistics(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTableUpdateTableColumnStats(HiveAlterHandler.java:787)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:247)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3809)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3779)
> at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy3.alter_table_with_environment_context(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9617)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9601)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(U
[jira] [Resolved] (HIVE-11064) ALTER TABLE CASCADE ERROR unbalanced calls to openTransaction/commitTransaction
[
https://issues.apache.org/jira/browse/HIVE-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang resolved HIVE-11064.
Resolution: Cannot Reproduce
> ALTER TABLE CASCADE ERROR unbalanced calls to
> openTransaction/commitTransaction
> ---
>
> Key: HIVE-11064
> URL: https://issues.apache.org/jira/browse/HIVE-11064
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
> Environment: CDH5.4.0
>Reporter: fatkun
>Assignee: Chaoyu Tang
>
> my hive version hive-1.1.0-cdh5.4.0
> follower this step, the exception throw
>
> use hive client
> {code}
> CREATE TABLE test1 (name string) PARTITIONED BY (pt string);
> ALTER TABLE test1 ADD PARTITION (pt='1');
> ALTER TABLE test1 CHANGE name name1 string;
> ALTER TABLE test1 CHANGE name1 name string cascade;
> {code}
> then throw exception,
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table.
> java.lang.RuntimeException: commitTransaction was called but
> openTransactionCalls = 0. This probably indicates that there are unbalanced
> calls to openTransaction/commitTransaction
>
> metasotre log
> {quote}
> MetaException(message:java.lang.RuntimeException: commitTransaction was
> called but openTransactionCalls = 0. This probably indicates that there are
> unbalanced calls to openTransaction/commitTransaction)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5257)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3338)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_cascade(HiveMetaStore.java:3290)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
> at com.sun.proxy.$Proxy5.alter_table_with_cascade(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_cascade.getResult(ThriftHiveMetastore.java:9131)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_cascade.getResult(ThriftHiveMetastore.java:9115)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: commitTransaction was called but
> openTransactionCalls = 0. This probably indicates that there are unbalanced
> calls to openTransaction/commitTransaction
> at
> org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:448)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
> at com.sun.proxy.$Proxy0.commitTransaction(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:242)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3318)
> ... 19 more
> {quote}
> I debug the code, may this function "private void
> updatePartColumnStatsForAlterColumns" wrong.some transaction rollback, but I
> don't known the exact error.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16530) Improve execution logs for REPL commands
[ https://issues.apache.org/jira/browse/HIVE-16530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16530: Status: Patch Available (was: Open) > Improve execution logs for REPL commands > > > Key: HIVE-16530 > URL: https://issues.apache.org/jira/browse/HIVE-16530 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.2.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Attachments: Bootstrap_ReplDump_Console_Log.png, > Bootstrap_ReplLoad_Console_Log.png, HIVE-16530.01.patch, HIVE-16530.02.patch, > Incremental_ReplDump_Console_Log.png, Incremental_ReplLoad_Console_Log.png > > > This is the log format that is being proposed for Hive Repl query logs > For bootstrap case: > Hive will log a message for each object as it is being bootstrapped and it > will be in the following sequence > - Tables first (views are tables for this purpose) at time including > partitions (depth first), followed by functions, constraints > - The ordering is based on the ordering of listStatus API of HDFS > - For each object, a message at the beginning of the replication will be > logged > - Every partition bootstrapped will be followed by a message saying the > number of partitions bootstrapped so far (for the table) and the partition > name > - And a message at the end of bootstrap of an object > Incremental case: > - We will have DB Name, event id and event type will be part of the log > header (for debugging/troubleshooting) > - We will have information of current event ID and total number of events to > replicate for every event replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16530) Improve execution logs for REPL commands
[ https://issues.apache.org/jira/browse/HIVE-16530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16530: Attachment: HIVE-16530.02.patch > Improve execution logs for REPL commands > > > Key: HIVE-16530 > URL: https://issues.apache.org/jira/browse/HIVE-16530 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.2.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Attachments: Bootstrap_ReplDump_Console_Log.png, > Bootstrap_ReplLoad_Console_Log.png, HIVE-16530.01.patch, HIVE-16530.02.patch, > Incremental_ReplDump_Console_Log.png, Incremental_ReplLoad_Console_Log.png > > > This is the log format that is being proposed for Hive Repl query logs > For bootstrap case: > Hive will log a message for each object as it is being bootstrapped and it > will be in the following sequence > - Tables first (views are tables for this purpose) at time including > partitions (depth first), followed by functions, constraints > - The ordering is based on the ordering of listStatus API of HDFS > - For each object, a message at the beginning of the replication will be > logged > - Every partition bootstrapped will be followed by a message saying the > number of partitions bootstrapped so far (for the table) and the partition > name > - And a message at the end of bootstrap of an object > Incremental case: > - We will have DB Name, event id and event type will be part of the log > header (for debugging/troubleshooting) > - We will have information of current event ID and total number of events to > replicate for every event replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16530) Improve execution logs for REPL commands
[ https://issues.apache.org/jira/browse/HIVE-16530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16530: Status: Open (was: Patch Available) > Improve execution logs for REPL commands > > > Key: HIVE-16530 > URL: https://issues.apache.org/jira/browse/HIVE-16530 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.2.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Attachments: Bootstrap_ReplDump_Console_Log.png, > Bootstrap_ReplLoad_Console_Log.png, HIVE-16530.01.patch, > Incremental_ReplDump_Console_Log.png, Incremental_ReplLoad_Console_Log.png > > > This is the log format that is being proposed for Hive Repl query logs > For bootstrap case: > Hive will log a message for each object as it is being bootstrapped and it > will be in the following sequence > - Tables first (views are tables for this purpose) at time including > partitions (depth first), followed by functions, constraints > - The ordering is based on the ordering of listStatus API of HDFS > - For each object, a message at the beginning of the replication will be > logged > - Every partition bootstrapped will be followed by a message saying the > number of partitions bootstrapped so far (for the table) and the partition > name > - And a message at the end of bootstrap of an object > Incremental case: > - We will have DB Name, event id and event type will be part of the log > header (for debugging/troubleshooting) > - We will have information of current event ID and total number of events to > replicate for every event replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16530) Improve execution logs for REPL commands
[ https://issues.apache.org/jira/browse/HIVE-16530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16530: Attachment: (was: HIVE-16530.02.patch) > Improve execution logs for REPL commands > > > Key: HIVE-16530 > URL: https://issues.apache.org/jira/browse/HIVE-16530 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.2.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Attachments: Bootstrap_ReplDump_Console_Log.png, > Bootstrap_ReplLoad_Console_Log.png, HIVE-16530.01.patch, > Incremental_ReplDump_Console_Log.png, Incremental_ReplLoad_Console_Log.png > > > This is the log format that is being proposed for Hive Repl query logs > For bootstrap case: > Hive will log a message for each object as it is being bootstrapped and it > will be in the following sequence > - Tables first (views are tables for this purpose) at time including > partitions (depth first), followed by functions, constraints > - The ordering is based on the ordering of listStatus API of HDFS > - For each object, a message at the beginning of the replication will be > logged > - Every partition bootstrapped will be followed by a message saying the > number of partitions bootstrapped so far (for the table) and the partition > name > - And a message at the end of bootstrap of an object > Incremental case: > - We will have DB Name, event id and event type will be part of the log > header (for debugging/troubleshooting) > - We will have information of current event ID and total number of events to > replicate for every event replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16584) Warning messages should use LogHelper.printInfo instead of printing to the infoStream directly
[ https://issues.apache.org/jira/browse/HIVE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16584: -- Attachment: HIVE-16584.02.patch Same as before, just removed unused imports. > Warning messages should use LogHelper.printInfo instead of printing to the > infoStream directly > -- > > Key: HIVE-16584 > URL: https://issues.apache.org/jira/browse/HIVE-16584 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16584.02.patch, HIVE-16584.patch > > > There are several cases when warning messages are printed to the console > outputstream directly. These warnings do not show up in the BeeLine output. > We should use the the printInfo method instead. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15104) Hive on Spark generate more shuffle data than hive on mr
[
https://issues.apache.org/jira/browse/HIVE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998177#comment-15998177
]
Rui Li commented on HIVE-15104:
---
I looked at the shuffle writers of Spark and none of them seem to need the
hashCode/partitionId after the HiveKey is serialized. But I got a problem
during implementation. The plan is to implement this Spark trait:
{code}
trait KryoRegistrator {
def registerClasses(kryo: Kryo): Unit
}
{code}
Then we set this implementing class to {{spark.kryo.registrator}}. At runtime,
Spark will use reflection to instantiate our class and call its registerClasses
to register the optimized SerDe for HiveKey.
However, Kryo is relocated in Hive. After build, the method signature of our
class will actually be:
{{public void registerClasses(org.apache.hive.com.esotericsoftware.kryo.Kryo
kryo)}}.
When Spark calls the method, we get an {{AbstractMethodError}}. I suppose this
is because the {{public void registerClasses(com.esotericsoftware.kryo.Kryo
kryo)}} method is not really implemented.
Does anybody know how this can be resolved?
> Hive on Spark generate more shuffle data than hive on mr
>
>
> Key: HIVE-15104
> URL: https://issues.apache.org/jira/browse/HIVE-15104
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.2.1
>Reporter: wangwenli
>Assignee: Rui Li
>
> the same sql, running on spark and mr engine, will generate different size
> of shuffle data.
> i think it is because of hive on mr just serialize part of HiveKey, but hive
> on spark which using kryo will serialize full of Hivekey object.
> what is your opionion?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15483) Database and table name is case sensitive when used in show grant
[
https://issues.apache.org/jira/browse/HIVE-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998165#comment-15998165
]
Zoltan Haindrich commented on HIVE-15483:
-
+1
> Database and table name is case sensitive when used in show grant
> -
>
> Key: HIVE-15483
> URL: https://issues.apache.org/jira/browse/HIVE-15483
> Project: Hive
> Issue Type: Bug
> Components: Authorization, SQLStandardAuthorization
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15483.1.patch
>
>
> When use SQLStdAuth, db name and table name is case sensitive when use
> {{show grant}} command.
> {code}
> 0: jdbc:hive2://localhost:21066/> show grant on table p1;
> +---+++-+--+-++---++--+--+
> | database | table | partition | column |principal_name|
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+--+-++---++--+--+
> | default | p1 || | userx| USER
>| DELETE | true | 1481872357000 | userx|
> | default | p1 || | userx| USER
>| INSERT | true | 1481872357000 | userx|
> | default | p1 || | userx| USER
>| SELECT | true | 1481872357000 | userx|
> | default | p1 || | userx| USER
>| UPDATE | true | 1481872357000 | userx|
> +---+++-+--+-++---++--+--+
> 7 rows selected (0.158 seconds)
> 0: jdbc:hive2://localhost:21066/> show grant on table P1;
> +---+++-+-+-++---+-+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+-+--+--+
> +---+++-+-+-++---+-+--+--+
> No rows selected (9.608 seconds)
> 0: jdbc:hive2://localhost:21066/> show grant on table defaulT.p1;
> +---+++-+-+-++---+-+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+-+--+--+
> +---+++-+-+-++---+-+--+--+
> No rows selected (0.06 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16449) BeeLineDriver should handle query result sorting
[ https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-16449: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) pushed to master, thank you Peter for taking care of this! > BeeLineDriver should handle query result sorting > > > Key: HIVE-16449 > URL: https://issues.apache.org/jira/browse/HIVE-16449 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Fix For: 3.0.0 > > Attachments: HIVE-16449.02.patch, HIVE-16449.03.patch, > HIVE-16449.04.patch, HIVE-16449.05.patch, HIVE-16449.06.patch, > HIVE-16449.07.patch, HIVE-16449.08.patch, HIVE-16449.patch > > > The CLI driver supports the following features: > -- SORT_QUERY_RESULTS > -- HASH_QUERY_RESULTS > -- SORT_AND_HASH_QUERY_RESULTS > BeeLineDriver should find a way to support these -- This message was sent by Atlassian JIRA (v6.3.15#6346)
