[jira] [Commented] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765766#comment-16765766
]
Hive QA commented on HIVE-21241:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} common: The patch generated 0 new + 0 unchanged - 14
fixed = 0 total (was 14) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} common generated 0 new + 64 unchanged - 1 fixed = 64
total (was 65) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} common generated 0 new + 26 unchanged - 1 fixed = 26
total (was 27) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16028/dev-support/hive-personality.sh
|
| git revision | master / 8e3c5fc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16028/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch, HIVE-21241.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765758#comment-16765758
]
Hive QA commented on HIVE-21246:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958311/HIVE-21246.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15784 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
(batchId=256)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16027/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16027/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16027/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958311 - PreCommit-HIVE-Build
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a flag gets passed into utilities. If the class boolean is set,
> the passed-in SerDe is overwritten. This is not documented anywhere and it's
> weird to do it, just pass in the SerDe to use instead of the SerDe you don't
> want to use and a flag to change it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21232) LLAP: Add a cache-miss friendly split affinity provider
[ https://issues.apache.org/jira/browse/HIVE-21232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-21232: -- Assignee: Gopal V > LLAP: Add a cache-miss friendly split affinity provider > --- > > Key: HIVE-21232 > URL: https://issues.apache.org/jira/browse/HIVE-21232 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > > If one of the LLAP nodes have data-locality, preferring that over another > does have advantages for the first query or a more general cache-miss. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765738#comment-16765738
]
Hive QA commented on HIVE-21246:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 2298 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} ql: The patch generated 0 new + 506 unchanged - 1
fixed = 506 total (was 507) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16027/dev-support/hive-personality.sh
|
| git revision | master / 8e3c5fc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16027/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a flag gets passed into utilities. If the class boolean is set,
> the passed-in SerDe is overwritten. This is not documented anywhere and it's
> weird to do it, just pass in the SerDe to use instead of the SerDe you don't
> want to use and a flag to change it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21206) Bootstrap replication is slow as it opens lot of metastore connections.
[
https://issues.apache.org/jira/browse/HIVE-21206?focusedWorklogId=197403&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-197403
]
ASF GitHub Bot logged work on HIVE-21206:
-
Author: ASF GitHub Bot
Created on: 12/Feb/19 06:55
Start Date: 12/Feb/19 06:55
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #529: HIVE-21206:
Bootstrap replication is slow as it opens lot of metastore connections.
URL: https://github.com/apache/hive/pull/529
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 197403)
Time Spent: 20m (was: 10m)
> Bootstrap replication is slow as it opens lot of metastore connections.
> ---
>
> Key: HIVE-21206
> URL: https://issues.apache.org/jira/browse/HIVE-21206
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Fix For: 4.0.0
>
> Attachments: HIVE-21206.01.patch, HIVE-21206.02.patch,
> HIVE-21206.03.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hive bootstrap replication of 1TB data onprem to onprem in Hive3 is running
> slower compared to Hive2.
> Time taken for bootstrap replication of table with 1000 partitions are as
> below:
> ||Hive2- Hive2|| Hive3 - Hive3 ||
> |Bootstrap: 7m| BootStrap: 17m |
> Every MoveTask is closing and opening new metastore connection which is
> causing slow down.
> {code}
> 2019-02-08T12:28:30,174 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-5:MOVE] in serial mode
> 2019-02-08T12:28:30,177 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table1 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/.hive-staging_hive_2019-02-08_12-28-23_584_1482331698286040936-3/-ext-10001
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://ctr-e139-1542663976389-62755-01-14.hwx.site:9083
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - HMSC::open(): Could not find delegation
> token. Creating KERBEROS-based thrift connection.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 4
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Connected to metastore.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> ugi=hive/ctr-e139-1542663976389-62755-01-14.hwx.s...@hwqe.hortonworks.com
> (auth:KERBEROS) retries=24 delay=5 lifetime=0
> 2019-02-08T12:28:30,325 INFO
> [org.apache.ranger.audit.queue.AuditBatchQueue1]: provider.BaseAuditHandler
> (:()) - Audit Status Log: name=hiveServer2.async.multi_dest.batch,
> finalDestination=hiveServer2.async.multi_dest.batch.solr, interval=01:00.002
> minutes, events=2, succcessCount=1, totalEvents=56, totalSuccessCount=25
> 2019-02-08T12:28:30,520 INFO [HiveServer2-Background-Pool: Thread-1134]:
> common.FileUtils (FileUtils.java:mkdir(580)) - Creating directory if it
> doesn't exist:
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/base_001
> 2019-02-08T12:28:31,245 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-11:MOVE] in serial mode
> 2019-02-08T12:28:31,245 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore,
> current connections: 3
> 2019-02-08T12:28:31,246 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table2 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table2/.hive-staging_hive_2019-02-08_12-28-23_810_7457138692783022870-3/-ext-10002
> 2019-02-08T12:28:31,327 INFO [HiveServer2-Background-Pool: Thread-
[jira] [Commented] (HIVE-21244) NPE in Hive Proto Logger
[ https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765711#comment-16765711 ] Harish Jaiprakash commented on HIVE-21244: -- The log message is incorrect, we are not closing a file, but logging into the new file. The old message was different, because it was logged before close. > NPE in Hive Proto Logger > > > Key: HIVE-21244 > URL: https://issues.apache.org/jira/browse/HIVE-21244 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21244.1.patch > > > [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308] > can cause NPE. There is no uncaught exception handler for this thread. This > NPE can silently fail and drop the event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21250) NPE in HiveProtoLoggingHook for eventPerFile mode.
[ https://issues.apache.org/jira/browse/HIVE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21250: - Resolution: Duplicate Status: Resolved (was: Patch Available) > NPE in HiveProtoLoggingHook for eventPerFile mode. > -- > > Key: HIVE-21250 > URL: https://issues.apache.org/jira/browse/HIVE-21250 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21250.01.patch > > > When eventPerFile is enabled, writer is set to null after the first event, it > causes an NPE in the next path until handleTick comes back. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21250) NPE in HiveProtoLoggingHook for eventPerFile mode.
[ https://issues.apache.org/jira/browse/HIVE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21250: - Status: Patch Available (was: Open) [~anishek], [~prasanth_j], Please take a look. Just 2 lines changed, hope tests are not required for this. > NPE in HiveProtoLoggingHook for eventPerFile mode. > -- > > Key: HIVE-21250 > URL: https://issues.apache.org/jira/browse/HIVE-21250 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21250.01.patch > > > When eventPerFile is enabled, writer is set to null after the first event, it > causes an NPE in the next path until handleTick comes back. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21250) NPE in HiveProtoLoggingHook for eventPerFile mode.
[ https://issues.apache.org/jira/browse/HIVE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21250: - Attachment: HIVE-21250.01.patch > NPE in HiveProtoLoggingHook for eventPerFile mode. > -- > > Key: HIVE-21250 > URL: https://issues.apache.org/jira/browse/HIVE-21250 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21250.01.patch > > > When eventPerFile is enabled, writer is set to null after the first event, it > causes an NPE in the next path until handleTick comes back. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21250) NPE in HiveProtoLoggingHook for eventPerFile mode.
[ https://issues.apache.org/jira/browse/HIVE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned HIVE-21250: > NPE in HiveProtoLoggingHook for eventPerFile mode. > -- > > Key: HIVE-21250 > URL: https://issues.apache.org/jira/browse/HIVE-21250 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21250.01.patch > > > When eventPerFile is enabled, writer is set to null after the first event, it > causes an NPE in the next path until handleTick comes back. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21245) Make hive.fetch.output.serde Default to LazySimpleSerde
[
https://issues.apache.org/jira/browse/HIVE-21245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765688#comment-16765688
]
Hive QA commented on HIVE-21245:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16026/dev-support/hive-personality.sh
|
| git revision | master / 8e3c5fc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16026/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make hive.fetch.output.serde Default to LazySimpleSerde
> ---
>
> Key: HIVE-21245
> URL: https://issues.apache.org/jira/browse/HIVE-21245
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21245.1.patch
>
>
> For all intents and purposes, it already is:
> {code:java|title=HiveSessionImpl.java}
> private static final String FETCH_WORK_SERDE_CLASS =
> "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe";
> @Override
> public HiveConf getHiveConf() {
> sessionConf.setVar(HiveConf.ConfVars.HIVEFETCHOUTPUTSERDE,
> FETCH_WORK_SERDE_CLASS);
> return sessionConf;
> }
> {code}
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java#L489-L492
> The current default is:
> {code}
> // Serde for FetchTask
> HIVEFETCHOUTPUTSERDE("hive.fetch.output.serde",
> "org.apache.hadoop.hive.serde2.DelimitedJSONSerDe",
> "The SerDe used by FetchTask to serialize the fetch output."),
> {code}
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}} altogether. It's a
> weird thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21206) Bootstrap replication is slow as it opens lot of metastore connections.
[
https://issues.apache.org/jira/browse/HIVE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-21206:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
03.patch is committed to master.
Thanks [~maheshk114] for the review!
> Bootstrap replication is slow as it opens lot of metastore connections.
> ---
>
> Key: HIVE-21206
> URL: https://issues.apache.org/jira/browse/HIVE-21206
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Fix For: 4.0.0
>
> Attachments: HIVE-21206.01.patch, HIVE-21206.02.patch,
> HIVE-21206.03.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive bootstrap replication of 1TB data onprem to onprem in Hive3 is running
> slower compared to Hive2.
> Time taken for bootstrap replication of table with 1000 partitions are as
> below:
> ||Hive2- Hive2|| Hive3 - Hive3 ||
> |Bootstrap: 7m| BootStrap: 17m |
> Every MoveTask is closing and opening new metastore connection which is
> causing slow down.
> {code}
> 2019-02-08T12:28:30,174 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-5:MOVE] in serial mode
> 2019-02-08T12:28:30,177 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table1 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/.hive-staging_hive_2019-02-08_12-28-23_584_1482331698286040936-3/-ext-10001
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://ctr-e139-1542663976389-62755-01-14.hwx.site:9083
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - HMSC::open(): Could not find delegation
> token. Creating KERBEROS-based thrift connection.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 4
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Connected to metastore.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> ugi=hive/ctr-e139-1542663976389-62755-01-14.hwx.s...@hwqe.hortonworks.com
> (auth:KERBEROS) retries=24 delay=5 lifetime=0
> 2019-02-08T12:28:30,325 INFO
> [org.apache.ranger.audit.queue.AuditBatchQueue1]: provider.BaseAuditHandler
> (:()) - Audit Status Log: name=hiveServer2.async.multi_dest.batch,
> finalDestination=hiveServer2.async.multi_dest.batch.solr, interval=01:00.002
> minutes, events=2, succcessCount=1, totalEvents=56, totalSuccessCount=25
> 2019-02-08T12:28:30,520 INFO [HiveServer2-Background-Pool: Thread-1134]:
> common.FileUtils (FileUtils.java:mkdir(580)) - Creating directory if it
> doesn't exist:
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/base_001
> 2019-02-08T12:28:31,245 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-11:MOVE] in serial mode
> 2019-02-08T12:28:31,245 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore,
> current connections: 3
> 2019-02-08T12:28:31,246 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table2 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table2/.hive-staging_hive_2019-02-08_12-28-23_810_7457138692783022870-3/-ext-10002
> 2019-02-08T12:28:31,327 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://ctr-e139-1542663976389-62755-01-14.hwx.site:9083
> 2019-02-08T12:28:31,327 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - HMSC::open(): Could not find delegation
> token. Creating KERBEROS-based thrift connection.
> 2019-02-08T12:28:31,336 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 4
> 2019-02-08T12:28:31,337 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) -
[jira] [Commented] (HIVE-21223) CachedStore returns null partition when partition does not exist
[
https://issues.apache.org/jira/browse/HIVE-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765678#comment-16765678
]
Hive QA commented on HIVE-21223:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958304/HIVE-21223.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15784 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16025/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16025/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16025/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958304 - PreCommit-HIVE-Build
> CachedStore returns null partition when partition does not exist
>
>
> Key: HIVE-21223
> URL: https://issues.apache.org/jira/browse/HIVE-21223
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21223.1.patch, HIVE-21223.2.patch
>
>
> CachedStore can return null partition for getPartitionWithAuth() when
> partition does not exist. null value serialization in thrift will break the
> connection. Instead if partition does not exist it should throw
> NoSuchObjectException.
> Clients will see this exception
> {code:java}
> org.apache.thrift.TApplicationException: get_partition_with_auth failed:
> unknown result
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition_with_auth(ThriftHiveMetastore.java:3017)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition_with_auth(ThriftHiveMetastore.java:2990)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1679)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1671)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2976)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.SynchronizedMetaStoreClient.getPartitionWithAuthInfo(SynchronizedMetaStoreClient.java:101)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2870)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2835)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1950)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2490)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2481)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
> at
> java.util.concurrent.Thread
[jira] [Commented] (HIVE-21081) DATE_FORMAT incorrectly returns results on the last week of the calendar year
[
https://issues.apache.org/jira/browse/HIVE-21081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765672#comment-16765672
]
Rahul Pathak commented on HIVE-21081:
-
This looks like expected behavior.
Y represent week year and y is used to represent calendar year.
A week year is in sync with a WEEK_OF_YEAR cycle.
All weeks between the first and last weeks (inclusive) have the same week year
value.
Therefore, the first and last days of a week year may have different calendar
year values.
https://docs.oracle.com/javase/9/docs/api/java/util/GregorianCalendar.html#week_year
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF
> DATE_FORMAT incorrectly returns results on the last week of the calendar year
> -
>
> Key: HIVE-21081
> URL: https://issues.apache.org/jira/browse/HIVE-21081
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.1.1, 2.3.2, 2.3.3
>Reporter: Wilson Lu
>Priority: Minor
>
> The Hive DATE_FORMAT does not perform the correct operation on the last week
> of the calendar year. The following statements incorrectly formats the data:
> select DATE_FORMAT('2017-12-31', 'MM')
> select DATE_FORMAT('2018-12-30', 'MM')
> select DATE_FORMAT('2018-12-31', 'MM')
> select DATE_FORMAT('2019-12-29', 'MM')
> select DATE_FORMAT('2019-12-30', 'MM')
> select DATE_FORMAT('2019-12-31', 'MM')
> select DATE_FORMAT( '2019-12-29', 'MMdd')
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21223) CachedStore returns null partition when partition does not exist
[
https://issues.apache.org/jira/browse/HIVE-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765661#comment-16765661
]
Hive QA commented on HIVE-21223:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
4s{color} | {color:blue} standalone-metastore/metastore-server in master has
183 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16025/dev-support/hive-personality.sh
|
| git revision | master / 4ddc9de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16025/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CachedStore returns null partition when partition does not exist
>
>
> Key: HIVE-21223
> URL: https://issues.apache.org/jira/browse/HIVE-21223
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21223.1.patch, HIVE-21223.2.patch
>
>
> CachedStore can return null partition for getPartitionWithAuth() when
> partition does not exist. null value serialization in thrift will break the
> connection. Instead if partition does not exist it should throw
> NoSuchObjectException.
> Clients will see this exception
> {code:java}
> org.apache.thrift.TApplicationException: get_partition_with_auth failed:
> unknown result
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition_with_auth(ThriftHiveMetastore.java:3017)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition_with_auth(ThriftHiveMetastore.java:2990)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1679)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:
[jira] [Commented] (HIVE-21244) NPE in Hive Proto Logger
[
https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765644#comment-16765644
]
Hive QA commented on HIVE-21244:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958303/HIVE-21244.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15784 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16024/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16024/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16024/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958303 - PreCommit-HIVE-Build
> NPE in Hive Proto Logger
>
>
> Key: HIVE-21244
> URL: https://issues.apache.org/jira/browse/HIVE-21244
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21244.1.patch
>
>
> [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308]
> can cause NPE. There is no uncaught exception handler for this thread. This
> NPE can silently fail and drop the event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21244) NPE in Hive Proto Logger
[
https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765624#comment-16765624
]
Hive QA commented on HIVE-21244:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 2298 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 1 new + 6 unchanged - 0 fixed
= 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16024/dev-support/hive-personality.sh
|
| git revision | master / 4ddc9de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16024/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16024/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NPE in Hive Proto Logger
>
>
> Key: HIVE-21244
> URL: https://issues.apache.org/jira/browse/HIVE-21244
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21244.1.patch
>
>
> [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308]
> can cause NPE. There is no uncaught exception handler for this thread. This
> NPE can silently fail and drop the event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21055) Replication load command executing copy in serial mode even if parallel execution is enabled using with clause
[ https://issues.apache.org/jira/browse/HIVE-21055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21055: --- Attachment: HIVE-21055.03.patch > Replication load command executing copy in serial mode even if parallel > execution is enabled using with clause > -- > > Key: HIVE-21055 > URL: https://issues.apache.org/jira/browse/HIVE-21055 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21055.01.patch, HIVE-21055.02.patch, > HIVE-21055.03.patch > > > For repl load command use can specify the execution mode as part of "with" > clause. But the config for executing task in parallel or serial is not read > from the command specific config. It is read from the hive server config. So > even if user specifies to run the tasks in parallel during repl load command, > the tasks are getting executed serially. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21055) Replication load command executing copy in serial mode even if parallel execution is enabled using with clause
[ https://issues.apache.org/jira/browse/HIVE-21055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21055: --- Status: Patch Available (was: Open) > Replication load command executing copy in serial mode even if parallel > execution is enabled using with clause > -- > > Key: HIVE-21055 > URL: https://issues.apache.org/jira/browse/HIVE-21055 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21055.01.patch, HIVE-21055.02.patch, > HIVE-21055.03.patch > > > For repl load command use can specify the execution mode as part of "with" > clause. But the config for executing task in parallel or serial is not read > from the command specific config. It is read from the hive server config. So > even if user specifies to run the tasks in parallel during repl load command, > the tasks are getting executed serially. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21055) Replication load command executing copy in serial mode even if parallel execution is enabled using with clause
[ https://issues.apache.org/jira/browse/HIVE-21055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21055: --- Status: Open (was: Patch Available) > Replication load command executing copy in serial mode even if parallel > execution is enabled using with clause > -- > > Key: HIVE-21055 > URL: https://issues.apache.org/jira/browse/HIVE-21055 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21055.01.patch, HIVE-21055.02.patch > > > For repl load command use can specify the execution mode as part of "with" > clause. But the config for executing task in parallel or serial is not read > from the command specific config. It is read from the hive server config. So > even if user specifies to run the tasks in parallel during repl load command, > the tasks are getting executed serially. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21206) Bootstrap replication is slow as it opens lot of metastore connections.
[
https://issues.apache.org/jira/browse/HIVE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765605#comment-16765605
]
Hive QA commented on HIVE-21206:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958289/HIVE-21206.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15784 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16023/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16023/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16023/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958289 - PreCommit-HIVE-Build
> Bootstrap replication is slow as it opens lot of metastore connections.
> ---
>
> Key: HIVE-21206
> URL: https://issues.apache.org/jira/browse/HIVE-21206
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21206.01.patch, HIVE-21206.02.patch,
> HIVE-21206.03.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive bootstrap replication of 1TB data onprem to onprem in Hive3 is running
> slower compared to Hive2.
> Time taken for bootstrap replication of table with 1000 partitions are as
> below:
> ||Hive2- Hive2|| Hive3 - Hive3 ||
> |Bootstrap: 7m| BootStrap: 17m |
> Every MoveTask is closing and opening new metastore connection which is
> causing slow down.
> {code}
> 2019-02-08T12:28:30,174 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-5:MOVE] in serial mode
> 2019-02-08T12:28:30,177 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table1 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/.hive-staging_hive_2019-02-08_12-28-23_584_1482331698286040936-3/-ext-10001
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://ctr-e139-1542663976389-62755-01-14.hwx.site:9083
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - HMSC::open(): Could not find delegation
> token. Creating KERBEROS-based thrift connection.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 4
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Connected to metastore.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> ugi=hive/ctr-e139-1542663976389-62755-01-14.hwx.s...@hwqe.hortonworks.com
> (auth:KERBEROS) retries=24 delay=5 lifetime=0
> 2019-02-08T12:28:30,325 INFO
> [org.apache.ranger.audit.queue.AuditBatchQueue1]: provider.BaseAuditHandler
> (:()) - Audit Status Log: name=hiveServer2.async.multi_dest.batch,
> finalDestination=hiveServer2.async.multi_dest.batch.solr, interval=01:00.002
> minutes, events=2, succcessCount=1, totalEvents=56, totalSuccessCount=25
> 2019-02-08T12:28:30,520 INFO [HiveServer2-Background-Pool: Thread-1134]:
> common.FileUtils (FileUtils.java:mkdir(580)) - Creating directory if it
> doesn't exist:
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/base_001
> 2019-02-08T12:28:31,245 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-11:MOVE] in serial mode
> 2019-02-08T12:28:31,245 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore,
> current connections: 3
> 2019-02-08T12:28:31,246 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table2 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table2/.hive-staging_hive_2019-02-08_12-28-23_810_7457138692783022870-3/-
[jira] [Commented] (HIVE-21206) Bootstrap replication is slow as it opens lot of metastore connections.
[
https://issues.apache.org/jira/browse/HIVE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765591#comment-16765591
]
Hive QA commented on HIVE-21206:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 2298 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 342 unchanged - 1
fixed = 342 total (was 343) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16023/dev-support/hive-personality.sh
|
| git revision | master / 4ddc9de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16023/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Bootstrap replication is slow as it opens lot of metastore connections.
> ---
>
> Key: HIVE-21206
> URL: https://issues.apache.org/jira/browse/HIVE-21206
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21206.01.patch, HIVE-21206.02.patch,
> HIVE-21206.03.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive bootstrap replication of 1TB data onprem to onprem in Hive3 is running
> slower compared to Hive2.
> Time taken for bootstrap replication of table with 1000 partitions are as
> below:
> ||Hive2- Hive2|| Hive3 - Hive3 ||
> |Bootstrap: 7m| BootStrap: 17m |
> Every MoveTask is closing and opening new metastore connection which is
> causing slow down.
> {code}
> 2019-02-08T12:28:30,174 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-5:MOVE] in serial mode
> 2019-02-08T12:28:30,177 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table1 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/.hive-staging_hive_2019-02-08_12-28-23_584_1482331698286040936-3/-ext-10001
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Trying
[jira] [Commented] (HIVE-18101) ISO 8601 UDF
[
https://issues.apache.org/jira/browse/HIVE-18101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765579#comment-16765579
]
Hive QA commented on HIVE-18101:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898370/HIVE-18101.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 15784 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestActivePassiveHA.testActivePassiveHA (batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testClientConnectionsOnFailover
(batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testConnectionActivePassiveHAServiceDiscovery
(batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailover (batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailoverUnauthorized
(batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testNoConnectionOnPassive (batchId=261)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16022/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16022/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16022/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12898370 - PreCommit-HIVE-Build
> ISO 8601 UDF
>
>
> Key: HIVE-18101
> URL: https://issues.apache.org/jira/browse/HIVE-18101
> Project: Hive
> Issue Type: New Feature
> Components: Contrib
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-18101.1.patch
>
>
> Please consider my submission of a UDF for parsing ISO 8601 strings into
> timestamps. It uses Java 1.8's
> {{java.time.format.DateTimeFormatter.ISO_OFFSET_DATE_TIME}} class.
> https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html
> https://en.wikipedia.org/wiki/ISO_8601
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21236) SharedWorkOptimizer should check table properties
[ https://issues.apache.org/jira/browse/HIVE-21236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21236: --- Attachment: HIVE-21236.02.patch > SharedWorkOptimizer should check table properties > - > > Key: HIVE-21236 > URL: https://issues.apache.org/jira/browse/HIVE-21236 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21236.01.patch, HIVE-21236.02.patch, > HIVE-21236.patch > > > For instance, Calcite may have pushed computation to Druid or a JDBC source, > rest of table structures may look the same, but the embedded query is > different. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21249) Reduce memory footprint in ObjectStore.refreshPrivileges
[
https://issues.apache.org/jira/browse/HIVE-21249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai reassigned HIVE-21249:
-
> Reduce memory footprint in ObjectStore.refreshPrivileges
> -
>
> Key: HIVE-21249
> URL: https://issues.apache.org/jira/browse/HIVE-21249
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
>
> We found there're could be many records in TBL_COL_PRIVS for a single table
> (a table granted to many users), thus result a OOM in
> ObjectStore.listTableAllColumnGrants. We shall reduce the memory footprint
> for ObjectStore.refreshPrivileges. Here is the stack of OOM:
> {code}
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:690)
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:710)
> org.apache.hadoop.hive.metastore.ObjectStore.listTableAllColumnGrants(ObjectStore.java:6629)
> org.apache.hadoop.hive.metastore.ObjectStore.refreshPrivileges(ObjectStore.java:6200)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> com.sun.proxy.$Proxy32.refreshPrivileges(, line not available)
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.refresh_privileges(HiveMetaStore.java:6507)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> com.sun.proxy.$Proxy34.refresh_privileges(, line not available)
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17608)
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17592)
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> java.security.AccessController.doPrivileged(Native method)
> javax.security.auth.Subject.doAs(Subject.java:422)
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21249) Reduce memory footprint in ObjectStore.refreshPrivileges
[
https://issues.apache.org/jira/browse/HIVE-21249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-21249:
--
Status: Patch Available (was: Open)
> Reduce memory footprint in ObjectStore.refreshPrivileges
> -
>
> Key: HIVE-21249
> URL: https://issues.apache.org/jira/browse/HIVE-21249
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21249.1.patch
>
>
> We found there're could be many records in TBL_COL_PRIVS for a single table
> (a table granted to many users), thus result a OOM in
> ObjectStore.listTableAllColumnGrants. We shall reduce the memory footprint
> for ObjectStore.refreshPrivileges. Here is the stack of OOM:
> {code}
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:690)
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:710)
> org.apache.hadoop.hive.metastore.ObjectStore.listTableAllColumnGrants(ObjectStore.java:6629)
> org.apache.hadoop.hive.metastore.ObjectStore.refreshPrivileges(ObjectStore.java:6200)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> com.sun.proxy.$Proxy32.refreshPrivileges(, line not available)
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.refresh_privileges(HiveMetaStore.java:6507)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> com.sun.proxy.$Proxy34.refresh_privileges(, line not available)
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17608)
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17592)
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> java.security.AccessController.doPrivileged(Native method)
> javax.security.auth.Subject.doAs(Subject.java:422)
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21249) Reduce memory footprint in ObjectStore.refreshPrivileges
[
https://issues.apache.org/jira/browse/HIVE-21249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-21249:
--
Attachment: HIVE-21249.1.patch
> Reduce memory footprint in ObjectStore.refreshPrivileges
> -
>
> Key: HIVE-21249
> URL: https://issues.apache.org/jira/browse/HIVE-21249
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21249.1.patch
>
>
> We found there're could be many records in TBL_COL_PRIVS for a single table
> (a table granted to many users), thus result a OOM in
> ObjectStore.listTableAllColumnGrants. We shall reduce the memory footprint
> for ObjectStore.refreshPrivileges. Here is the stack of OOM:
> {code}
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:690)
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:710)
> org.apache.hadoop.hive.metastore.ObjectStore.listTableAllColumnGrants(ObjectStore.java:6629)
> org.apache.hadoop.hive.metastore.ObjectStore.refreshPrivileges(ObjectStore.java:6200)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> com.sun.proxy.$Proxy32.refreshPrivileges(, line not available)
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.refresh_privileges(HiveMetaStore.java:6507)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> com.sun.proxy.$Proxy34.refresh_privileges(, line not available)
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17608)
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17592)
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> java.security.AccessController.doPrivileged(Native method)
> javax.security.auth.Subject.doAs(Subject.java:422)
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21248) WebHCat returns HTTP error code 500 rather than 429 when submitting large number of jobs in stress tests
[
https://issues.apache.org/jira/browse/HIVE-21248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai reassigned HIVE-21248:
-
> WebHCat returns HTTP error code 500 rather than 429 when submitting large
> number of jobs in stress tests
>
>
> Key: HIVE-21248
> URL: https://issues.apache.org/jira/browse/HIVE-21248
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
>
> Saw the exception in webhcat.log:
> {code}
> java.lang.NoSuchMethodError:
> javax.ws.rs.core.Response$Status$Family.familyOf(I)Ljavax/ws/rs/core/Response$Status$Family;
> at
> org.glassfish.jersey.message.internal.Statuses$StatusImpl.(Statuses.java:63)
> ~[jersey-common-2.25.1.jar:?]
> at
> org.glassfish.jersey.message.internal.Statuses$StatusImpl.(Statuses.java:54)
> ~[jersey-common-2.25.1.jar:?]
> at
> org.glassfish.jersey.message.internal.Statuses.from(Statuses.java:132)
> ~[jersey-common-2.25.1.jar:?]
> at
> org.glassfish.jersey.message.internal.OutboundJaxrsResponse$Builder.status(OutboundJaxrsResponse.java:414)
> ~[jersey-common-2.25.1.jar:?]
> at javax.ws.rs.core.Response.status(Response.java:128)
> ~[jsr311-api-1.1.1.jar:?]
> at
> org.apache.hive.hcatalog.templeton.SimpleWebException.buildMessage(SimpleWebException.java:67)
> ~[hive-webhcat-3.1.0.3.0.2.0-50.jar:3.1.0.3.0.2.0-50]
> at
> org.apache.hive.hcatalog.templeton.SimpleWebException.getResponse(SimpleWebException.java:51)
> ~[hive-webhcat-3.1.0.3.0.2.0-50.jar:3.1.0.3.0.2.0-50]
> at
> org.apache.hive.hcatalog.templeton.SimpleExceptionMapper.toResponse(SimpleExceptionMapper.java:33)
> ~[hive-webhcat-3.1.0.3.0.2.0-50.jar:3.1.0.3.0.2.0-50]
> at
> org.apache.hive.hcatalog.templeton.SimpleExceptionMapper.toResponse(SimpleExceptionMapper.java:29)
> ~[hive-webhcat-3.1.0.3.0.2.0-50.jar:3.1.0.3.0.2.0-50]
> at
> com.sun.jersey.spi.container.ContainerResponse.mapException(ContainerResponse.java:480)
> ~[jersey-server-1.19.jar:1.19]
> at
> com.sun.jersey.spi.container.ContainerResponse.mapMappableContainerException(ContainerResponse.java:417)
> ~[jersey-server-1.19.jar:1.19]
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1477)
> ~[jersey-server-1.19.jar:1.19]
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> ~[jersey-server-1.19.jar:1.19]
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> ~[jersey-server-1.19.jar:1.19]
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> ~[jersey-servlet-1.19.jar:1.19]
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> ~[jersey-servlet-1.19.jar:1.19]
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> ~[jersey-servlet-1.19.jar:1.19]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> ~[javax.servlet-api-3.1.0.jar:3.1.0]
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
> at
> org.apache.hive.hcatalog.templeton.Main$XFrameOptionsFilter.doFilter(Main.java:299)
> ~[hive-webhcat-3.1.0.3.0.2.0-50.jar:3.1.0.3.0.2.0-50]
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
> ~[hadoop-auth-3.1.1.3.0.2.0-50.jar:?]
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
> ~[hadoop-auth-3.1.1.3.0.2.0-50.jar:?]
> at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:90)
> ~[hadoop-hdfs-3.1.1.3.0.2.0-50.jar:?]
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> [jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
[jira] [Updated] (HIVE-21247) Webhcat beeline in secure mode
[ https://issues.apache.org/jira/browse/HIVE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-21247: -- Attachment: (was: HIVE-21247.1.patch) > Webhcat beeline in secure mode > -- > > Key: HIVE-21247 > URL: https://issues.apache.org/jira/browse/HIVE-21247 > Project: Hive > Issue Type: Improvement > Components: WebHCat >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21247.1.patch > > > Follow up HIVE-20550, we need to make beeline work in secure mode. That > means, we need to get a delegation token from hiveserver2, and pass that to > beeline. This is similar to HIVE-5133, I make two changes: > 1. Make a jdbc connection to hs2, pull delegation token from HiveConnection, > and pass along > 2. In hive jdbc driver, check for token file in HADOOP_TOKEN_FILE_LOCATION, > and extract delegation token if exists > There's one config needed in webhcat-site.xml, to point > "templeton.hive.hs2.url" to hive jdbc url. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21247) Webhcat beeline in secure mode
[ https://issues.apache.org/jira/browse/HIVE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-21247: -- Attachment: HIVE-21247.1.patch > Webhcat beeline in secure mode > -- > > Key: HIVE-21247 > URL: https://issues.apache.org/jira/browse/HIVE-21247 > Project: Hive > Issue Type: Improvement > Components: WebHCat >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21247.1.patch > > > Follow up HIVE-20550, we need to make beeline work in secure mode. That > means, we need to get a delegation token from hiveserver2, and pass that to > beeline. This is similar to HIVE-5133, I make two changes: > 1. Make a jdbc connection to hs2, pull delegation token from HiveConnection, > and pass along > 2. In hive jdbc driver, check for token file in HADOOP_TOKEN_FILE_LOCATION, > and extract delegation token if exists > There's one config needed in webhcat-site.xml, to point > "templeton.hive.hs2.url" to hive jdbc url. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18101) ISO 8601 UDF
[
https://issues.apache.org/jira/browse/HIVE-18101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765562#comment-16765562
]
Hive QA commented on HIVE-18101:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} contrib in master has 10 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} contrib: The patch generated 13 new + 0 unchanged - 0
fixed = 13 total (was 0) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16022/dev-support/hive-personality.sh
|
| git revision | master / 4ddc9de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16022/yetus/diff-checkstyle-contrib.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16022/yetus/whitespace-eol.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16022/yetus/patch-asflicense-problems.txt
|
| modules | C: contrib U: contrib |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16022/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ISO 8601 UDF
>
>
> Key: HIVE-18101
> URL: https://issues.apache.org/jira/browse/HIVE-18101
> Project: Hive
> Issue Type: New Feature
> Components: Contrib
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-18101.1.patch
>
>
> Please consider my submission of a UDF for parsing ISO 8601 strings into
> timestamps. It uses Java 1.8's
> {{java.time.format.DateTimeFormatter.ISO_OFFSET_DATE_TIME}} class.
> https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html
> https://en.wikipedia.org/wiki/ISO_8601
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21248) WebHCat returns HTTP error code 500 rather than 429 when submitting large number of jobs in stress tests
[
https://issues.apache.org/jira/browse/HIVE-21248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765560#comment-16765560
]
Daniel Dai commented on HIVE-21248:
---
The issue will be addressed as part of HIVE-21247, when WebHCat is pulling
hive-jdbc dependencies and need to exclude jsr311-api-1.1.1.jar due to the
change anyway.
> WebHCat returns HTTP error code 500 rather than 429 when submitting large
> number of jobs in stress tests
>
>
> Key: HIVE-21248
> URL: https://issues.apache.org/jira/browse/HIVE-21248
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
>
> Saw the exception in webhcat.log:
> {code}
> java.lang.NoSuchMethodError:
> javax.ws.rs.core.Response$Status$Family.familyOf(I)Ljavax/ws/rs/core/Response$Status$Family;
> at
> org.glassfish.jersey.message.internal.Statuses$StatusImpl.(Statuses.java:63)
> ~[jersey-common-2.25.1.jar:?]
> at
> org.glassfish.jersey.message.internal.Statuses$StatusImpl.(Statuses.java:54)
> ~[jersey-common-2.25.1.jar:?]
> at
> org.glassfish.jersey.message.internal.Statuses.from(Statuses.java:132)
> ~[jersey-common-2.25.1.jar:?]
> at
> org.glassfish.jersey.message.internal.OutboundJaxrsResponse$Builder.status(OutboundJaxrsResponse.java:414)
> ~[jersey-common-2.25.1.jar:?]
> at javax.ws.rs.core.Response.status(Response.java:128)
> ~[jsr311-api-1.1.1.jar:?]
> at
> org.apache.hive.hcatalog.templeton.SimpleWebException.buildMessage(SimpleWebException.java:67)
> ~[hive-webhcat-3.1.0.3.0.2.0-50.jar:3.1.0.3.0.2.0-50]
> at
> org.apache.hive.hcatalog.templeton.SimpleWebException.getResponse(SimpleWebException.java:51)
> ~[hive-webhcat-3.1.0.3.0.2.0-50.jar:3.1.0.3.0.2.0-50]
> at
> org.apache.hive.hcatalog.templeton.SimpleExceptionMapper.toResponse(SimpleExceptionMapper.java:33)
> ~[hive-webhcat-3.1.0.3.0.2.0-50.jar:3.1.0.3.0.2.0-50]
> at
> org.apache.hive.hcatalog.templeton.SimpleExceptionMapper.toResponse(SimpleExceptionMapper.java:29)
> ~[hive-webhcat-3.1.0.3.0.2.0-50.jar:3.1.0.3.0.2.0-50]
> at
> com.sun.jersey.spi.container.ContainerResponse.mapException(ContainerResponse.java:480)
> ~[jersey-server-1.19.jar:1.19]
> at
> com.sun.jersey.spi.container.ContainerResponse.mapMappableContainerException(ContainerResponse.java:417)
> ~[jersey-server-1.19.jar:1.19]
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1477)
> ~[jersey-server-1.19.jar:1.19]
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> ~[jersey-server-1.19.jar:1.19]
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> ~[jersey-server-1.19.jar:1.19]
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> ~[jersey-servlet-1.19.jar:1.19]
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> ~[jersey-servlet-1.19.jar:1.19]
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> ~[jersey-servlet-1.19.jar:1.19]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> ~[javax.servlet-api-3.1.0.jar:3.1.0]
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
> at
> org.apache.hive.hcatalog.templeton.Main$XFrameOptionsFilter.doFilter(Main.java:299)
> ~[hive-webhcat-3.1.0.3.0.2.0-50.jar:3.1.0.3.0.2.0-50]
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
> ~[hadoop-auth-3.1.1.3.0.2.0-50.jar:?]
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
> ~[hadoop-auth-3.1.1.3.0.2.0-50.jar:?]
> at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:90)
> ~[hadoop-hdfs-3.1.1.3.0.2.0-50.jar:?]
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.jav
[jira] [Updated] (HIVE-21247) Webhcat beeline in secure mode
[ https://issues.apache.org/jira/browse/HIVE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-21247: -- Description: Follow up HIVE-20550, we need to make beeline work in secure mode. That means, we need to get a delegation token from hiveserver2, and pass that to beeline. This is similar to HIVE-5133, I make two changes: 1. Make a jdbc connection to hs2, pull delegation token from HiveConnection, and pass along 2. In hive jdbc driver, check for token file in HADOOP_TOKEN_FILE_LOCATION, and extract delegation token if exists There's one config needed in webhcat-site.xml, to point "templeton.hive.hs2.url" to hive jdbc url. was: Follow up HIVE-20550, we need to make beeline work in secure mode. That means, we need to get a delegation token from hiveserver2, and pass that to beeline. This is similar to HIVE-5133, I make two changes: 1. Make a jdbc connection to hs2, pull delegation token from HiveConnection, and pass along 2. In hive jdbc driver, check for token file in HADOOP_TOKEN_FILE_LOCATION, and extract delegation token if exists > Webhcat beeline in secure mode > -- > > Key: HIVE-21247 > URL: https://issues.apache.org/jira/browse/HIVE-21247 > Project: Hive > Issue Type: Improvement > Components: WebHCat >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21247.1.patch > > > Follow up HIVE-20550, we need to make beeline work in secure mode. That > means, we need to get a delegation token from hiveserver2, and pass that to > beeline. This is similar to HIVE-5133, I make two changes: > 1. Make a jdbc connection to hs2, pull delegation token from HiveConnection, > and pass along > 2. In hive jdbc driver, check for token file in HADOOP_TOKEN_FILE_LOCATION, > and extract delegation token if exists > There's one config needed in webhcat-site.xml, to point > "templeton.hive.hs2.url" to hive jdbc url. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21247) Webhcat beeline in secure mode
[ https://issues.apache.org/jira/browse/HIVE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-21247: -- Attachment: HIVE-21247.1.patch > Webhcat beeline in secure mode > -- > > Key: HIVE-21247 > URL: https://issues.apache.org/jira/browse/HIVE-21247 > Project: Hive > Issue Type: Improvement > Components: WebHCat >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21247.1.patch > > > Follow up HIVE-20550, we need to make beeline work in secure mode. That > means, we need to get a delegation token from hiveserver2, and pass that to > beeline. This is similar to HIVE-5133, I make two changes: > 1. Make a jdbc connection to hs2, pull delegation token from HiveConnection, > and pass along > 2. In hive jdbc driver, check for token file in HADOOP_TOKEN_FILE_LOCATION, > and extract delegation token if exists -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21247) Webhcat beeline in secure mode
[ https://issues.apache.org/jira/browse/HIVE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai reassigned HIVE-21247: - > Webhcat beeline in secure mode > -- > > Key: HIVE-21247 > URL: https://issues.apache.org/jira/browse/HIVE-21247 > Project: Hive > Issue Type: Improvement > Components: WebHCat >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > > Follow up HIVE-20550, we need to make beeline work in secure mode. That > means, we need to get a delegation token from hiveserver2, and pass that to > beeline. This is similar to HIVE-5133, I make two changes: > 1. Make a jdbc connection to hs2, pull delegation token from HiveConnection, > and pass along > 2. In hive jdbc driver, check for token file in HADOOP_TOKEN_FILE_LOCATION, > and extract delegation token if exists -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20900) serde2.JsonSerDe no longer supports timestamp.formats
[
https://issues.apache.org/jira/browse/HIVE-20900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765539#comment-16765539
]
Hive QA commented on HIVE-20900:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12948011/HIVE-20900.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15785 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniHiveKafkaCliDriver.testCliDriver[kafka_storage_handler]
(batchId=275)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16021/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16021/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16021/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12948011 - PreCommit-HIVE-Build
> serde2.JsonSerDe no longer supports timestamp.formats
> -
>
> Key: HIVE-20900
> URL: https://issues.apache.org/jira/browse/HIVE-20900
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20900.1.patch, HIVE-20900.2.patch
>
>
> Looks like HIVE-18545 broke this.
> Also json_serde_tsformat.q only tested the hcat version of JsonSerde, and the
> format in that test used the ISO timestamp format which apparently is now
> parsed by the default timestamp parsing, so the test was too simple.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20900) serde2.JsonSerDe no longer supports timestamp.formats
[
https://issues.apache.org/jira/browse/HIVE-20900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765535#comment-16765535
]
Hive QA commented on HIVE-20900:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 2298 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} serde: The patch generated 0 new + 5 unchanged - 1
fixed = 5 total (was 6) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch . passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
46s{color} | {color:red} serde generated 1 new + 196 unchanged - 1 fixed = 197
total (was 197) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:serde |
| | Found reliance on default encoding in
org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.parsePrimitiveValue(String,
PrimitiveObjectInspector):in
org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.parsePrimitiveValue(String,
PrimitiveObjectInspector): String.getBytes() At
HiveJsonStructReader.java:[line 353] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16021/dev-support/hive-personality.sh
|
| git revision | master / 4ddc9de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16021/yetus/new-findbugs-serde.html
|
| modules | C: serde ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16021/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> serde2.JsonSerDe no longer supports timestamp.formats
> -
>
> Key: HIVE-20900
> URL: https://issues.apache.org/jira/browse/HIVE-20900
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Jason De
[jira] [Commented] (HIVE-19740) Hiveserver2 can't connect to metastore when using Hive 3.0
[ https://issues.apache.org/jira/browse/HIVE-19740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765534#comment-16765534 ] t oo commented on HIVE-19740: - bump > Hiveserver2 can't connect to metastore when using Hive 3.0 > -- > > Key: HIVE-19740 > URL: https://issues.apache.org/jira/browse/HIVE-19740 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: heyang wang >Priority: Major > Attachments: hive-site.xml > > > I am using docker to deploy Hadoop 2.7, Hive 3.0 and Spark 2.3. > After starting all the docker image. Hive server2 can't start while > outputting the following error log: > 2018-05-30T14:13:53,832 WARN [main]: server.HiveServer2 > (HiveServer2.java:startHiveServer2(1041)) - Error starting HiveServer2 on > attempt 1, will retry in 6ms > java.lang.RuntimeException: Error initializing notification event poll > at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:269) > ~[hive-service-3.0.0.jar:3.0.0] > at > org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1013) > [hive-service-3.0.0.jar:3.0.0] > at > org.apache.hive.service.server.HiveServer2.access$1800(HiveServer2.java:134) > [hive-service-3.0.0.jar:3.0.0] > at > org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1282) > [hive-service-3.0.0.jar:3.0.0] > at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1126) > [hive-service-3.0.0.jar:3.0.0] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:1.8.0_131] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_131] > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131] > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > [hadoop-common-2.7.4.jar:?] > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > [hadoop-common-2.7.4.jar:?] > Caused by: java.io.IOException: org.apache.thrift.TApplicationException: > Internal error processing get_current_notificationEventId > at > org.apache.hadoop.hive.metastore.messaging.EventUtils$MSClientNotificationFetcher.getCurrentNotificationEventId(EventUtils.java:75) > ~[hive-exec-3.0.0.jar:3.0.0] > at > org.apache.hadoop.hive.ql.metadata.events.NotificationEventPoll.(NotificationEventPoll.java:103) > ~[hive-exec-3.0.0.jar:3.0.0] > at > org.apache.hadoop.hive.ql.metadata.events.NotificationEventPoll.initialize(NotificationEventPoll.java:59) > ~[hive-exec-3.0.0.jar:3.0.0] > at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:267) > ~[hive-service-3.0.0.jar:3.0.0] > ... 10 more > Caused by: org.apache.thrift.TApplicationException: Internal error > processing get_current_notificationEventId > at > org.apache.thrift.TApplicationException.read(TApplicationException.java:111) > ~[hive-exec-3.0.0.jar:3.0.0] > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) > ~[hive-exec-3.0.0.jar:3.0.0] > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_current_notificationEventId(ThriftHiveMetastore.java:5541) > ~[hive-exec-3.0.0.jar:3.0.0] > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_current_notificationEventId(ThriftHiveMetastore.java:5529) > ~[hive-exec-3.0.0.jar:3.0.0] > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getCurrentNotificationEventId(HiveMetaStoreClient.java:2713) > ~[hive-exec-3.0.0.jar:3.0.0] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:1.8.0_131] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_131] > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131] > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) > ~[hive-exec-3.0.0.jar:3.0.0] > at com.sun.proxy.$Proxy34.getCurrentNotificationEventId(Unknown Source) > ~[?:?] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:1.8.0_131] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_131] > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131] > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2763) > ~[hive-exec-3.0.0.jar:3.0.0] > at com.sun.proxy.$Proxy34.getCurrentNotificationEventId(Unknown Source) > ~[?:?] > at >
[jira] [Commented] (HIVE-19821) Distributed HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-19821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765529#comment-16765529 ] t oo commented on HIVE-19821: - bump > Distributed HiveServer2 > --- > > Key: HIVE-19821 > URL: https://issues.apache.org/jira/browse/HIVE-19821 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-19821.1.WIP.patch, HIVE-19821.2.WIP.patch, > HIVE-19821_ Distributed HiveServer2.pdf > > > HS2 deployments often hit OOM issues due to a number of factors: (1) too many > concurrent connections, (2) query that scan a large number of partitions have > to pull a lot of metadata into memory (e.g. a query reading thousands of > partitions requires loading thousands of partitions into memory), (3) very > large queries can take up a lot of heap space, especially during query > parsing. There are a number of other factors that cause HiveServer2 to run > out of memory, these are just some of the more commons ones. > Distributed HS2 proposes to do all query parsing, compilation, planning, and > execution coordination inside a dedicated container. This should > significantly decrease memory pressure on HS2 and allow HS2 to scale to a > larger number of concurrent users. > For HoS (and I think Hive-on-Tez) this just requires moving all query > compilation, planning, etc. inside the application master for the > corresponding Hive session. > The main benefit here is isolation. A poorly written Hive query cannot bring > down an entire HiveServer2 instance and force all other queries to fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21241:
---
Status: Open (was: Patch Available)
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch, HIVE-21241.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21241:
---
Status: Patch Available (was: Open)
Added new patch to address checkstyle complaints and also fix the unit tests.
Apparently a bunch of places are passing a 'null' value into the constructor of
this TimestampParser. The Hive code should really be better about not doing
that, but in this instance I'm not going to touch the offending callers.
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch, HIVE-21241.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21241:
---
Attachment: HIVE-21241.2.patch
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch, HIVE-21241.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21246:
---
Status: Patch Available (was: Open)
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a flag gets passed into utilities. If the class boolean is set,
> the passed-in SerDe is overwritten. This is not documented anywhere and it's
> weird to do it, just pass in the SerDe to use instead of the SerDe you don't
> want to use and a flag to change it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21246:
---
Attachment: HIVE-21246.1.patch
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a flag gets passed into utilities. If the class boolean is set,
> the passed-in SerDe is overwritten. This is not documented anywhere and it's
> weird to do it, just pass in the SerDe to use instead of the SerDe you don't
> want to use and a flag to change it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR reassigned HIVE-21246:
--
Assignee: BELUGA BEHR
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a flag gets passed into utilities. If the class boolean is set,
> the passed-in SerDe is overwritten. This is not documented anywhere and it's
> weird to do it, just pass in the SerDe to use instead of the SerDe you don't
> want to use and a flag to change it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-12408) SQLStdAuthorizer should not require external table creator to be owner of directory, in addition to rw permissions
[
https://issues.apache.org/jira/browse/HIVE-12408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765479#comment-16765479
]
t oo commented on HIVE-12408:
-
can this be ported to branch 2? It causes issues in AWS env
> SQLStdAuthorizer should not require external table creator to be owner of
> directory, in addition to rw permissions
> --
>
> Key: HIVE-12408
> URL: https://issues.apache.org/jira/browse/HIVE-12408
> Project: Hive
> Issue Type: Bug
> Components: Authorization, Security, SQLStandardAuthorization
>Affects Versions: 0.14.0
> Environment: HDP 2.2 + Kerberos
>Reporter: Hari Sekhon
>Assignee: Akira Ajisaka
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-12408.001.patch, HIVE-12408.002.patch
>
>
> When trying to create an external table via beeline in Hive using the
> SQLStdAuthorizer it expects the table creator to be the owner of the
> directory path and ignores the group rwx permission that is granted to the
> user.
> {code}Error: Error while compiling statement: FAILED:
> HiveAccessControlException Permission denied: Principal [name=hari,
> type=USER] does not have following privileges for operation CREATETABLE
> [[INSERT, DELETE, OBJECT OWNERSHIP] on Object [type=DFS_URI,
> name=/etl/path/to/hdfs/dir]] (state=42000,code=4){code}
> All it should be checking is read access to that directory.
> The directory owner requirement breaks the ability of more than one user to
> create external table definitions to a given location. For example this is a
> flume landing directory with json data, and the /etl tree is owned by the
> flume user. Even chowning the tree to another user would still break access
> to other users who are able to read the directory in hdfs but would still
> unable to create external tables on top of it.
> This looks like a remnant of the owner only access model in SQLStdAuth and is
> a separate issue to HIVE-11864 / HIVE-12324.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765478#comment-16765478
]
Hive QA commented on HIVE-21241:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958278/HIVE-21241.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 63 failed/errored test(s), 15792 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[json_serde1] (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[json_serde2] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[json_serde_qualified_types]
(batchId=97)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_mixed_partition_formats2]
(batchId=64)
org.apache.hadoop.hive.cli.TestMiniHiveKafkaCliDriver.testCliDriver[kafka_storage_handler]
(batchId=275)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testCompletedTxnComponents
(batchId=326)
org.apache.hive.hcatalog.data.TestJsonSerDe.testLooseJsonReadability
(batchId=215)
org.apache.hive.hcatalog.data.TestJsonSerDe.testMapValues (batchId=215)
org.apache.hive.hcatalog.data.TestJsonSerDe.testRW (batchId=215)
org.apache.hive.hcatalog.data.TestJsonSerDe.testRobustRead (batchId=215)
org.apache.hive.hcatalog.data.TestJsonSerDe.testUpperCaseKey (batchId=215)
org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[6]
(batchId=211)
org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned.testHCatDynamicPartitionedTable[6]
(batchId=211)
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[6]
(batchId=213)
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTable[6]
(batchId=213)
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatExternalDynamicCustomLocation[6]
(batchId=213)
org.apache.hive.hcatalog.mapreduce.TestHCatExternalNonPartitioned.testHCatNonPartitionedTable[6]
(batchId=214)
org.apache.hive.hcatalog.mapreduce.TestHCatExternalPartitioned.testHCatPartitionedTable[6]
(batchId=210)
org.apache.hive.hcatalog.mapreduce.TestHCatMutableDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[6]
(batchId=208)
org.apache.hive.hcatalog.mapreduce.TestHCatMutableDynamicPartitioned.testHCatDynamicPartitionedTable[6]
(batchId=208)
org.apache.hive.hcatalog.mapreduce.TestHCatMutableNonPartitioned.testHCatNonPartitionedTable[6]
(batchId=214)
org.apache.hive.hcatalog.mapreduce.TestHCatMutablePartitioned.testHCatPartitionedTable[6]
(batchId=212)
org.apache.hive.hcatalog.mapreduce.TestHCatNonPartitioned.testHCatNonPartitionedTable[6]
(batchId=209)
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[6]
(batchId=209)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testMapWithComplexData[6]
(batchId=207)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema[6]
(batchId=207)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag[6]
(batchId=207)
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.testReadDataFromEncryptedHiveTableByHCatMR[6]
(batchId=207)
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.testReadDataFromEncryptedHiveTableByPig[6]
(batchId=207)
org.apache.hive.hcatalog.pig.TestHCatStorer.testBagNStruct[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testDateCharTypes[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testDynamicPartitioningMultiPartColsInDataNoSpec[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testDynamicPartitioningMultiPartColsInDataPartialSpec[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testDynamicPartitioningMultiPartColsNoDataInDataNoSpec[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testEmptyStore[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testMultiPartColsInData[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testNoAlias[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testPartColsInData[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testPartitionPublish[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testStoreFuncAllSimpleTypes[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testStoreFuncSimple[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testStoreInPartiitonedTbl[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testStoreMultiTables[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testStoreWithNoCtorArgs[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testStoreWithNoSchema[6]
(batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteChar[6] (batc
[jira] [Comment Edited] (HIVE-21238) Fix Surrogate Key issue
[ https://issues.apache.org/jira/browse/HIVE-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765391#comment-16765391 ] Miklos Gergely edited comment on HIVE-21238 at 2/11/19 10:35 PM: - [~vgarg] I agree that this is not the best method, it would be much better if the function itself could get it from some sort of context (MapredContext maybe?). Unfortunately I don't see how it could be done at the moment. When I've implemented this function we faced the same issue, and at the end we chose this not-so-good solution. I'll create a Jira for the improvement of this. GenericUdfSurrogateKey can not be tested in the q file test envrionment as there is no write id present in that environment. was (Author: mgergely): [~vgarg] I agree that this is not the best method, it would be much better if the function itself could get it from some sort of context. Unfortunately I don't see how it could be done at the moment. When I've implemented this function we faced the same issue, and at the end we chose this not-so-good solution. I'll create a Jira for the improvement of this. > Fix Surrogate Key issue > --- > > Key: HIVE-21238 > URL: https://issues.apache.org/jira/browse/HIVE-21238 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-21238.01.patch, HIVE-21238.02.patch > > > When SemanticAnalyzer is trying to set the write id for the surrogate key > function it is looking for such functions only in the main operator, while it > should also look for it in it's parents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21238) Fix Surrogate Key issue
[ https://issues.apache.org/jira/browse/HIVE-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765391#comment-16765391 ] Miklos Gergely edited comment on HIVE-21238 at 2/11/19 10:36 PM: - [~vgarg] I agree that this is not the best method, it would be much better if the function itself could get it from some sort of context (MapredContext maybe?). Unfortunately I don't see how it could be done at the moment. When I've implemented this function we faced the same issue, and at the end we chose this not-so-good solution. I'll create a Jira for the improvement of this. GenericUdfSurrogateKey can not be tested in the q file test envrionment as there is no write id present in that environment. [~vgarg] could you please merge this? was (Author: mgergely): [~vgarg] I agree that this is not the best method, it would be much better if the function itself could get it from some sort of context (MapredContext maybe?). Unfortunately I don't see how it could be done at the moment. When I've implemented this function we faced the same issue, and at the end we chose this not-so-good solution. I'll create a Jira for the improvement of this. GenericUdfSurrogateKey can not be tested in the q file test envrionment as there is no write id present in that environment. > Fix Surrogate Key issue > --- > > Key: HIVE-21238 > URL: https://issues.apache.org/jira/browse/HIVE-21238 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-21238.01.patch, HIVE-21238.02.patch > > > When SemanticAnalyzer is trying to set the write id for the surrogate key > function it is looking for such functions only in the main operator, while it > should also look for it in it's parents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765400#comment-16765400
]
Hive QA commented on HIVE-21241:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} common: The patch generated 17 new + 0 unchanged - 14
fixed = 17 total (was 14) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} common generated 0 new + 64 unchanged - 1 fixed = 64
total (was 65) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} common generated 0 new + 26 unchanged - 1 fixed = 26
total (was 27) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16020/dev-support/hive-personality.sh
|
| git revision | master / 4ddc9de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16020/yetus/diff-checkstyle-common.txt
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16020/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21238) Fix Surrogate Key issue
[ https://issues.apache.org/jira/browse/HIVE-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765391#comment-16765391 ] Miklos Gergely commented on HIVE-21238: --- [~vgarg] I agree that this is not the best method, it would be much better if the function itself could get it from some sort of context. Unfortunately I don't see how it could be done at the moment. When I've implemented this function we faced the same issue, and at the end we chose this not-so-good solution. I'll create a Jira for the improvement of this. > Fix Surrogate Key issue > --- > > Key: HIVE-21238 > URL: https://issues.apache.org/jira/browse/HIVE-21238 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-21238.01.patch, HIVE-21238.02.patch > > > When SemanticAnalyzer is trying to set the write id for the surrogate key > function it is looking for such functions only in the main operator, while it > should also look for it in it's parents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21245) Make hive.fetch.output.serde Default to LazySimpleSerde
[
https://issues.apache.org/jira/browse/HIVE-21245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21245:
---
Attachment: HIVE-21245.1.patch
> Make hive.fetch.output.serde Default to LazySimpleSerde
> ---
>
> Key: HIVE-21245
> URL: https://issues.apache.org/jira/browse/HIVE-21245
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21245.1.patch
>
>
> For all intents and purposes, it already is:
> {code:java|title=HiveSessionImpl.java}
> private static final String FETCH_WORK_SERDE_CLASS =
> "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe";
> @Override
> public HiveConf getHiveConf() {
> sessionConf.setVar(HiveConf.ConfVars.HIVEFETCHOUTPUTSERDE,
> FETCH_WORK_SERDE_CLASS);
> return sessionConf;
> }
> {code}
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java#L489-L492
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}} altogether. It's a
> weird thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21245) Make hive.fetch.output.serde Default to LazySimpleSerde
[
https://issues.apache.org/jira/browse/HIVE-21245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21245:
---
Description:
For all intents and purposes, it already is:
{code:java|title=HiveSessionImpl.java}
private static final String FETCH_WORK_SERDE_CLASS =
"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe";
@Override
public HiveConf getHiveConf() {
sessionConf.setVar(HiveConf.ConfVars.HIVEFETCHOUTPUTSERDE,
FETCH_WORK_SERDE_CLASS);
return sessionConf;
}
{code}
https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java#L489-L492
The current default is:
{code}
// Serde for FetchTask
HIVEFETCHOUTPUTSERDE("hive.fetch.output.serde",
"org.apache.hadoop.hive.serde2.DelimitedJSONSerDe",
"The SerDe used by FetchTask to serialize the fetch output."),
{code}
Ultimately, I'd like to get rid of
{{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}} altogether. It's a weird
thing.
was:
For all intents and purposes, it already is:
{code:java|title=HiveSessionImpl.java}
private static final String FETCH_WORK_SERDE_CLASS =
"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe";
@Override
public HiveConf getHiveConf() {
sessionConf.setVar(HiveConf.ConfVars.HIVEFETCHOUTPUTSERDE,
FETCH_WORK_SERDE_CLASS);
return sessionConf;
}
{code}
https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java#L489-L492
Ultimately, I'd like to get rid of
{{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}} altogether. It's a weird
thing.
> Make hive.fetch.output.serde Default to LazySimpleSerde
> ---
>
> Key: HIVE-21245
> URL: https://issues.apache.org/jira/browse/HIVE-21245
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21245.1.patch
>
>
> For all intents and purposes, it already is:
> {code:java|title=HiveSessionImpl.java}
> private static final String FETCH_WORK_SERDE_CLASS =
> "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe";
> @Override
> public HiveConf getHiveConf() {
> sessionConf.setVar(HiveConf.ConfVars.HIVEFETCHOUTPUTSERDE,
> FETCH_WORK_SERDE_CLASS);
> return sessionConf;
> }
> {code}
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java#L489-L492
> The current default is:
> {code}
> // Serde for FetchTask
> HIVEFETCHOUTPUTSERDE("hive.fetch.output.serde",
> "org.apache.hadoop.hive.serde2.DelimitedJSONSerDe",
> "The SerDe used by FetchTask to serialize the fetch output."),
> {code}
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}} altogether. It's a
> weird thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21245) Make hive.fetch.output.serde Default to LazySimpleSerde
[
https://issues.apache.org/jira/browse/HIVE-21245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21245:
---
Status: Patch Available (was: Open)
> Make hive.fetch.output.serde Default to LazySimpleSerde
> ---
>
> Key: HIVE-21245
> URL: https://issues.apache.org/jira/browse/HIVE-21245
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21245.1.patch
>
>
> For all intents and purposes, it already is:
> {code:java|title=HiveSessionImpl.java}
> private static final String FETCH_WORK_SERDE_CLASS =
> "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe";
> @Override
> public HiveConf getHiveConf() {
> sessionConf.setVar(HiveConf.ConfVars.HIVEFETCHOUTPUTSERDE,
> FETCH_WORK_SERDE_CLASS);
> return sessionConf;
> }
> {code}
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java#L489-L492
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}} altogether. It's a
> weird thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21245) Make hive.fetch.output.serde Default to LazySimpleSerde
[
https://issues.apache.org/jira/browse/HIVE-21245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR reassigned HIVE-21245:
--
Assignee: BELUGA BEHR
> Make hive.fetch.output.serde Default to LazySimpleSerde
> ---
>
> Key: HIVE-21245
> URL: https://issues.apache.org/jira/browse/HIVE-21245
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Fix For: 4.0.0
>
>
> For all intents and purposes, it already is:
> {code:java|title=HiveSessionImpl.java}
> private static final String FETCH_WORK_SERDE_CLASS =
> "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe";
> @Override
> public HiveConf getHiveConf() {
> sessionConf.setVar(HiveConf.ConfVars.HIVEFETCHOUTPUTSERDE,
> FETCH_WORK_SERDE_CLASS);
> return sessionConf;
> }
> {code}
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java#L489-L492
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}} altogether. It's a
> weird thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21238) Fix Surrogate Key issue
[
https://issues.apache.org/jira/browse/HIVE-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765385#comment-16765385
]
Hive QA commented on HIVE-21238:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958241/HIVE-21238.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15784 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16019/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16019/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16019/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958241 - PreCommit-HIVE-Build
> Fix Surrogate Key issue
> ---
>
> Key: HIVE-21238
> URL: https://issues.apache.org/jira/browse/HIVE-21238
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 3.0.1
>
> Attachments: HIVE-21238.01.patch, HIVE-21238.02.patch
>
>
> When SemanticAnalyzer is trying to set the write id for the surrogate key
> function it is looking for such functions only in the main operator, while it
> should also look for it in it's parents.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21218) KafkaSerDe doesn't support topics created via Confluent Avro serializer
[
https://issues.apache.org/jira/browse/HIVE-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765379#comment-16765379
]
slim bouguerra edited comment on HIVE-21218 at 2/11/19 9:07 PM:
[~milan.baran] can you please generate the patch with --no-prefix option ?
{code}
git diff master --no-prefix > HIVE-21218.2.patch
{code}
also make sure that your master branch is up to date.
was (Author: bslim):
[~milan.baran] can you please generate the patch with --no-prefix option ?
{code} git diff master --no-prefix > HIVE-21218.2.patch \{code}
also make sure that your master branch is up to date.
> KafkaSerDe doesn't support topics created via Confluent Avro serializer
> ---
>
> Key: HIVE-21218
> URL: https://issues.apache.org/jira/browse/HIVE-21218
> Project: Hive
> Issue Type: Bug
> Components: kafka integration, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Milan Baran
>Assignee: Milan Baran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21218.patch
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> According to [Google
> groups|https://groups.google.com/forum/#!topic/confluent-platform/JYhlXN0u9_A]
> the Confluent avro serialzier uses propertiary format for kafka value -
> <4 bytes of schema ID> conforms to schema>.
> This format does not cause any problem for Confluent kafka deserializer which
> respect the format however for hive kafka handler its bit a problem to
> correctly deserialize kafka value, because Hive uses custom deserializer from
> bytes to objects and ignores kafka consumer ser/deser classes provided via
> table property.
> It would be nice to support Confluent format with magic byte.
> Also it would be great to support Schema registry as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21218) KafkaSerDe doesn't support topics created via Confluent Avro serializer
[
https://issues.apache.org/jira/browse/HIVE-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765379#comment-16765379
]
slim bouguerra commented on HIVE-21218:
---
[~milan.baran] can you please generate the patch with --no-prefix option ?
{code} git diff master --no-prefix > HIVE-21218.2.patch \{code}
also make sure that your master branch is up to date.
> KafkaSerDe doesn't support topics created via Confluent Avro serializer
> ---
>
> Key: HIVE-21218
> URL: https://issues.apache.org/jira/browse/HIVE-21218
> Project: Hive
> Issue Type: Bug
> Components: kafka integration, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Milan Baran
>Assignee: Milan Baran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21218.patch
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> According to [Google
> groups|https://groups.google.com/forum/#!topic/confluent-platform/JYhlXN0u9_A]
> the Confluent avro serialzier uses propertiary format for kafka value -
> <4 bytes of schema ID> conforms to schema>.
> This format does not cause any problem for Confluent kafka deserializer which
> respect the format however for hive kafka handler its bit a problem to
> correctly deserialize kafka value, because Hive uses custom deserializer from
> bytes to objects and ignores kafka consumer ser/deser classes provided via
> table property.
> It would be nice to support Confluent format with magic byte.
> Also it would be great to support Schema registry as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21238) Fix Surrogate Key issue
[
https://issues.apache.org/jira/browse/HIVE-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765374#comment-16765374
]
Hive QA commented on HIVE-21238:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2298 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16019/dev-support/hive-personality.sh
|
| git revision | master / 4ddc9de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16019/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix Surrogate Key issue
> ---
>
> Key: HIVE-21238
> URL: https://issues.apache.org/jira/browse/HIVE-21238
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 3.0.1
>
> Attachments: HIVE-21238.01.patch, HIVE-21238.02.patch
>
>
> When SemanticAnalyzer is trying to set the write id for the surrogate key
> function it is looking for such functions only in the main operator, while it
> should also look for it in it's parents.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21223) CachedStore returns null partition when partition does not exist
[
https://issues.apache.org/jira/browse/HIVE-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21223:
-
Attachment: HIVE-21223.2.patch
> CachedStore returns null partition when partition does not exist
>
>
> Key: HIVE-21223
> URL: https://issues.apache.org/jira/browse/HIVE-21223
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21223.1.patch, HIVE-21223.2.patch
>
>
> CachedStore can return null partition for getPartitionWithAuth() when
> partition does not exist. null value serialization in thrift will break the
> connection. Instead if partition does not exist it should throw
> NoSuchObjectException.
> Clients will see this exception
> {code:java}
> org.apache.thrift.TApplicationException: get_partition_with_auth failed:
> unknown result
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition_with_auth(ThriftHiveMetastore.java:3017)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition_with_auth(ThriftHiveMetastore.java:2990)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1679)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1671)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2976)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.SynchronizedMetaStoreClient.getPartitionWithAuthInfo(SynchronizedMetaStoreClient.java:101)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2870)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2835)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1950)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2490)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2481)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21244) NPE in Hive Proto Logger
[ https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765366#comment-16765366 ] Prasanth Jayachandran commented on HIVE-21244: -- [~harishjp]/[~anishek] I tested this patch on cluster and it seems to work correctly. Can you please review? > NPE in Hive Proto Logger > > > Key: HIVE-21244 > URL: https://issues.apache.org/jira/browse/HIVE-21244 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21244.1.patch > > > [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308] > can cause NPE. There is no uncaught exception handler for this thread. This > NPE can silently fail and drop the event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21244) NPE in Hive Proto Logger
[ https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21244: - Attachment: HIVE-21244.1.patch > NPE in Hive Proto Logger > > > Key: HIVE-21244 > URL: https://issues.apache.org/jira/browse/HIVE-21244 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21244.1.patch > > > [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308] > can cause NPE. There is no uncaught exception handler for this thread. This > NPE can silently fail and drop the event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21244) NPE in Hive Proto Logger
[ https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21244: - Status: Patch Available (was: Open) > NPE in Hive Proto Logger > > > Key: HIVE-21244 > URL: https://issues.apache.org/jira/browse/HIVE-21244 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21244.1.patch > > > [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308] > can cause NPE. There is no uncaught exception handler for this thread. This > NPE can silently fail and drop the event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21244) NPE in Hive Proto Logger
[ https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-21244: > NPE in Hive Proto Logger > > > Key: HIVE-21244 > URL: https://issues.apache.org/jira/browse/HIVE-21244 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308] > can cause NPE. There is no uncaught exception handler for this thread. This > NPE can silently fail and drop the event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21239) Beeline help LDAP connection example incorrect
[
https://issues.apache.org/jira/browse/HIVE-21239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765352#comment-16765352
]
Hive QA commented on HIVE-21239:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958265/HIVE-21239.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15784 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=109)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16018/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16018/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16018/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958265 - PreCommit-HIVE-Build
> Beeline help LDAP connection example incorrect
> --
>
> Key: HIVE-21239
> URL: https://issues.apache.org/jira/browse/HIVE-21239
> Project: Hive
> Issue Type: Bug
> Environment: This was tested on a test environment with SSL and LDAP
> authentication enabled, and seems to be reproducible on any environment with
> LDAP authentication available in HiveServer2.
>Reporter: Zsolt Herczeg
>Assignee: Zoltan Chovan
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21239.patch
>
>
> There's the following connection example string in the beeline -h command
> output:
>
> {code:java}
> 5. Connect using LDAP authentication
> $ beeline -u jdbc:hive2://hs2.local:10013/default
>
> {code}
> When a user attempts to connect like above, it'll fail with LDAP
> authentication failure. This is because username and passwords are not picked
> up in the shown form. A working example would be:
> {code:java}
> $ beeline -n -p -u
> jdbc:hive2://hs2.local:10013/default
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-21193) Support LZO Compression with CombineHiveInputFormat
[
https://issues.apache.org/jira/browse/HIVE-21193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-21193 started by BELUGA BEHR.
--
> Support LZO Compression with CombineHiveInputFormat
> ---
>
> Key: HIVE-21193
> URL: https://issues.apache.org/jira/browse/HIVE-21193
> Project: Hive
> Issue Type: Improvement
> Components: Compression
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21193.1.patch
>
>
> In regards to LZO compression with Hive...
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LZO
> It does not work out of the box if there are {{.lzo.index}} files present.
> As I understand it, this is because of the default Hive input format
> {{CombineHiveInputFormat}} does not handle this correctly. It does not like
> that there are a mix of data files and some index files, it lumps them
> altogether when making the combined splits and Mappers fail when they try to
> process the {{.lzo.index}} files as data. When using the original
> {{HiveInputFormat}}, it correctly identifies the {{.lzo.index}} files because
> it considers each file individually.
> Allow {{CombineHiveInputFormat}} to short-circuit LZO files and to not
> combine them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-21195) Review of DefaultGraphWalker Class
[
https://issues.apache.org/jira/browse/HIVE-21195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-21195 started by BELUGA BEHR.
--
> Review of DefaultGraphWalker Class
> --
>
> Key: HIVE-21195
> URL: https://issues.apache.org/jira/browse/HIVE-21195
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21195.1.patch
>
>
> {code:java}
> protected final List toWalk = new ArrayList();
> ...
> while (toWalk.size() > 0) {
> Node nd = toWalk.remove(0);
> {code}
> Every time this loop runs, the first item of a list is removed. For an
> {{ArrayList}}, this means that every time the first item is removed, all of
> the remaining items in the list are copied down one position so that the
> first item is always at array index 0. This is expensive in a tight loop.
> Use a {{Queue}} implementation that does not have this behavior.
> {{ArrayDeque}}
> {quote}
> This class is likely to be faster than Stack when used as a stack, and faster
> than LinkedList when used as a queue.
> {quote}
> https://docs.oracle.com/javase/7/docs/api/java/util/ArrayDeque.html
> Add a little bit extra cleanup since it's being looked at.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21206) Bootstrap replication is slow as it opens lot of metastore connections.
[
https://issues.apache.org/jira/browse/HIVE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-21206:
Attachment: HIVE-21206.03.patch
> Bootstrap replication is slow as it opens lot of metastore connections.
> ---
>
> Key: HIVE-21206
> URL: https://issues.apache.org/jira/browse/HIVE-21206
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21206.01.patch, HIVE-21206.02.patch,
> HIVE-21206.03.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive bootstrap replication of 1TB data onprem to onprem in Hive3 is running
> slower compared to Hive2.
> Time taken for bootstrap replication of table with 1000 partitions are as
> below:
> ||Hive2- Hive2|| Hive3 - Hive3 ||
> |Bootstrap: 7m| BootStrap: 17m |
> Every MoveTask is closing and opening new metastore connection which is
> causing slow down.
> {code}
> 2019-02-08T12:28:30,174 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-5:MOVE] in serial mode
> 2019-02-08T12:28:30,177 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table1 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/.hive-staging_hive_2019-02-08_12-28-23_584_1482331698286040936-3/-ext-10001
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://ctr-e139-1542663976389-62755-01-14.hwx.site:9083
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - HMSC::open(): Could not find delegation
> token. Creating KERBEROS-based thrift connection.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 4
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Connected to metastore.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> ugi=hive/ctr-e139-1542663976389-62755-01-14.hwx.s...@hwqe.hortonworks.com
> (auth:KERBEROS) retries=24 delay=5 lifetime=0
> 2019-02-08T12:28:30,325 INFO
> [org.apache.ranger.audit.queue.AuditBatchQueue1]: provider.BaseAuditHandler
> (:()) - Audit Status Log: name=hiveServer2.async.multi_dest.batch,
> finalDestination=hiveServer2.async.multi_dest.batch.solr, interval=01:00.002
> minutes, events=2, succcessCount=1, totalEvents=56, totalSuccessCount=25
> 2019-02-08T12:28:30,520 INFO [HiveServer2-Background-Pool: Thread-1134]:
> common.FileUtils (FileUtils.java:mkdir(580)) - Creating directory if it
> doesn't exist:
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/base_001
> 2019-02-08T12:28:31,245 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-11:MOVE] in serial mode
> 2019-02-08T12:28:31,245 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore,
> current connections: 3
> 2019-02-08T12:28:31,246 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table2 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table2/.hive-staging_hive_2019-02-08_12-28-23_810_7457138692783022870-3/-ext-10002
> 2019-02-08T12:28:31,327 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://ctr-e139-1542663976389-62755-01-14.hwx.site:9083
> 2019-02-08T12:28:31,327 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - HMSC::open(): Could not find delegation
> token. Creating KERBEROS-based thrift connection.
> 2019-02-08T12:28:31,336 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 4
> 2019-02-08T12:28:31,337 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Connected to metastore.
> 2019-02-08T12:28:31,337 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient
[jira] [Commented] (HIVE-21239) Beeline help LDAP connection example incorrect
[
https://issues.apache.org/jira/browse/HIVE-21239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765302#comment-16765302
]
Hive QA commented on HIVE-21239:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16018/dev-support/hive-personality.sh
|
| git revision | master / 4ddc9de |
| Default Java | 1.8.0_111 |
| modules | C: beeline U: beeline |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16018/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Beeline help LDAP connection example incorrect
> --
>
> Key: HIVE-21239
> URL: https://issues.apache.org/jira/browse/HIVE-21239
> Project: Hive
> Issue Type: Bug
> Environment: This was tested on a test environment with SSL and LDAP
> authentication enabled, and seems to be reproducible on any environment with
> LDAP authentication available in HiveServer2.
>Reporter: Zsolt Herczeg
>Assignee: Zoltan Chovan
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21239.patch
>
>
> There's the following connection example string in the beeline -h command
> output:
>
> {code:java}
> 5. Connect using LDAP authentication
> $ beeline -u jdbc:hive2://hs2.local:10013/default
>
> {code}
> When a user attempts to connect like above, it'll fail with LDAP
> authentication failure. This is because username and passwords are not picked
> up in the shown form. A working example would be:
> {code:java}
> $ beeline -n -p -u
> jdbc:hive2://hs2.local:10013/default
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21223) CachedStore returns null partition when partition does not exist
[
https://issues.apache.org/jira/browse/HIVE-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765293#comment-16765293
]
Vaibhav Gumashta commented on HIVE-21223:
-
+1
> CachedStore returns null partition when partition does not exist
>
>
> Key: HIVE-21223
> URL: https://issues.apache.org/jira/browse/HIVE-21223
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21223.1.patch
>
>
> CachedStore can return null partition for getPartitionWithAuth() when
> partition does not exist. null value serialization in thrift will break the
> connection. Instead if partition does not exist it should throw
> NoSuchObjectException.
> Clients will see this exception
> {code:java}
> org.apache.thrift.TApplicationException: get_partition_with_auth failed:
> unknown result
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition_with_auth(ThriftHiveMetastore.java:3017)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition_with_auth(ThriftHiveMetastore.java:2990)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1679)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1671)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2976)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.SynchronizedMetaStoreClient.getPartitionWithAuthInfo(SynchronizedMetaStoreClient.java:101)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2870)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2835)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1950)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2490)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2481)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21206) Bootstrap replication is slow as it opens lot of metastore connections.
[
https://issues.apache.org/jira/browse/HIVE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-21206:
Status: Patch Available (was: Open)
03.patch corrected the test code.
> Bootstrap replication is slow as it opens lot of metastore connections.
> ---
>
> Key: HIVE-21206
> URL: https://issues.apache.org/jira/browse/HIVE-21206
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21206.01.patch, HIVE-21206.02.patch,
> HIVE-21206.03.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive bootstrap replication of 1TB data onprem to onprem in Hive3 is running
> slower compared to Hive2.
> Time taken for bootstrap replication of table with 1000 partitions are as
> below:
> ||Hive2- Hive2|| Hive3 - Hive3 ||
> |Bootstrap: 7m| BootStrap: 17m |
> Every MoveTask is closing and opening new metastore connection which is
> causing slow down.
> {code}
> 2019-02-08T12:28:30,174 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-5:MOVE] in serial mode
> 2019-02-08T12:28:30,177 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table1 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/.hive-staging_hive_2019-02-08_12-28-23_584_1482331698286040936-3/-ext-10001
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://ctr-e139-1542663976389-62755-01-14.hwx.site:9083
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - HMSC::open(): Could not find delegation
> token. Creating KERBEROS-based thrift connection.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 4
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Connected to metastore.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> ugi=hive/ctr-e139-1542663976389-62755-01-14.hwx.s...@hwqe.hortonworks.com
> (auth:KERBEROS) retries=24 delay=5 lifetime=0
> 2019-02-08T12:28:30,325 INFO
> [org.apache.ranger.audit.queue.AuditBatchQueue1]: provider.BaseAuditHandler
> (:()) - Audit Status Log: name=hiveServer2.async.multi_dest.batch,
> finalDestination=hiveServer2.async.multi_dest.batch.solr, interval=01:00.002
> minutes, events=2, succcessCount=1, totalEvents=56, totalSuccessCount=25
> 2019-02-08T12:28:30,520 INFO [HiveServer2-Background-Pool: Thread-1134]:
> common.FileUtils (FileUtils.java:mkdir(580)) - Creating directory if it
> doesn't exist:
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/base_001
> 2019-02-08T12:28:31,245 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-11:MOVE] in serial mode
> 2019-02-08T12:28:31,245 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore,
> current connections: 3
> 2019-02-08T12:28:31,246 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table2 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table2/.hive-staging_hive_2019-02-08_12-28-23_810_7457138692783022870-3/-ext-10002
> 2019-02-08T12:28:31,327 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://ctr-e139-1542663976389-62755-01-14.hwx.site:9083
> 2019-02-08T12:28:31,327 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - HMSC::open(): Could not find delegation
> token. Creating KERBEROS-based thrift connection.
> 2019-02-08T12:28:31,336 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 4
> 2019-02-08T12:28:31,337 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Connected to metastore.
> 2019-02-08T12:28:31,337 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.RetryingMetaSto
[jira] [Updated] (HIVE-21206) Bootstrap replication is slow as it opens lot of metastore connections.
[
https://issues.apache.org/jira/browse/HIVE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-21206:
Status: Open (was: Patch Available)
> Bootstrap replication is slow as it opens lot of metastore connections.
> ---
>
> Key: HIVE-21206
> URL: https://issues.apache.org/jira/browse/HIVE-21206
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21206.01.patch, HIVE-21206.02.patch,
> HIVE-21206.03.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive bootstrap replication of 1TB data onprem to onprem in Hive3 is running
> slower compared to Hive2.
> Time taken for bootstrap replication of table with 1000 partitions are as
> below:
> ||Hive2- Hive2|| Hive3 - Hive3 ||
> |Bootstrap: 7m| BootStrap: 17m |
> Every MoveTask is closing and opening new metastore connection which is
> causing slow down.
> {code}
> 2019-02-08T12:28:30,174 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-5:MOVE] in serial mode
> 2019-02-08T12:28:30,177 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table1 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/.hive-staging_hive_2019-02-08_12-28-23_584_1482331698286040936-3/-ext-10001
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://ctr-e139-1542663976389-62755-01-14.hwx.site:9083
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - HMSC::open(): Could not find delegation
> token. Creating KERBEROS-based thrift connection.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 4
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Connected to metastore.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> ugi=hive/ctr-e139-1542663976389-62755-01-14.hwx.s...@hwqe.hortonworks.com
> (auth:KERBEROS) retries=24 delay=5 lifetime=0
> 2019-02-08T12:28:30,325 INFO
> [org.apache.ranger.audit.queue.AuditBatchQueue1]: provider.BaseAuditHandler
> (:()) - Audit Status Log: name=hiveServer2.async.multi_dest.batch,
> finalDestination=hiveServer2.async.multi_dest.batch.solr, interval=01:00.002
> minutes, events=2, succcessCount=1, totalEvents=56, totalSuccessCount=25
> 2019-02-08T12:28:30,520 INFO [HiveServer2-Background-Pool: Thread-1134]:
> common.FileUtils (FileUtils.java:mkdir(580)) - Creating directory if it
> doesn't exist:
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/base_001
> 2019-02-08T12:28:31,245 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-11:MOVE] in serial mode
> 2019-02-08T12:28:31,245 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore,
> current connections: 3
> 2019-02-08T12:28:31,246 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table2 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table2/.hive-staging_hive_2019-02-08_12-28-23_810_7457138692783022870-3/-ext-10002
> 2019-02-08T12:28:31,327 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://ctr-e139-1542663976389-62755-01-14.hwx.site:9083
> 2019-02-08T12:28:31,327 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - HMSC::open(): Could not find delegation
> token. Creating KERBEROS-based thrift connection.
> 2019-02-08T12:28:31,336 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 4
> 2019-02-08T12:28:31,337 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Connected to metastore.
> 2019-02-08T12:28:31,337 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreC
[jira] [Commented] (HIVE-21235) LLAP: make the name of log4j2 properties file configurable
[ https://issues.apache.org/jira/browse/HIVE-21235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765292#comment-16765292 ] Ashutosh Chauhan commented on HIVE-21235: - +1 > LLAP: make the name of log4j2 properties file configurable > -- > > Key: HIVE-21235 > URL: https://issues.apache.org/jira/browse/HIVE-21235 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21235.1.patch > > > For llap daemon, the name of llap-daemon-log4j2.properties is fixed. If a > conf dir and jar contain the same filename, it will mess up log4j2 > initialization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21238) Fix Surrogate Key issue
[ https://issues.apache.org/jira/browse/HIVE-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765271#comment-16765271 ] Vineet Garg commented on HIVE-21238: Unfortunately the rewrite to add GenericUDFSurrogateKey happens at AST level when we don't have writeId. Can't think of any better way at the moment. [~mgergely] Can you create another JIRA to improve this? Also can you add q test for this? > Fix Surrogate Key issue > --- > > Key: HIVE-21238 > URL: https://issues.apache.org/jira/browse/HIVE-21238 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-21238.01.patch, HIVE-21238.02.patch > > > When SemanticAnalyzer is trying to set the write id for the surrogate key > function it is looking for such functions only in the main operator, while it > should also look for it in it's parents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21206) Bootstrap replication is slow as it opens lot of metastore connections.
[
https://issues.apache.org/jira/browse/HIVE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765272#comment-16765272
]
Hive QA commented on HIVE-21206:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958247/HIVE-21206.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 15784 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
(batchId=230)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16017/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16017/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16017/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958247 - PreCommit-HIVE-Build
> Bootstrap replication is slow as it opens lot of metastore connections.
> ---
>
> Key: HIVE-21206
> URL: https://issues.apache.org/jira/browse/HIVE-21206
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21206.01.patch, HIVE-21206.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive bootstrap replication of 1TB data onprem to onprem in Hive3 is running
> slower compared to Hive2.
> Time taken for bootstrap replication of table with 1000 partitions are as
> below:
> ||Hive2- Hive2|| Hive3 - Hive3 ||
> |Bootstrap: 7m| BootStrap: 17m |
> Every MoveTask is closing and opening new metastore connection which is
> causing slow down.
> {code}
> 2019-02-08T12:28:30,174 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-5:MOVE] in serial mode
> 2019-02-08T12:28:30,177 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table1 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/.hive-staging_hive_2019-02-08_12-28-23_584_1482331698286040936-3/-ext-10001
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://ctr-e139-1542663976389-62755-01-14.hwx.site:9083
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - HMSC::open(): Could not find delegation
> token. Creating KERBEROS-based thrift connection.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 4
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Connected to metastore.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> ugi=hive/ctr-e139-1542663976389-62755-01-14.hwx.s...@hwqe
[jira] [Commented] (HIVE-21238) Fix Surrogate Key issue
[ https://issues.apache.org/jira/browse/HIVE-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765269#comment-16765269 ] Vineet Garg commented on HIVE-21238: [~mgergely] Hmm now for every query irrespective of being INSERT or SELECT we will traverse all the operators and all the column exprs within operator to get to GenericUDFSurrogateKey. I wonder if there is a way better way to pass on writeId to GenericUDFSurrogateKey. > Fix Surrogate Key issue > --- > > Key: HIVE-21238 > URL: https://issues.apache.org/jira/browse/HIVE-21238 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-21238.01.patch, HIVE-21238.02.patch > > > When SemanticAnalyzer is trying to set the write id for the surrogate key > function it is looking for such functions only in the main operator, while it > should also look for it in it's parents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20375) Json SerDe ignoring the timestamp.formats property
[ https://issues.apache.org/jira/browse/HIVE-20375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765263#comment-16765263 ] BELUGA BEHR commented on HIVE-20375: Check out... HIVE-21240 > Json SerDe ignoring the timestamp.formats property > -- > > Key: HIVE-20375 > URL: https://issues.apache.org/jira/browse/HIVE-20375 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: slim bouguerra >Assignee: Ashutosh Chauhan >Priority: Major > > JsonSerd is supposed to accept "timestamp.formats" SerDe property to allow > different timestamp formats, after recent refactor I see that this is not > working anymore. > Looking at the code I can see that The serde is not using the constructed > parser with added format > https://github.com/apache/hive/blob/1105ef3974d8a324637d3d35881a739af3aeb382/serde/src/java/org/apache/hadoop/hive/serde2/json/HiveJsonStructReader.java#L82 > But instead it is using Converter > https://github.com/apache/hive/blob/1105ef3974d8a324637d3d35881a739af3aeb382/serde/src/java/org/apache/hadoop/hive/serde2/json/HiveJsonStructReader.java#L324 > Then converter is using > org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter.TimestampConverter > This converter does not have any knowledge about user formats or what so > ever... > It is using this static converter > org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils#getTimestampFromString -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21206) Bootstrap replication is slow as it opens lot of metastore connections.
[
https://issues.apache.org/jira/browse/HIVE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765229#comment-16765229
]
Hive QA commented on HIVE-21206:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 2298 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} ql: The patch generated 0 new + 342 unchanged - 1
fixed = 342 total (was 343) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16017/dev-support/hive-personality.sh
|
| git revision | master / 4ddc9de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16017/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Bootstrap replication is slow as it opens lot of metastore connections.
> ---
>
> Key: HIVE-21206
> URL: https://issues.apache.org/jira/browse/HIVE-21206
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21206.01.patch, HIVE-21206.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive bootstrap replication of 1TB data onprem to onprem in Hive3 is running
> slower compared to Hive2.
> Time taken for bootstrap replication of table with 1000 partitions are as
> below:
> ||Hive2- Hive2|| Hive3 - Hive3 ||
> |Bootstrap: 7m| BootStrap: 17m |
> Every MoveTask is closing and opening new metastore connection which is
> causing slow down.
> {code}
> 2019-02-08T12:28:30,174 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-5:MOVE] in serial mode
> 2019-02-08T12:28:30,177 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table1 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/.hive-staging_hive_2019-02-08_12-28-23_584_1482331698286040936-3/-ext-10001
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore
[jira] [Commented] (HIVE-21218) KafkaSerDe doesn't support topics created via Confluent Avro serializer
[
https://issues.apache.org/jira/browse/HIVE-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765201#comment-16765201
]
Hive QA commented on HIVE-21218:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958256/HIVE-21218.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16016/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16016/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16016/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-02-11 17:36:54.044
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16016/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-02-11 17:36:54.048
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 4ddc9de HIVE-21233: ACID: Make validate_acid_sort_order()
Stateful UDF (Gopal V, reviewed by Vaibhav Gumashta)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 4ddc9de HIVE-21233: ACID: Make validate_acid_sort_order()
Stateful UDF (Gopal V, reviewed by Vaibhav Gumashta)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-02-11 17:36:54.796
+ rm -rf ../yetus_PreCommit-HIVE-Build-16016
+ mkdir ../yetus_PreCommit-HIVE-Build-16016
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16016
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16016/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:13: trailing whitespace.
If you use Confluent Avro serialzier/deserializer with schema registry you may
want to remove 5 bytes from beginning that represents magic byte + schema ID
from registry.
warning: 1 line adds whitespace errors.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc6240499844750741780.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc6240499844750741780.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(process-resource-bundles) on project hive-shims: Execution
process-resource-bundles of goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed.
ConcurrentModificationException -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-shim
[jira] [Updated] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21241:
---
Status: Patch Available (was: Open)
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16716) Clean up javadoc from errors in module ql
[
https://issues.apache.org/jira/browse/HIVE-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765181#comment-16765181
]
Hive QA commented on HIVE-16716:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958245/HIVE-16716.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15784 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[1]
(batchId=209)
org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=260)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16015/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16015/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16015/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958245 - PreCommit-HIVE-Build
> Clean up javadoc from errors in module ql
> -
>
> Key: HIVE-16716
> URL: https://issues.apache.org/jira/browse/HIVE-16716
> Project: Hive
> Issue Type: Sub-task
>Reporter: Janos Gub
>Assignee: Robert Kucsora
>Priority: Major
> Attachments: HIVE-16716-v2.patch, HIVE-16716.2.patch,
> HIVE-16716.3.patch, HIVE-16716.4.patch, HIVE-16716.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21238) Fix Surrogate Key issue
[ https://issues.apache.org/jira/browse/HIVE-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21238: -- Status: Patch Available (was: Open) > Fix Surrogate Key issue > --- > > Key: HIVE-21238 > URL: https://issues.apache.org/jira/browse/HIVE-21238 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-21238.01.patch, HIVE-21238.02.patch > > > When SemanticAnalyzer is trying to set the write id for the surrogate key > function it is looking for such functions only in the main operator, while it > should also look for it in it's parents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21241:
---
Attachment: HIVE-21241.1.patch
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-21241:
--
Labels: pull-request-available (was: )
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?focusedWorklogId=197053&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-197053
]
ASF GitHub Bot logged work on HIVE-21241:
-
Author: ASF GitHub Bot
Created on: 11/Feb/19 17:13
Start Date: 11/Feb/19 17:13
Worklog Time Spent: 10m
Work Description: BELUGABEHR commented on pull request #531: HIVE-21241:
Migrate TimeStamp Parser From Joda Time
URL: https://github.com/apache/hive/pull/531
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 197053)
Time Spent: 10m
Remaining Estimate: 0h
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR reassigned HIVE-21241:
--
Assignee: BELUGA BEHR
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Fix For: 4.0.0
>
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21239) Beeline help LDAP connection example incorrect
[
https://issues.apache.org/jira/browse/HIVE-21239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-21239:
-
Status: Patch Available (was: Open)
> Beeline help LDAP connection example incorrect
> --
>
> Key: HIVE-21239
> URL: https://issues.apache.org/jira/browse/HIVE-21239
> Project: Hive
> Issue Type: Bug
> Environment: This was tested on a test environment with SSL and LDAP
> authentication enabled, and seems to be reproducible on any environment with
> LDAP authentication available in HiveServer2.
>Reporter: Zsolt Herczeg
>Assignee: Zoltan Chovan
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21239.patch
>
>
> There's the following connection example string in the beeline -h command
> output:
>
> {code:java}
> 5. Connect using LDAP authentication
> $ beeline -u jdbc:hive2://hs2.local:10013/default
>
> {code}
> When a user attempts to connect like above, it'll fail with LDAP
> authentication failure. This is because username and passwords are not picked
> up in the shown form. A working example would be:
> {code:java}
> $ beeline -n -p -u
> jdbc:hive2://hs2.local:10013/default
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21239) Beeline help LDAP connection example incorrect
[
https://issues.apache.org/jira/browse/HIVE-21239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765161#comment-16765161
]
Peter Vary commented on HIVE-21239:
---
[~zchovan]: I usually prefer to have the pre-commit tests anyway.
> Beeline help LDAP connection example incorrect
> --
>
> Key: HIVE-21239
> URL: https://issues.apache.org/jira/browse/HIVE-21239
> Project: Hive
> Issue Type: Bug
> Environment: This was tested on a test environment with SSL and LDAP
> authentication enabled, and seems to be reproducible on any environment with
> LDAP authentication available in HiveServer2.
>Reporter: Zsolt Herczeg
>Assignee: Zoltan Chovan
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21239.patch
>
>
> There's the following connection example string in the beeline -h command
> output:
>
> {code:java}
> 5. Connect using LDAP authentication
> $ beeline -u jdbc:hive2://hs2.local:10013/default
>
> {code}
> When a user attempts to connect like above, it'll fail with LDAP
> authentication failure. This is because username and passwords are not picked
> up in the shown form. A working example would be:
> {code:java}
> $ beeline -n -p -u
> jdbc:hive2://hs2.local:10013/default
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16716) Clean up javadoc from errors in module ql
[
https://issues.apache.org/jira/browse/HIVE-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765132#comment-16765132
]
Hive QA commented on HIVE-16716:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
58s{color} | {color:blue} ql in master has 2298 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
8s{color} | {color:red} ql: The patch generated 5 new + 2435 unchanged - 6
fixed = 2440 total (was 2441) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
59s{color} | {color:red} ql generated 41 new + 59 unchanged - 41 fixed = 100
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16015/dev-support/hive-personality.sh
|
| git revision | master / 4ddc9de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16015/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16015/yetus/whitespace-eol.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16015/yetus/diff-javadoc-javadoc-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16015/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean up javadoc from errors in module ql
> -
>
> Key: HIVE-16716
> URL: https://issues.apache.org/jira/browse/HIVE-16716
> Project: Hive
> Issue Type: Sub-task
>Reporter: Janos Gub
>Assignee: Robert Kucsora
>Priority: Major
> Attachments: HIVE-16716-v2.patch, HIVE-16716.2.patch,
> HIVE-16716.3.patch, HIVE-16716.4.patch, HIVE-16716.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21206) Bootstrap replication is slow as it opens lot of metastore connections.
[
https://issues.apache.org/jira/browse/HIVE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-21206:
Status: Patch Available (was: Open)
> Bootstrap replication is slow as it opens lot of metastore connections.
> ---
>
> Key: HIVE-21206
> URL: https://issues.apache.org/jira/browse/HIVE-21206
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21206.01.patch, HIVE-21206.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive bootstrap replication of 1TB data onprem to onprem in Hive3 is running
> slower compared to Hive2.
> Time taken for bootstrap replication of table with 1000 partitions are as
> below:
> ||Hive2- Hive2|| Hive3 - Hive3 ||
> |Bootstrap: 7m| BootStrap: 17m |
> Every MoveTask is closing and opening new metastore connection which is
> causing slow down.
> {code}
> 2019-02-08T12:28:30,174 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-5:MOVE] in serial mode
> 2019-02-08T12:28:30,177 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table1 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/.hive-staging_hive_2019-02-08_12-28-23_584_1482331698286040936-3/-ext-10001
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://ctr-e139-1542663976389-62755-01-14.hwx.site:9083
> 2019-02-08T12:28:30,189 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - HMSC::open(): Could not find delegation
> token. Creating KERBEROS-based thrift connection.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 4
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Connected to metastore.
> 2019-02-08T12:28:30,206 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> ugi=hive/ctr-e139-1542663976389-62755-01-14.hwx.s...@hwqe.hortonworks.com
> (auth:KERBEROS) retries=24 delay=5 lifetime=0
> 2019-02-08T12:28:30,325 INFO
> [org.apache.ranger.audit.queue.AuditBatchQueue1]: provider.BaseAuditHandler
> (:()) - Audit Status Log: name=hiveServer2.async.multi_dest.batch,
> finalDestination=hiveServer2.async.multi_dest.batch.solr, interval=01:00.002
> minutes, events=2, succcessCount=1, totalEvents=56, totalSuccessCount=25
> 2019-02-08T12:28:30,520 INFO [HiveServer2-Background-Pool: Thread-1134]:
> common.FileUtils (FileUtils.java:mkdir(580)) - Creating directory if it
> doesn't exist:
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table1/base_001
> 2019-02-08T12:28:31,245 INFO [HiveServer2-Background-Pool: Thread-1134]:
> ql.Driver (:()) - Starting task [Stage-11:MOVE] in serial mode
> 2019-02-08T12:28:31,245 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore,
> current connections: 3
> 2019-02-08T12:28:31,246 INFO [HiveServer2-Background-Pool: Thread-1134]:
> exec.Task (:()) - Loading data to table nondefault.nondefault_table2 from
> hdfs://mycluster1/warehouse/tablespace/managed/hive/nondefault.db/nondefault_table2/.hive-staging_hive_2019-02-08_12-28-23_810_7457138692783022870-3/-ext-10002
> 2019-02-08T12:28:31,327 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://ctr-e139-1542663976389-62755-01-14.hwx.site:9083
> 2019-02-08T12:28:31,327 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - HMSC::open(): Could not find delegation
> token. Creating KERBEROS-based thrift connection.
> 2019-02-08T12:28:31,336 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 4
> 2019-02-08T12:28:31,337 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.HiveMetaStoreClient (:()) - Connected to metastore.
> 2019-02-08T12:28:31,337 INFO [HiveServer2-Background-Pool: Thread-1134]:
> metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
> org
[jira] [Commented] (HIVE-21071) Improve getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765121#comment-16765121
]
BELUGA BEHR commented on HIVE-21071:
[~pvary] Thanks! Are you able to merge?
> Improve getInputSummary
> ---
>
> Key: HIVE-21071
> URL: https://issues.apache.org/jira/browse/HIVE-21071
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21071.1.patch, HIVE-21071.10.patch,
> HIVE-21071.11.patch, HIVE-21071.2.patch, HIVE-21071.3.patch,
> HIVE-21071.4.patch, HIVE-21071.5.patch, HIVE-21071.6.patch,
> HIVE-21071.7.patch, HIVE-21071.8.patch, HIVE-21071.9.patch
>
>
> There is a global lock in the {{getInptSummary}} code, so it is important
> that it be fast. The current implementation has quite a bit of overhead that
> can be re-engineered.
> For example, the current implementation keeps a map of File Path to
> ContentSummary object. This map is populated by several threads
> concurrently. The method then loops through the map, in a single thread, at
> the end to add up all of the ContentSummary objects and ignores the paths.
> The code can be be re-engineered to not use a map, or a collection at all, to
> store the results and instead just keep a running tally. By keeping a tally,
> there is no {{O\(n)}} operation at the end to perform the addition.
> There are other things can be improved. The method returns an object which
> is never used anywhere, so change method to void return type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21240) JSON SerDe Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR reassigned HIVE-21240: -- Assignee: BELUGA BEHR > JSON SerDe Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Fix For: 4.0.0 > > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Summary: JSON SerDe Deserialize Re-Write (was: JSON SerDe Re-Write) > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Fix For: 4.0.0 > > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Attachment: HIVE-21240.1.patch > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Status: Patch Available (was: Open) > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.1.1, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?focusedWorklogId=197011&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-197011 ] ASF GitHub Bot logged work on HIVE-21240: - Author: ASF GitHub Bot Created on: 11/Feb/19 16:14 Start Date: 11/Feb/19 16:14 Worklog Time Spent: 10m Work Description: BELUGABEHR commented on pull request #530: HIVE-21240: JSON SerDe Deserialize Re-Write URL: https://github.com/apache/hive/pull/530 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 197011) Time Spent: 10m Remaining Estimate: 0h > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21240: -- Labels: pull-request-available (was: ) > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21224) Upgrade tests JUnit3 to JUnit4
[
https://issues.apache.org/jira/browse/HIVE-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765103#comment-16765103
]
Hive QA commented on HIVE-21224:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958244/HIVE-21224.5.patch
{color:green}SUCCESS:{color} +1 due to 153 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15780 tests
executed
*Failed tests:*
{noformat}
TestGenericUDFConcat - did not produce a TEST-*.xml file (likely timed out)
(batchId=287)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=254)
TestMTQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=252)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16014/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16014/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16014/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958244 - PreCommit-HIVE-Build
> Upgrade tests JUnit3 to JUnit4
> --
>
> Key: HIVE-21224
> URL: https://issues.apache.org/jira/browse/HIVE-21224
> Project: Hive
> Issue Type: Improvement
>Reporter: Bruno Pusztahazi
>Assignee: Bruno Pusztahazi
>Priority: Major
> Attachments: HIVE-21224.1.patch, HIVE-21224.2.patch,
> HIVE-21224.3.patch, HIVE-21224.4.patch, HIVE-21224.5.patch
>
>
> Old JUnit3 tests should be upgraded to JUnit4
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21224) Upgrade tests JUnit3 to JUnit4
[
https://issues.apache.org/jira/browse/HIVE-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765098#comment-16765098
]
Hive QA commented on HIVE-21224:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 2298 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} cli in master has 13 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} contrib in master has 10 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} druid-handler in master has 3 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} hcatalog/core in master has 30 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/webhcat/java-client in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} hcatalog/webhcat/svr in master has 96 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} common: The patch generated 0 new + 9 unchanged - 3
fixed = 9 total (was 12) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} serde: The patch generated 2 new + 288 unchanged - 20
fixed = 290 total (was 308) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 2 new + 916 unchanged - 94
fixed = 918 total (was 1010) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} service: The patch generated 0 new + 17 unchanged -
6 fixed = 17 total (was 23) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} cli: The patch generated 0 new + 9 unchanged - 1
fixed = 9 total (was 10) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch contrib passed checkstyle {color} |
|
[jira] [Commented] (HIVE-21239) Beeline help LDAP connection example incorrect
[
https://issues.apache.org/jira/browse/HIVE-21239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765093#comment-16765093
]
Zoltan Chovan commented on HIVE-21239:
--
I've modified the BeeLine.properties file, where the help text is present and
uploaded a patch, however I'm not sure if this commit needs precommit tests, as
there is no change in Java code. [~pvary] could you advise?
Additionally, I've tried to bring the examples into a common format.
> Beeline help LDAP connection example incorrect
> --
>
> Key: HIVE-21239
> URL: https://issues.apache.org/jira/browse/HIVE-21239
> Project: Hive
> Issue Type: Bug
> Environment: This was tested on a test environment with SSL and LDAP
> authentication enabled, and seems to be reproducible on any environment with
> LDAP authentication available in HiveServer2.
>Reporter: Zsolt Herczeg
>Assignee: Zoltan Chovan
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21239.patch
>
>
> There's the following connection example string in the beeline -h command
> output:
>
> {code:java}
> 5. Connect using LDAP authentication
> $ beeline -u jdbc:hive2://hs2.local:10013/default
>
> {code}
> When a user attempts to connect like above, it'll fail with LDAP
> authentication failure. This is because username and passwords are not picked
> up in the shown form. A working example would be:
> {code:java}
> $ beeline -n -p -u
> jdbc:hive2://hs2.local:10013/default
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
