[jira] [Commented] (HIVE-13199) NDC stopped working in LLAP logging
[
https://issues.apache.org/jira/browse/HIVE-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181568#comment-15181568
]
Hive QA commented on HIVE-13199:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791072/HIVE-13199.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9781 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7165/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7165/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7165/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12791072 - PreCommit-HIVE-TRUNK-Build
> NDC stopped working in LLAP logging
> ---
>
> Key: HIVE-13199
> URL: https://issues.apache.org/jira/browse/HIVE-13199
> Project: Hive
> Issue Type: Bug
> Components: llap, Logging
>Affects Versions: 2.1.0
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13199.1.patch
>
>
> NDC context were missing from the log lines. Reason for it is NDC class is
> part of log4j-1.2-api (bridge jar). This is added as compile time dependency.
> Due to the absence of this jar in llap daemons, the NDC context failed to
> initialize. Log4j2 replaced NDC with ThreadContext. Hence we need the bridge
> jar.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HIVE-13132) Hive should lazily load and cache metastore (permanent) functions
[
https://issues.apache.org/jira/browse/HIVE-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Hsu reopened HIVE-13132:
> Hive should lazily load and cache metastore (permanent) functions
> -
>
> Key: HIVE-13132
> URL: https://issues.apache.org/jira/browse/HIVE-13132
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 0.13.1
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13132.1.patch
>
>
> In Hive 0.13.1, we have noticed that as the number of databases increases,
> the start-up time of the Hive interactive shell increases. This is because
> during start-up, all databases are iterated over to fetch the permanent
> functions to display in the {{SHOW FUNCTIONS}} output.
> {noformat:title=FunctionRegistry.java}
> private static Set getFunctionNames(boolean searchMetastore) {
> Set functionNames = mFunctions.keySet();
> if (searchMetastore) {
> functionNames = new HashSet(functionNames);
> try {
> Hive db = getHive();
> List dbNames = db.getAllDatabases();
> for (String dbName : dbNames) {
> List funcNames = db.getFunctions(dbName, "*");
> for (String funcName : funcNames) {
> functionNames.add(FunctionUtils.qualifyFunctionName(funcName,
> dbName));
> }
> }
> } catch (Exception e) {
> LOG.error(e);
> // Continue on, we can still return the functions we've gotten to
> this point.
> }
> }
> return functionNames;
> }
> {noformat}
> Instead of eagerly loading all metastore functions, we should only load them
> the first time {{SHOW FUNCTIONS}} is invoked. We should also cache the
> results.
> Note that this issue may have been fixed by HIVE-2573, though I haven't
> verified this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13132) Hive should lazily load and cache metastore (permanent) functions
[
https://issues.apache.org/jira/browse/HIVE-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181558#comment-15181558
]
Anthony Hsu commented on HIVE-13132:
Correction: CLI start-up time was reduce from 20+ seconds to < 5 seconds.
> Hive should lazily load and cache metastore (permanent) functions
> -
>
> Key: HIVE-13132
> URL: https://issues.apache.org/jira/browse/HIVE-13132
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 0.13.1
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13132.1.patch
>
>
> In Hive 0.13.1, we have noticed that as the number of databases increases,
> the start-up time of the Hive interactive shell increases. This is because
> during start-up, all databases are iterated over to fetch the permanent
> functions to display in the {{SHOW FUNCTIONS}} output.
> {noformat:title=FunctionRegistry.java}
> private static Set getFunctionNames(boolean searchMetastore) {
> Set functionNames = mFunctions.keySet();
> if (searchMetastore) {
> functionNames = new HashSet(functionNames);
> try {
> Hive db = getHive();
> List dbNames = db.getAllDatabases();
> for (String dbName : dbNames) {
> List funcNames = db.getFunctions(dbName, "*");
> for (String funcName : funcNames) {
> functionNames.add(FunctionUtils.qualifyFunctionName(funcName,
> dbName));
> }
> }
> } catch (Exception e) {
> LOG.error(e);
> // Continue on, we can still return the functions we've gotten to
> this point.
> }
> }
> return functionNames;
> }
> {noformat}
> Instead of eagerly loading all metastore functions, we should only load them
> the first time {{SHOW FUNCTIONS}} is invoked. We should also cache the
> results.
> Note that this issue may have been fixed by HIVE-2573, though I haven't
> verified this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-13132) Hive should lazily load and cache metastore (permanent) functions
[
https://issues.apache.org/jira/browse/HIVE-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Hsu resolved HIVE-13132.
Resolution: Won't Fix
> Hive should lazily load and cache metastore (permanent) functions
> -
>
> Key: HIVE-13132
> URL: https://issues.apache.org/jira/browse/HIVE-13132
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 0.13.1
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13132.1.patch
>
>
> In Hive 0.13.1, we have noticed that as the number of databases increases,
> the start-up time of the Hive interactive shell increases. This is because
> during start-up, all databases are iterated over to fetch the permanent
> functions to display in the {{SHOW FUNCTIONS}} output.
> {noformat:title=FunctionRegistry.java}
> private static Set getFunctionNames(boolean searchMetastore) {
> Set functionNames = mFunctions.keySet();
> if (searchMetastore) {
> functionNames = new HashSet(functionNames);
> try {
> Hive db = getHive();
> List dbNames = db.getAllDatabases();
> for (String dbName : dbNames) {
> List funcNames = db.getFunctions(dbName, "*");
> for (String funcName : funcNames) {
> functionNames.add(FunctionUtils.qualifyFunctionName(funcName,
> dbName));
> }
> }
> } catch (Exception e) {
> LOG.error(e);
> // Continue on, we can still return the functions we've gotten to
> this point.
> }
> }
> return functionNames;
> }
> {noformat}
> Instead of eagerly loading all metastore functions, we should only load them
> the first time {{SHOW FUNCTIONS}} is invoked. We should also cache the
> results.
> Note that this issue may have been fixed by HIVE-2573, though I haven't
> verified this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13132) Hive should lazily load and cache metastore (permanent) functions
[
https://issues.apache.org/jira/browse/HIVE-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Hsu updated HIVE-13132:
---
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
> Hive should lazily load and cache metastore (permanent) functions
> -
>
> Key: HIVE-13132
> URL: https://issues.apache.org/jira/browse/HIVE-13132
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 0.13.1
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13132.1.patch
>
>
> In Hive 0.13.1, we have noticed that as the number of databases increases,
> the start-up time of the Hive interactive shell increases. This is because
> during start-up, all databases are iterated over to fetch the permanent
> functions to display in the {{SHOW FUNCTIONS}} output.
> {noformat:title=FunctionRegistry.java}
> private static Set getFunctionNames(boolean searchMetastore) {
> Set functionNames = mFunctions.keySet();
> if (searchMetastore) {
> functionNames = new HashSet(functionNames);
> try {
> Hive db = getHive();
> List dbNames = db.getAllDatabases();
> for (String dbName : dbNames) {
> List funcNames = db.getFunctions(dbName, "*");
> for (String funcName : funcNames) {
> functionNames.add(FunctionUtils.qualifyFunctionName(funcName,
> dbName));
> }
> }
> } catch (Exception e) {
> LOG.error(e);
> // Continue on, we can still return the functions we've gotten to
> this point.
> }
> }
> return functionNames;
> }
> {noformat}
> Instead of eagerly loading all metastore functions, we should only load them
> the first time {{SHOW FUNCTIONS}} is invoked. We should also cache the
> results.
> Note that this issue may have been fixed by HIVE-2573, though I haven't
> verified this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13132) Hive should lazily load and cache metastore (permanent) functions
[
https://issues.apache.org/jira/browse/HIVE-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181545#comment-15181545
]
Anthony Hsu commented on HIVE-13132:
[~jdere], thanks for pointing out HIVE-10319! I backported it to Hive 0.13.1
and it reduced my CLI start-up time from 20+ seconds to < 1 second. {{SHOW
FUNCTIONS}} also now takes < 1 second rather than 20+ seconds. This ticket is
no longer needed, so I will resolve it.
> Hive should lazily load and cache metastore (permanent) functions
> -
>
> Key: HIVE-13132
> URL: https://issues.apache.org/jira/browse/HIVE-13132
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 0.13.1
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13132.1.patch
>
>
> In Hive 0.13.1, we have noticed that as the number of databases increases,
> the start-up time of the Hive interactive shell increases. This is because
> during start-up, all databases are iterated over to fetch the permanent
> functions to display in the {{SHOW FUNCTIONS}} output.
> {noformat:title=FunctionRegistry.java}
> private static Set getFunctionNames(boolean searchMetastore) {
> Set functionNames = mFunctions.keySet();
> if (searchMetastore) {
> functionNames = new HashSet(functionNames);
> try {
> Hive db = getHive();
> List dbNames = db.getAllDatabases();
> for (String dbName : dbNames) {
> List funcNames = db.getFunctions(dbName, "*");
> for (String funcName : funcNames) {
> functionNames.add(FunctionUtils.qualifyFunctionName(funcName,
> dbName));
> }
> }
> } catch (Exception e) {
> LOG.error(e);
> // Continue on, we can still return the functions we've gotten to
> this point.
> }
> }
> return functionNames;
> }
> {noformat}
> Instead of eagerly loading all metastore functions, we should only load them
> the first time {{SHOW FUNCTIONS}} is invoked. We should also cache the
> results.
> Note that this issue may have been fixed by HIVE-2573, though I haven't
> verified this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13198) Authorization issues with cascading views
[ https://issues.apache.org/jira/browse/HIVE-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13198: --- Status: Patch Available (was: Open) > Authorization issues with cascading views > - > > Key: HIVE-13198 > URL: https://issues.apache.org/jira/browse/HIVE-13198 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13198.01.patch > > > Here is a use case. They have a base table t1, from which they create a view > v1. They further create a view v2 from v1 by applying a filter. User has > access to only view v2, not view v1 or table t1. When user tries to access > v2, they are denied access. > Steps to recreate: > There is a base table t1 that exists in the default database with primary key > id and some employee data (name, ssn etc) > Create view v1 - “create view v1 as select * from default.t1;” > Created v2 - “create view v2 as select * from v1 where id =1;” > Permissions provided for user to select all columns from view v2. When user > runs select * from v2, hive throws an error “user does not have permissions > to select view v1". > Apparently Hive is converting the query to underlying views. > SELECT * FROM v2 LIMIT 100 > To > select `v1`.`id`, `v1`.`name`, `v1`.`ssn`, `v1`.`join_date`, `v1`.`location` > from `hr`.`v1` where `v1`.`id`=1 > Hive should only check for permissions for the view being run in the query, > not any parent views. (This is consistent with ORACLE). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13198) Authorization issues with cascading views
[ https://issues.apache.org/jira/browse/HIVE-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13198: --- Attachment: HIVE-13198.01.patch > Authorization issues with cascading views > - > > Key: HIVE-13198 > URL: https://issues.apache.org/jira/browse/HIVE-13198 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13198.01.patch > > > Here is a use case. They have a base table t1, from which they create a view > v1. They further create a view v2 from v1 by applying a filter. User has > access to only view v2, not view v1 or table t1. When user tries to access > v2, they are denied access. > Steps to recreate: > There is a base table t1 that exists in the default database with primary key > id and some employee data (name, ssn etc) > Create view v1 - “create view v1 as select * from default.t1;” > Created v2 - “create view v2 as select * from v1 where id =1;” > Permissions provided for user to select all columns from view v2. When user > runs select * from v2, hive throws an error “user does not have permissions > to select view v1". > Apparently Hive is converting the query to underlying views. > SELECT * FROM v2 LIMIT 100 > To > select `v1`.`id`, `v1`.`name`, `v1`.`ssn`, `v1`.`join_date`, `v1`.`location` > from `hr`.`v1` where `v1`.`id`=1 > Hive should only check for permissions for the view being run in the query, > not any parent views. (This is consistent with ORACLE). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12270: --- Attachment: HIVE-12270.3.patch Revised the patch after discussing with Szehon. We will getMSC via thread local Hive object for token api calls in different threads, instead of passing around and sharing one MSC. [~szehon] Could you review the patch? Thanks > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.1.nothrift.patch, HIVE-12270.1.patch, > HIVE-12270.2.patch, HIVE-12270.3.nothrift.patch, HIVE-12270.3.patch, > HIVE-12270.nothrift.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12270: --- Attachment: HIVE-12270.3.nothrift.patch > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.1.nothrift.patch, HIVE-12270.1.patch, > HIVE-12270.2.patch, HIVE-12270.3.nothrift.patch, HIVE-12270.nothrift.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12481) Occasionally "Request is a replay" will be thrown from HS2
[
https://issues.apache.org/jira/browse/HIVE-12481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181484#comment-15181484
]
Hive QA commented on HIVE-12481:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791044/HIVE-12481.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9781 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7164/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7164/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7164/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12791044 - PreCommit-HIVE-TRUNK-Build
> Occasionally "Request is a replay" will be thrown from HS2
> --
>
> Key: HIVE-12481
> URL: https://issues.apache.org/jira/browse/HIVE-12481
> Project: Hive
> Issue Type: Improvement
> Components: Authentication
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-12481.2.patch, HIVE-12481.3.patch, HIVE-12481.patch
>
>
> We have seen the following exception thrown from HS2 in secured cluster when
> many queries are running simultaneously on single HS2 instance.
> The cause I can guess is that it happens that two queries are submitted at
> the same time and have the same timestamp. For such case, we can add a retry
> for the query.
>
> {noformat}
> 2015-11-18 16:12:33,117 ERROR org.apache.thrift.transport.TSaslTransport:
> SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: Failure unspecified at GSS-API level (Mechanism level: Request
> is a replay (34))]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)
> at
> org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1651)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
> level: Request is a replay (34))
> at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
> at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
> at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155)
> ... 14 more
> Caused by: KrbException: Request is a replay (34)
> at
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Status: Patch Available (was: In Progress) > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch, > HIVE-13178.03.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Attachment: HIVE-13178.03.patch > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch, > HIVE-13178.03.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Status: In Progress (was: Patch Available) > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-12052) automatically populate file metadata to HBase metastore based on config or table properties
[ https://issues.apache.org/jira/browse/HIVE-12052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-12052: --- Assignee: Sergey Shelukhin > automatically populate file metadata to HBase metastore based on config or > table properties > --- > > Key: HIVE-12052 > URL: https://issues.apache.org/jira/browse/HIVE-12052 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > As discussed in HIVE-11500 > Should use a table property similar to auto.purge. > Then, when this setting is set, partitions are added (convertToMPart is a > good source to find all the paths for that), after compactions, after > load/non-ACID insert, and periodically (configurable), the storage locations > should be scanned for new files and cache updated accordingly. All the > updates should probably be in the background thread and taken from queue > (high pri from most ops, low pri from enabling the property and from periodic > updates) to avoid high load on HDFS from metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181414#comment-15181414 ] Sergey Shelukhin commented on HIVE-13204: - +1 pending tests > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13204.1.patch, HIVE-13204.2.patch, > HIVE-13204.3.patch, regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13204: --- Attachment: HIVE-13204.3.patch > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13204.1.patch, HIVE-13204.2.patch, > HIVE-13204.3.patch, regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13204: --- Status: Patch Available (was: In Progress) > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13204.1.patch, HIVE-13204.2.patch, > regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13204: --- Attachment: HIVE-13204.2.patch > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13204.1.patch, HIVE-13204.2.patch, > regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13175) Disallow making external tables transactional
[
https://issues.apache.org/jira/browse/HIVE-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181351#comment-15181351
]
Hive QA commented on HIVE-13175:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791062/HIVE-13175.4.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9768 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7163/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7163/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7163/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12791062 - PreCommit-HIVE-TRUNK-Build
> Disallow making external tables transactional
> -
>
> Key: HIVE-13175
> URL: https://issues.apache.org/jira/browse/HIVE-13175
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13175.1.patch, HIVE-13175.2.patch,
> HIVE-13175.3.patch, HIVE-13175.4.patch
>
>
> The fact that compactor rewrites contents of ACID tables is in conflict with
> what is expected of external tables.
> Conversely, end user can write to External table which certainly not what is
> expected of ACID table.
> So we should explicitly disallow making an external table ACID.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13211) normalize Hive.get overloads to go thru one path
[ https://issues.apache.org/jira/browse/HIVE-13211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-13211: --- Assignee: Sergey Shelukhin > normalize Hive.get overloads to go thru one path > > > Key: HIVE-13211 > URL: https://issues.apache.org/jira/browse/HIVE-13211 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13211.patch > > > There are multiple subtly different paths in Hive.get(...) methods. Some > close the old db on refresh, some don't. Some check if the client is > compatible with config, some don't. Also there were some parameters (don't > register functions, disallow embedded metastore) that were added recently. > Need to make this stuff go thru one path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12925) make sure metastore footer cache usage never creates embedded metastore, and doesn't get all functions
[ https://issues.apache.org/jira/browse/HIVE-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12925: Status: Patch Available (was: Open) > make sure metastore footer cache usage never creates embedded metastore, and > doesn't get all functions > -- > > Key: HIVE-12925 > URL: https://issues.apache.org/jira/browse/HIVE-12925 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12925.patch > > > See HIVE-12918, also some other JIRA that added an option to not pre-load all > functions. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12925) make sure metastore footer cache usage never creates embedded metastore, and doesn't get all functions
[ https://issues.apache.org/jira/browse/HIVE-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12925: Description: See HIVE-12918, also some other JIRA that added an option to not pre-load all functions. NO PRECOMMIT TESTS was:See HIVE-12918, also some other JIRA that added an option to not pre-load all functions. > make sure metastore footer cache usage never creates embedded metastore, and > doesn't get all functions > -- > > Key: HIVE-12925 > URL: https://issues.apache.org/jira/browse/HIVE-12925 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12925.patch > > > See HIVE-12918, also some other JIRA that added an option to not pre-load all > functions. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12925) make sure metastore footer cache usage never creates embedded metastore, and doesn't get all functions
[ https://issues.apache.org/jira/browse/HIVE-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12925: Attachment: HIVE-12925.patch A trivial patch after the refactoring in the last iteration of HIVE-11675, and HIVE-13211. > make sure metastore footer cache usage never creates embedded metastore, and > doesn't get all functions > -- > > Key: HIVE-12925 > URL: https://issues.apache.org/jira/browse/HIVE-12925 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12925.patch > > > See HIVE-12918, also some other JIRA that added an option to not pre-load all > functions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13211) normalize Hive.get overloads to go thru one path
[ https://issues.apache.org/jira/browse/HIVE-13211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13211: Status: Patch Available (was: Open) > normalize Hive.get overloads to go thru one path > > > Key: HIVE-13211 > URL: https://issues.apache.org/jira/browse/HIVE-13211 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > Attachments: HIVE-13211.patch > > > There are multiple subtly different paths in Hive.get(...) methods. Some > close the old db on refresh, some don't. Some check if the client is > compatible with config, some don't. Also there were some parameters (don't > register functions, disallow embedded metastore) that were added recently. > Need to make this stuff go thru one path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13211) normalize Hive.get overloads to go thru one path
[ https://issues.apache.org/jira/browse/HIVE-13211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13211: Attachment: HIVE-13211.patch [~ashutoshc] can you take a look? This basically makes all the peculiar paths go thru getInternal and merges all their features (getting conf from session, checking client for compat, etc.) > normalize Hive.get overloads to go thru one path > > > Key: HIVE-13211 > URL: https://issues.apache.org/jira/browse/HIVE-13211 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > Attachments: HIVE-13211.patch > > > There are multiple subtly different paths in Hive.get(...) methods. Some > close the old db on refresh, some don't. Some check if the client is > compatible with config, some don't. Also there were some parameters (don't > register functions, disallow embedded metastore) that were added recently. > Need to make this stuff go thru one path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181323#comment-15181323 ] Ashutosh Chauhan commented on HIVE-11160: - * Can you add explain for insert statements ? * It seems we run analyze table for all partitions, we should run it only for new partitions getting generated in the query ? > Auto-gather column stats > > > Key: HIVE-11160 > URL: https://issues.apache.org/jira/browse/HIVE-11160 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch, > HIVE-11160.03.patch, HIVE-11160.04.patch, HIVE-11160.05.patch > > > Hive will collect table stats when set hive.stats.autogather=true during the > INSERT OVERWRITE command. And then the users need to collect the column stats > themselves using "Analyze" command. In this patch, the column stats will also > be collected automatically. More specifically, INSERT OVERWRITE will > automatically create new column stats. INSERT INTO will automatically merge > new column stats with existing ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13169) HiveServer2: Support delegation token based connection when using http transport
[
https://issues.apache.org/jira/browse/HIVE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181318#comment-15181318
]
Lefty Leverenz commented on HIVE-13169:
---
Thanks for the doc, [~thejas]. Can we remove the TODOC13 label from HIVE-5155
now or should I nudge [~prasadm] for a review?
A fine point: Since configuration parameter hive.server2.transport.mode is
deprecated (0.14.0) in favor of connection URL parameter transportMode, should
this be reworded?
{quote}
Support for delegation tokens with HiveServer2 binary transport mode
hive.server2.transport.mode has been available starting 0.13.0 ...
{quote}
* [deprecation notice for hive.server2.transport.mode |
https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLWhenHiveServer2IsRunninginHTTPMode]
> HiveServer2: Support delegation token based connection when using http
> transport
>
>
> Key: HIVE-13169
> URL: https://issues.apache.org/jira/browse/HIVE-13169
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, JDBC
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Thejas M Nair
> Fix For: 2.1.0
>
> Attachments: HIVE-13169-branch-1.2.patch, HIVE-13169.1.patch,
> HIVE-13169.2.patch, HIVE-13169.3.patch, HIVE-13169.3.patch,
> HIVE-13169.4.patch, HIVE-13169.5.patch
>
>
> HIVE-5155 introduced support for delegation token based connection. However,
> it was intended for tcp transport mode. We need to have similar mechanisms
> for http transport.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13209) metastore get_delegation_token fails with null ip address
[
https://issues.apache.org/jira/browse/HIVE-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180721#comment-15180721
]
Sushanth Sowmyan commented on HIVE-13209:
-
Looks straightforward enough. +1.
> metastore get_delegation_token fails with null ip address
> -
>
> Key: HIVE-13209
> URL: https://issues.apache.org/jira/browse/HIVE-13209
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Thejas M Nair
> Fix For: 2.1.0
>
> Attachments: HIVE-13209.1.patch, HIVE-13209.2.patch
>
>
> After changes in HIVE-13169, metastore get_delegation_token fails with null
> ip address.
> {code}
> 2016-03-03 07:45:31,055 ERROR [pool-6-thread-22]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(159)) -
> MetaException(message:Unauthorized connection for super-user:
> HTTP/ from IP null)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_delegation_token(HiveMetaStore.java:5290)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy16.get_delegation_token(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_delegation_token.getResult(ThriftHiveMetastore.java:11492)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_delegation_token.getResult(ThriftHiveMetastore.java:11476)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:551)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:546)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:546)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13210) Revert changes in HIVE-12994 related to metastore
[
https://issues.apache.org/jira/browse/HIVE-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180715#comment-15180715
]
Hive QA commented on HIVE-13210:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791556/HIVE-13210.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 4 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/127/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/127/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-METASTORE-Test-127/
Messages:
{noformat}
LXC derby found.
LXC derby is not started. Starting container...
Container started.
Preparing derby container...
Container prepared.
Calling /hive/testutils/metastore/dbs/derby/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/derby/execute.sh ...
Tests executed.
LXC mysql found.
LXC mysql is not started. Starting container...
Container started.
Preparing mysql container...
Container prepared.
Calling /hive/testutils/metastore/dbs/mysql/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/mysql/execute.sh ...
Tests executed.
LXC oracle found.
LXC oracle is not started. Starting container...
Container started.
Preparing oracle container...
Container prepared.
Calling /hive/testutils/metastore/dbs/oracle/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/oracle/execute.sh ...
Tests executed.
LXC postgres found.
LXC postgres is not started. Starting container...
Container started.
Preparing postgres container...
Container prepared.
Calling /hive/testutils/metastore/dbs/postgres/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/postgres/execute.sh ...
Tests executed.
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12791556 - PreCommit-HIVE-METASTORE-Test
> Revert changes in HIVE-12994 related to metastore
> -
>
> Key: HIVE-13210
> URL: https://issues.apache.org/jira/browse/HIVE-13210
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13210.patch
>
>
> As we do not control what is written in the physical layer and thus we cannot
> ensure NULLS ORDER (and even if we did, currently we do not take advantage of
> it), it seems exposing the NULLS ORDER property at metastore level does not
> make much sense. We will revert that part of patch HIVE-12994.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13209) metastore get_delegation_token fails with null ip address
[
https://issues.apache.org/jira/browse/HIVE-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-13209:
-
Attachment: HIVE-13209.2.patch
2.patch - fix the issue that ip address string had a "/" in front. Using
saslServer.getRemoteAddress().getHostAddress() instead of
String.valueOf(saslServer.getRemoteAddress())
> metastore get_delegation_token fails with null ip address
> -
>
> Key: HIVE-13209
> URL: https://issues.apache.org/jira/browse/HIVE-13209
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Thejas M Nair
> Fix For: 2.1.0
>
> Attachments: HIVE-13209.1.patch, HIVE-13209.2.patch
>
>
> After changes in HIVE-13169, metastore get_delegation_token fails with null
> ip address.
> {code}
> 2016-03-03 07:45:31,055 ERROR [pool-6-thread-22]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(159)) -
> MetaException(message:Unauthorized connection for super-user:
> HTTP/ from IP null)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_delegation_token(HiveMetaStore.java:5290)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy16.get_delegation_token(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_delegation_token.getResult(ThriftHiveMetastore.java:11492)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_delegation_token.getResult(ThriftHiveMetastore.java:11476)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:551)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:546)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:546)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13210) Revert changes in HIVE-12994 related to metastore
[ https://issues.apache.org/jira/browse/HIVE-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13210: --- Attachment: HIVE-13210.patch > Revert changes in HIVE-12994 related to metastore > - > > Key: HIVE-13210 > URL: https://issues.apache.org/jira/browse/HIVE-13210 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13210.patch > > > As we do not control what is written in the physical layer and thus we cannot > ensure NULLS ORDER (and even if we did, currently we do not take advantage of > it), it seems exposing the NULLS ORDER property at metastore level does not > make much sense. We will revert that part of patch HIVE-12994. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13210) Revert changes in HIVE-12994 related to metastore
[ https://issues.apache.org/jira/browse/HIVE-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13210 started by Jesus Camacho Rodriguez. -- > Revert changes in HIVE-12994 related to metastore > - > > Key: HIVE-13210 > URL: https://issues.apache.org/jira/browse/HIVE-13210 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13210.patch > > > As we do not control what is written in the physical layer and thus we cannot > ensure NULLS ORDER (and even if we did, currently we do not take advantage of > it), it seems exposing the NULLS ORDER property at metastore level does not > make much sense. We will revert that part of patch HIVE-12994. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13210) Revert changes in HIVE-12994 related to metastore
[ https://issues.apache.org/jira/browse/HIVE-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13210: --- Status: Patch Available (was: In Progress) > Revert changes in HIVE-12994 related to metastore > - > > Key: HIVE-13210 > URL: https://issues.apache.org/jira/browse/HIVE-13210 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13210.patch > > > As we do not control what is written in the physical layer and thus we cannot > ensure NULLS ORDER (and even if we did, currently we do not take advantage of > it), it seems exposing the NULLS ORDER property at metastore level does not > make much sense. We will revert that part of patch HIVE-12994. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13171) Add unit test for hs2 webui
[ https://issues.apache.org/jira/browse/HIVE-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13171: - Attachment: HIVE-13171.3.patch Address review comments > Add unit test for hs2 webui > --- > > Key: HIVE-13171 > URL: https://issues.apache.org/jira/browse/HIVE-13171 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13171.2.patch, HIVE-13171.3.patch, HIVE-13171.patch > > > With more complex changes going into webui, it is hard to manually verify all > the kinds of cases. > With HIVE-12952, HS2 webui now uses jamon, which should be more unit-testable > than plain old jsp. We can perhaps add unit test for the jamon servlets, or > test the new OperationDisplay classes queried by the servlets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13169) HiveServer2: Support delegation token based connection when using http transport
[ https://issues.apache.org/jira/browse/HIVE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180684#comment-15180684 ] Thejas M Nair commented on HIVE-13169: -- Earlier updates to that section https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=30758725=127=126 > HiveServer2: Support delegation token based connection when using http > transport > > > Key: HIVE-13169 > URL: https://issues.apache.org/jira/browse/HIVE-13169 > Project: Hive > Issue Type: Bug > Components: HiveServer2, JDBC >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Thejas M Nair > Fix For: 2.1.0 > > Attachments: HIVE-13169-branch-1.2.patch, HIVE-13169.1.patch, > HIVE-13169.2.patch, HIVE-13169.3.patch, HIVE-13169.3.patch, > HIVE-13169.4.patch, HIVE-13169.5.patch > > > HIVE-5155 introduced support for delegation token based connection. However, > it was intended for tcp transport mode. We need to have similar mechanisms > for http transport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13169) HiveServer2: Support delegation token based connection when using http transport
[ https://issues.apache.org/jira/browse/HIVE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180682#comment-15180682 ] Thejas M Nair commented on HIVE-13169: -- Thanks for the reminder [~leftylev]. I have made an edit https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=30758725=128=127 . I had updated that section few weeks back. > HiveServer2: Support delegation token based connection when using http > transport > > > Key: HIVE-13169 > URL: https://issues.apache.org/jira/browse/HIVE-13169 > Project: Hive > Issue Type: Bug > Components: HiveServer2, JDBC >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Thejas M Nair > Fix For: 2.1.0 > > Attachments: HIVE-13169-branch-1.2.patch, HIVE-13169.1.patch, > HIVE-13169.2.patch, HIVE-13169.3.patch, HIVE-13169.3.patch, > HIVE-13169.4.patch, HIVE-13169.5.patch > > > HIVE-5155 introduced support for delegation token based connection. However, > it was intended for tcp transport mode. We need to have similar mechanisms > for http transport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12988) Improve dynamic partition loading IV
[ https://issues.apache.org/jira/browse/HIVE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12988: Attachment: HIVE-12988.6.patch > Improve dynamic partition loading IV > > > Key: HIVE-12988 > URL: https://issues.apache.org/jira/browse/HIVE-12988 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.2.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12988.2.patch, HIVE-12988.2.patch, > HIVE-12988.3.patch, HIVE-12988.4.patch, HIVE-12988.5.patch, > HIVE-12988.6.patch, HIVE-12988.patch > > > Parallelize copyFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12988) Improve dynamic partition loading IV
[ https://issues.apache.org/jira/browse/HIVE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12988: Status: Open (was: Patch Available) > Improve dynamic partition loading IV > > > Key: HIVE-12988 > URL: https://issues.apache.org/jira/browse/HIVE-12988 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12988.2.patch, HIVE-12988.2.patch, > HIVE-12988.3.patch, HIVE-12988.4.patch, HIVE-12988.5.patch, > HIVE-12988.6.patch, HIVE-12988.patch > > > Parallelize copyFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13169) HiveServer2: Support delegation token based connection when using http transport
[ https://issues.apache.org/jira/browse/HIVE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180672#comment-15180672 ] Lefty Leverenz commented on HIVE-13169: --- Does this need documentation? If so, please give it a TODOC2.1 label. Also see the doc comment on HIVE-5155 (TODOC13): * [HIVE-5155 doc needs review | https://issues.apache.org/jira/browse/HIVE-5155?focusedCommentId=14084408=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14084408] > HiveServer2: Support delegation token based connection when using http > transport > > > Key: HIVE-13169 > URL: https://issues.apache.org/jira/browse/HIVE-13169 > Project: Hive > Issue Type: Bug > Components: HiveServer2, JDBC >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Thejas M Nair > Fix For: 2.1.0 > > Attachments: HIVE-13169-branch-1.2.patch, HIVE-13169.1.patch, > HIVE-13169.2.patch, HIVE-13169.3.patch, HIVE-13169.3.patch, > HIVE-13169.4.patch, HIVE-13169.5.patch > > > HIVE-5155 introduced support for delegation token based connection. However, > it was intended for tcp transport mode. We need to have similar mechanisms > for http transport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12988) Improve dynamic partition loading IV
[ https://issues.apache.org/jira/browse/HIVE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12988: Status: Patch Available (was: Open) > Improve dynamic partition loading IV > > > Key: HIVE-12988 > URL: https://issues.apache.org/jira/browse/HIVE-12988 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12988.2.patch, HIVE-12988.2.patch, > HIVE-12988.3.patch, HIVE-12988.4.patch, HIVE-12988.5.patch, > HIVE-12988.6.patch, HIVE-12988.patch > > > Parallelize copyFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13209) metastore get_delegation_token fails with null ip address
[
https://issues.apache.org/jira/browse/HIVE-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-13209:
-
Status: Patch Available (was: Open)
> metastore get_delegation_token fails with null ip address
> -
>
> Key: HIVE-13209
> URL: https://issues.apache.org/jira/browse/HIVE-13209
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Thejas M Nair
> Fix For: 2.1.0
>
> Attachments: HIVE-13209.1.patch
>
>
> After changes in HIVE-13169, metastore get_delegation_token fails with null
> ip address.
> {code}
> 2016-03-03 07:45:31,055 ERROR [pool-6-thread-22]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(159)) -
> MetaException(message:Unauthorized connection for super-user:
> HTTP/ from IP null)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_delegation_token(HiveMetaStore.java:5290)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy16.get_delegation_token(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_delegation_token.getResult(ThriftHiveMetastore.java:11492)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_delegation_token.getResult(ThriftHiveMetastore.java:11476)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:551)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:546)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:546)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13209) metastore get_delegation_token fails with null ip address
[
https://issues.apache.org/jira/browse/HIVE-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-13209:
-
Attachment: HIVE-13209.1.patch
> metastore get_delegation_token fails with null ip address
> -
>
> Key: HIVE-13209
> URL: https://issues.apache.org/jira/browse/HIVE-13209
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Thejas M Nair
> Fix For: 2.1.0
>
> Attachments: HIVE-13209.1.patch
>
>
> After changes in HIVE-13169, metastore get_delegation_token fails with null
> ip address.
> {code}
> 2016-03-03 07:45:31,055 ERROR [pool-6-thread-22]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(159)) -
> MetaException(message:Unauthorized connection for super-user:
> HTTP/ from IP null)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_delegation_token(HiveMetaStore.java:5290)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy16.get_delegation_token(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_delegation_token.getResult(ThriftHiveMetastore.java:11492)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_delegation_token.getResult(ThriftHiveMetastore.java:11476)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:551)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:546)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:546)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-12439) CompactionTxnHandler.markCleaned() and TxnHandler.openTxns() misc improvements
[ https://issues.apache.org/jira/browse/HIVE-12439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-12439: Assignee: Wei Zheng (was: Eugene Koifman) > CompactionTxnHandler.markCleaned() and TxnHandler.openTxns() misc improvements > -- > > Key: HIVE-12439 > URL: https://issues.apache.org/jira/browse/HIVE-12439 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > > # add a safeguard to make sure IN clause is not too large; break up by txn id > to delete from TXN_COMPONENTS where tc_txnid in ... > # TxnHandler. openTxns() - use 1 insert with many rows in values() clause, > rather than 1 DB roundtrip per row -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180583#comment-15180583
]
Jason Dere commented on HIVE-13063:
---
Changes look fine, let's see how the pre-commit run looks
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: HIVE-13063.patch, Screen Shot 2016-02-17 at 7.20.57
> PM.png, Screen Shot 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-12634) Add command to kill an ACID transacton
[ https://issues.apache.org/jira/browse/HIVE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-12634: Assignee: Wei Zheng (was: Eugene Koifman) > Add command to kill an ACID transacton > -- > > Key: HIVE-12634 > URL: https://issues.apache.org/jira/browse/HIVE-12634 > Project: Hive > Issue Type: New Feature > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > > Should add a CLI command to abort a (runaway) transaction. > This should clean up all state related to this txn. > The initiator of this (if still alive) will get an error trying to > heartbeat/commit, i.e. will become aware that the txn is dead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10632) Make sure TXN_COMPONENTS gets cleaned up if table is dropped before compaction.
[ https://issues.apache.org/jira/browse/HIVE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180565#comment-15180565 ] Wei Zheng commented on HIVE-10632: -- [~alangates] Can you take another look? > Make sure TXN_COMPONENTS gets cleaned up if table is dropped before > compaction. > --- > > Key: HIVE-10632 > URL: https://issues.apache.org/jira/browse/HIVE-10632 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-10632.1.patch, HIVE-10632.2.patch, > HIVE-10632.3.patch, HIVE-10632.4.patch, HIVE-10632.5.patch, > HIVE-10632.6.patch, HIVE-10632.7.patch > > > The compaction process will clean up entries in TXNS, > COMPLETED_TXN_COMPONENTS, TXN_COMPONENTS. If the table/partition is dropped > before compaction is complete there will be data left in these tables. Need > to investigate if there are other situations where this may happen and > address it. > see HIVE-10595 for additional info -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10632) Make sure TXN_COMPONENTS gets cleaned up if table is dropped before compaction.
[
https://issues.apache.org/jira/browse/HIVE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180557#comment-15180557
]
Hive QA commented on HIVE-10632:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791033/HIVE-10632.7.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9767 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7162/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7162/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7162/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12791033 - PreCommit-HIVE-TRUNK-Build
> Make sure TXN_COMPONENTS gets cleaned up if table is dropped before
> compaction.
> ---
>
> Key: HIVE-10632
> URL: https://issues.apache.org/jira/browse/HIVE-10632
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>Priority: Critical
> Attachments: HIVE-10632.1.patch, HIVE-10632.2.patch,
> HIVE-10632.3.patch, HIVE-10632.4.patch, HIVE-10632.5.patch,
> HIVE-10632.6.patch, HIVE-10632.7.patch
>
>
> The compaction process will clean up entries in TXNS,
> COMPLETED_TXN_COMPONENTS, TXN_COMPONENTS. If the table/partition is dropped
> before compaction is complete there will be data left in these tables. Need
> to investigate if there are other situations where this may happen and
> address it.
> see HIVE-10595 for additional info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13169) HiveServer2: Support delegation token based connection when using http transport
[ https://issues.apache.org/jira/browse/HIVE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-13169: - Attachment: HIVE-13169-branch-1.2.patch attaching patch for branch-1.2 for reference > HiveServer2: Support delegation token based connection when using http > transport > > > Key: HIVE-13169 > URL: https://issues.apache.org/jira/browse/HIVE-13169 > Project: Hive > Issue Type: Bug > Components: HiveServer2, JDBC >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Thejas M Nair > Fix For: 2.1.0 > > Attachments: HIVE-13169-branch-1.2.patch, HIVE-13169.1.patch, > HIVE-13169.2.patch, HIVE-13169.3.patch, HIVE-13169.3.patch, > HIVE-13169.4.patch, HIVE-13169.5.patch > > > HIVE-5155 introduced support for delegation token based connection. However, > it was intended for tcp transport mode. We need to have similar mechanisms > for http transport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13169) HiveServer2: Support delegation token based connection when using http transport
[ https://issues.apache.org/jira/browse/HIVE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-13169: - Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Patch committed to master. Thanks for your patch [~vgumashta] and the reviews! > HiveServer2: Support delegation token based connection when using http > transport > > > Key: HIVE-13169 > URL: https://issues.apache.org/jira/browse/HIVE-13169 > Project: Hive > Issue Type: Bug > Components: HiveServer2, JDBC >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Thejas M Nair > Fix For: 2.1.0 > > Attachments: HIVE-13169.1.patch, HIVE-13169.2.patch, > HIVE-13169.3.patch, HIVE-13169.3.patch, HIVE-13169.4.patch, HIVE-13169.5.patch > > > HIVE-5155 introduced support for delegation token based connection. However, > it was intended for tcp transport mode. We need to have similar mechanisms > for http transport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11160) Auto-gather column stats
[ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11160: --- Status: Patch Available (was: Open) > Auto-gather column stats > > > Key: HIVE-11160 > URL: https://issues.apache.org/jira/browse/HIVE-11160 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch, > HIVE-11160.03.patch, HIVE-11160.04.patch, HIVE-11160.05.patch > > > Hive will collect table stats when set hive.stats.autogather=true during the > INSERT OVERWRITE command. And then the users need to collect the column stats > themselves using "Analyze" command. In this patch, the column stats will also > be collected automatically. More specifically, INSERT OVERWRITE will > automatically create new column stats. INSERT INTO will automatically merge > new column stats with existing ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11160) Auto-gather column stats
[ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11160: --- Status: Open (was: Patch Available) > Auto-gather column stats > > > Key: HIVE-11160 > URL: https://issues.apache.org/jira/browse/HIVE-11160 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch, > HIVE-11160.03.patch, HIVE-11160.04.patch, HIVE-11160.05.patch > > > Hive will collect table stats when set hive.stats.autogather=true during the > INSERT OVERWRITE command. And then the users need to collect the column stats > themselves using "Analyze" command. In this patch, the column stats will also > be collected automatically. More specifically, INSERT OVERWRITE will > automatically create new column stats. INSERT INTO will automatically merge > new column stats with existing ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11160) Auto-gather column stats
[ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11160: --- Attachment: HIVE-11160.05.patch address [~ashutoshc]'s comments. > Auto-gather column stats > > > Key: HIVE-11160 > URL: https://issues.apache.org/jira/browse/HIVE-11160 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch, > HIVE-11160.03.patch, HIVE-11160.04.patch, HIVE-11160.05.patch > > > Hive will collect table stats when set hive.stats.autogather=true during the > INSERT OVERWRITE command. And then the users need to collect the column stats > themselves using "Analyze" command. In this patch, the column stats will also > be collected automatically. More specifically, INSERT OVERWRITE will > automatically create new column stats. INSERT INTO will automatically merge > new column stats with existing ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13203) STRUCT type not allowing more than 266 fields. Table get created with more fields but throws error while querying against the table. SHOW CREATE TABLE skips having some
[ https://issues.apache.org/jira/browse/HIVE-13203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satyajit varma reassigned HIVE-13203: - Assignee: Satyajit varma (was: VISHNU S NAIR) > STRUCT type not allowing more than 266 fields. Table get created with more > fields but throws error while querying against the table. SHOW CREATE TABLE > skips having some fields in STRUCT. > -- > > Key: HIVE-13203 > URL: https://issues.apache.org/jira/browse/HIVE-13203 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 1.2.0 >Reporter: Satyajit varma >Assignee: Satyajit varma > > I am trying to create a table with multiple STRUCT column types, one of which > having more than 300 fields. > I get the table created with no problem, but when i load i face few errors in > loading up the data.when i check SHOW CREATE TABLE TABLENAME, i get fields > only till 266th column and rest of them do not appear in the table DDL.(check > the issue at StateOrProvinc, search for StateOrProvinc in the second DDL). > Would like to know, if there is any restriction on the number of fields to be > using in STRUCT datatype. > EXAMPLE: > BELOW IS THE DDL i run in hive and the second below DLL is the TABLE DDL i > get from SHOW CREATE TABLE. > CREATE TABLE property_feature.property_feature_new( > `_id` STRUCT<`mls_sysid`:String,`type`:String>, > all_attrs STRUCT<`1`: String ,`10`: String ,`100`: String ,`105`: String > ,`113`: String ,`118`: String ,`126`: String ,`1299`: String ,`1301`: String > ,`1305`: String ,`1307`: String ,`1319`: String ,`1325`: String ,`1327`: > String ,`1329`: String ,`1339`: String ,`1341`: String ,`1345`: String > ,`1351`: String ,`1353`: String ,`1359`: String ,`1363`: String ,`1365`: > String ,`1367`: String ,`1373`: String ,`1377`: String ,`1397`: String > ,`1401`: String ,`1403`: String ,`1405`: String ,`1409`: String ,`1411`: > String ,`1413`: String ,`1417`: String ,`1419`: String ,`1421`: String > ,`1423`: String ,`1431`: String ,`1437`: String ,`1439`: String ,`144`: > String ,`1441`: String ,`145`: String ,`1451`: String ,`1453`: String ,`146`: > String ,`1467`: String ,`1469`: String ,`1483`: String ,`1485`: String > ,`1491`: String ,`1493`: String ,`15`: String ,`1505`: String ,`1519`: String > ,`1521`: String ,`1523`: String ,`155`: String ,`156`: String ,`158`: String > ,`1586`: String ,`1678`: String ,`169`: String ,`17`: String ,`177`: String > ,`1783`: String ,`1785`: String ,`1787`: String ,`1789`: String ,`1793`: > String ,`1795`: String ,`1797`: String ,`1799`: String ,`1801`: String > ,`1803`: String ,`1805`: String ,`1807`: String ,`1809`: String ,`1811`: > String ,`1815`: String ,`1818`: String ,`1826`: String ,`1829`: String > ,`1874`: String ,`1876`: String ,`1878`: String ,`1880`: String ,`1882`: > String ,`207`: String ,`215`: String ,`226`: String ,`2293`: String ,`232`: > String ,`239`: String ,`241`: String ,`245`: String ,`2456`: String ,`248`: > String ,`2482`: String ,`2490`: String ,`2491`: String ,`2492`: String > ,`2493`: String ,`2495`: String ,`25`: String ,`2502`: String ,`2503`: String > ,`2517`: String ,`2522`: String ,`255`: String ,`256`: String ,`2573`: String > ,`2574`: String ,`260`: String ,`263`: String ,`265`: String ,`276`: String > ,`278`: String ,`286`: String ,`293`: String ,`296`: String ,`299`: String > ,`3`: String ,`30`: String ,`307`: String ,`312`: String ,`320`: String > ,`328`: String ,`340`: String ,`348`: String ,`354`: String ,`363`: String > ,`364`: String ,`369`: String ,`375`: String ,`376`: String ,`383`: String > ,`401`: String ,`405`: String ,`409`: String ,`412`: String ,`42`: String > ,`423`: String ,`444`: String ,`445`: String ,`446`: String ,`447`: String > ,`450`: String ,`451`: String ,`452`: String ,`453`: String ,`455`: String > ,`458`: String ,`461`: String ,`463`: String ,`464`: String ,`51`: String > ,`543472980`: String ,`57`: String ,`75`: String ,`90`: String ,`92`: String > ,`AVMYN`: String ,`AccessibilityAmenities`: String ,`AddressPublic`: String > ,`Amenities`: String ,`AmountofWaterFront`: String ,`AppliancesIncluded`: > String ,`AppraiserInitials`: String ,`AppraiserName`: String ,`Basement`: > String ,`BathsFull`: String ,`BathsHalf`: String ,`BathsTotal`: String > ,`BedsTotal`: String ,`Block`: String ,`Book`: String ,`BuildingNumber`: > String ,`City`: String ,`CloseDate`: String ,`ClosePrice`: String , > `CoListAgentDirectWorkPhone`: String ,`CoListAgentEmail`: String > ,`CoListAgentFullName`: String ,`CoListAgentMLSID`: String > ,`CoListAgent_MUI`: String ,`CoListOfficeMLSID`: String ,`CoListOfficeName`:
[jira] [Updated] (HIVE-13056) delegation tokens do not work with HS2 when used with http transport and kerberos
[
https://issues.apache.org/jira/browse/HIVE-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-13056:
-
Fix Version/s: 1.3.0
> delegation tokens do not work with HS2 when used with http transport and
> kerberos
> -
>
> Key: HIVE-13056
> URL: https://issues.apache.org/jira/browse/HIVE-13056
> Project: Hive
> Issue Type: Bug
> Components: Authentication
>Affects Versions: 1.2.1
>Reporter: Cheng Xu
>Assignee: Sushanth Sowmyan
>Priority: Critical
> Fix For: 1.3.0, 2.1.0, 2.0.1
>
> Attachments: HIVE-13056.patch
>
>
> We're getting a HiveSQLException on secure windows clusters.
> {code}
> 2016-02-08
> 13:48:09,535|beaver.machine|INFO|6114|140264674350912|MainThread|Job ID :
> 000-160208134528402-oozie-oozi-W
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|Workflow
> Name : hive2-wf
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|App Path
> :
> wasb://oozie1-hb...@humbtestings5jp.blob.core.windows.net/user/hrt_qa/test_hiveserver2
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|Status
> : KILLED
> 2016-02-08
> 13:48:09,537|beaver.machine|INFO|6114|140264674350912|MainThread|Run
> : 0
> 2016-02-08
> 13:48:09,537|beaver.machine|INFO|6114|140264674350912|MainThread|User
> : hrt_qa
> 2016-02-08
> 13:48:09,537|beaver.machine|INFO|6114|140264674350912|MainThread|Group
> : -
> 2016-02-08
> 13:48:09,547|beaver.machine|INFO|6114|140264674350912|MainThread|Created
> : 2016-02-08 13:47 GMT
> 2016-02-08
> 13:48:09,548|beaver.machine|INFO|6114|140264674350912|MainThread|Started
> : 2016-02-08 13:47 GMT
> 2016-02-08
> 13:48:09,552|beaver.machine|INFO|6114|140264674350912|MainThread|Last
> Modified : 2016-02-08 13:48 GMT
> 2016-02-08
> 13:48:09,553|beaver.machine|INFO|6114|140264674350912|MainThread|Ended
> : 2016-02-08 13:48 GMT
> 2016-02-08
> 13:48:09,553|beaver.machine|INFO|6114|140264674350912|MainThread|CoordAction
> ID: -
> 2016-02-08 13:48:09,566|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,566|beaver.machine|INFO|6114|140264674350912|MainThread|Actions
> 2016-02-08
> 13:48:09,567|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,567|beaver.machine|INFO|6114|140264674350912|MainThread|ID
> Status
> Ext ID Ext Status Err Code
> 2016-02-08
> 13:48:09,567|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,571|beaver.machine|INFO|6114|140264674350912|MainThread|000-160208134528402-oozie-oozi-W@:start:
> OK- OK
> -
> 2016-02-08
> 13:48:09,572|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,572|beaver.machine|INFO|6114|140264674350912|MainThread|000-160208134528402-oozie-oozi-W@hive-node
> ERROR - ERROR
> HiveSQLException
> 2016-02-08
> 13:48:09,572|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,572|beaver.machine|INFO|6114|140264674350912|MainThread|000-160208134528402-oozie-oozi-W@fail
> OK- OK
>E0729
> 2016-02-08
> 13:48:09,572|beaver.machine|INFO|6114|140264674350912|MainThread|
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13056) delegation tokens do not work with HS2 when used with http transport and kerberos
[
https://issues.apache.org/jira/browse/HIVE-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180370#comment-15180370
]
Thejas M Nair commented on HIVE-13056:
--
Cherry picked to branch-1 as well.
> delegation tokens do not work with HS2 when used with http transport and
> kerberos
> -
>
> Key: HIVE-13056
> URL: https://issues.apache.org/jira/browse/HIVE-13056
> Project: Hive
> Issue Type: Bug
> Components: Authentication
>Affects Versions: 1.2.1
>Reporter: Cheng Xu
>Assignee: Sushanth Sowmyan
>Priority: Critical
> Fix For: 1.3.0, 2.1.0, 2.0.1
>
> Attachments: HIVE-13056.patch
>
>
> We're getting a HiveSQLException on secure windows clusters.
> {code}
> 2016-02-08
> 13:48:09,535|beaver.machine|INFO|6114|140264674350912|MainThread|Job ID :
> 000-160208134528402-oozie-oozi-W
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|Workflow
> Name : hive2-wf
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|App Path
> :
> wasb://oozie1-hb...@humbtestings5jp.blob.core.windows.net/user/hrt_qa/test_hiveserver2
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|Status
> : KILLED
> 2016-02-08
> 13:48:09,537|beaver.machine|INFO|6114|140264674350912|MainThread|Run
> : 0
> 2016-02-08
> 13:48:09,537|beaver.machine|INFO|6114|140264674350912|MainThread|User
> : hrt_qa
> 2016-02-08
> 13:48:09,537|beaver.machine|INFO|6114|140264674350912|MainThread|Group
> : -
> 2016-02-08
> 13:48:09,547|beaver.machine|INFO|6114|140264674350912|MainThread|Created
> : 2016-02-08 13:47 GMT
> 2016-02-08
> 13:48:09,548|beaver.machine|INFO|6114|140264674350912|MainThread|Started
> : 2016-02-08 13:47 GMT
> 2016-02-08
> 13:48:09,552|beaver.machine|INFO|6114|140264674350912|MainThread|Last
> Modified : 2016-02-08 13:48 GMT
> 2016-02-08
> 13:48:09,553|beaver.machine|INFO|6114|140264674350912|MainThread|Ended
> : 2016-02-08 13:48 GMT
> 2016-02-08
> 13:48:09,553|beaver.machine|INFO|6114|140264674350912|MainThread|CoordAction
> ID: -
> 2016-02-08 13:48:09,566|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,566|beaver.machine|INFO|6114|140264674350912|MainThread|Actions
> 2016-02-08
> 13:48:09,567|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,567|beaver.machine|INFO|6114|140264674350912|MainThread|ID
> Status
> Ext ID Ext Status Err Code
> 2016-02-08
> 13:48:09,567|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,571|beaver.machine|INFO|6114|140264674350912|MainThread|000-160208134528402-oozie-oozi-W@:start:
> OK- OK
> -
> 2016-02-08
> 13:48:09,572|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,572|beaver.machine|INFO|6114|140264674350912|MainThread|000-160208134528402-oozie-oozi-W@hive-node
> ERROR - ERROR
> HiveSQLException
> 2016-02-08
> 13:48:09,572|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,572|beaver.machine|INFO|6114|140264674350912|MainThread|000-160208134528402-oozie-oozi-W@fail
> OK- OK
>E0729
> 2016-02-08
> 13:48:09,572|beaver.machine|INFO|6114|140264674350912|MainThread|
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180361#comment-15180361
]
Prasanth Jayachandran commented on HIVE-13160:
--
I am not sure how we were doing it before. I have to dig it up. [~sershe] do
you know how it was done before when getAllFunctions failed and how the
registration of functions happened?
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180353#comment-15180353
]
Aihua Xu commented on HIVE-13160:
-
I see.
So if that's the case and we couldn't register all the functions, then later
the queries with the functions would fail, right? How do you register the
functions for that scenario?
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
--
This
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180344#comment-15180344
]
Sergey Shelukhin commented on HIVE-13160:
-
Yeah that might actually be a good idea to fix this. In fact, it doesn't make
sense to retry later if it's throwing the Invalid method name error. It could
be done in a separate JIRA.
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180342#comment-15180342
]
Prasanth Jayachandran commented on HIVE-13160:
--
cc/ [~sershe]
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180333#comment-15180333
]
Prasanth Jayachandran commented on HIVE-13160:
--
[~aihuaxu] I have a setup that runs 1.2.0 version of HMS. Last week I was able
to run latest 2.1.0-SNAPSHOT CLI client with 1.2.0 HMS. The thrift service of
1.2.0 does not have get_all_functions interface. When the cli starts a session,
it will internally invoke registerAllFunctionOnce which in turn calls
getAllFunctions(). Since getAllFunctions() does not exist in old HMS it will
throw an error which will be ignored. This will let CLI start (and Session)
without any issues. With this patch registerAllFunctionOnce() throws exception
will not be caught by the CLI and fails to start. This means that new clients
cannot work with old HMS. Following is the full stacktrace
{code}
Exception in thread "main" java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: Invalid method name:
'get_all_functions'
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:563)
at
org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:503)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:709)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: Invalid method name:
'get_all_functions'
at
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:211)
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:341)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:296)
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:266)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:251)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:530)
... 9 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: Invalid method name:
'get_all_functions'
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3414)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:221)
at
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:206)
... 14 more
Caused by: org.apache.thrift.TApplicationException: Invalid method name:
'get_all_functions'
at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3490)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3478)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllFunctions(HiveMetaStoreClient.java:2172)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:153)
at com.sun.proxy.$Proxy28.getAllFunctions(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2109)
at com.sun.proxy.$Proxy28.getAllFunctions(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3411)
{code}
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
[jira] [Commented] (HIVE-12045) ClassNotFoundException for GenericUDF [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180316#comment-15180316
]
Xuefu Zhang commented on HIVE-12045:
[~ztoth], yes, this is included in CDH5.7.
> ClassNotFoundException for GenericUDF [Spark Branch]
>
>
> Key: HIVE-12045
> URL: https://issues.apache.org/jira/browse/HIVE-12045
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Environment: Cloudera QuickStart VM - CDH5.4.2
> beeline
>Reporter: Zsolt Tóth
>Assignee: Rui Li
> Fix For: spark-branch, 2.1.0
>
> Attachments: HIVE-12045.1-spark.patch, HIVE-12045.2-spark.patch,
> HIVE-12045.2-spark.patch, HIVE-12045.3-spark.patch, HIVE-12045.4-spark.patch,
> HIVE-12045.patch, example.jar, genUDF.patch, hive.log.gz
>
>
> If I execute the following query in beeline, I get ClassNotFoundException for
> the UDF class.
> {code}
> drop function myGenericUdf;
> create function myGenericUdf as 'org.example.myGenericUdf' using jar
> 'hdfs:///tmp/myudf.jar';
> select distinct myGenericUdf(1,2,1) from mytable;
> {code}
> In my example, myGenericUdf just looks for the 1st argument's value in the
> others and returns the index. I don't think this is related to the actual
> GenericUDF function.
> Note that:
> "select myGenericUdf(1,2,1) from mytable;" succeeds
> If I use the non-generic implementation of the same UDF, the select distinct
> call succeeds.
> StackTrace:
> {code}
> 15/10/06 05:20:25 ERROR exec.Utilities: Failed to load plan:
> hdfs://quickstart.cloudera:8020/tmp/hive/hive/f9de3f09-c12d-4528-9ee6-1f12932a14ae/hive_2015-10-06_05-20-07_438_6519207588897968406-20/-mr-10003/27cd7226-3e22-46f4-bddd-fb8fd4aa4b8d/map.xml:
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
> class: org.example.myGenericUDF
> Serialization trace:
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.GroupByOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
> class: org.example.myGenericUDF
> Serialization trace:
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.GroupByOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> at
>
[jira] [Commented] (HIVE-11837) comments do not support unicode characters well.
[
https://issues.apache.org/jira/browse/HIVE-11837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180309#comment-15180309
]
Hive QA commented on HIVE-11837:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791019/HIVE-11837.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 9736 tests executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-vector_decimal_2.q-schema_evol_text_fetchwork_table.q-constprog_semijoin.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_decimal_round.q-cbo_windowing.q-tez_schema_evolution.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7161/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7161/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7161/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12791019 - PreCommit-HIVE-TRUNK-Build
> comments do not support unicode characters well.
>
>
> Key: HIVE-11837
> URL: https://issues.apache.org/jira/browse/HIVE-11837
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 0.13.1, 1.1.0
> Environment: Hadoop 2.7
> Hive 0.13.1 / Hive 1.1.0
> RHEL 6.4 / SLES 11.3
>Reporter: Rudd Chen
>Assignee: Rudd Chen
>Priority: Minor
> Attachments: HIVE-11837.1.patch, HIVE-11837.patch
>
>
> the terminal encoding is set to UTF-8, It can display Chinese characters.
> then I create a table with a comment in Chinese, both "show create table" and
> "desc formatted table" can not display the Chinese characters in the table
> comments, meanwhile it can display Chinese characters in column comment.. See
> below:
> 0: jdbc:hive2://ha-cluster/default> create table tt(id int comment '列中文测试')
> comment '表中文测试';
> No rows affected (0.152 seconds)
> 0: jdbc:hive2://ha-cluster/default>
> 0: jdbc:hive2://ha-cluster/default>
> 0: jdbc:hive2://ha-cluster/default> desc formatted tt;
> +---+---+-+
> | col_name| data_type
> | comment |
> +---+---+-+
> | # col_name| data_type
> | comment |
> | | NULL
> | NULL|
> | id| int
> | 列中文测试 |
> | | NULL
> | NULL|
> | # Detailed Table Information | NULL
> | NULL|
> | Database: | default
> | NULL|
> | Owner:| admin
> | NULL|
> | CreateTime: | Wed Sep 16 11:13:34 CST 2015
> | NULL|
> | LastAccessTime: | UNKNOWN
> | NULL
[jira] [Resolved] (HIVE-13207) HIVE Acid Transaction with Parquet
[ https://issues.apache.org/jira/browse/HIVE-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates resolved HIVE-13207. --- Resolution: Duplicate > HIVE Acid Transaction with Parquet > -- > > Key: HIVE-13207 > URL: https://issues.apache.org/jira/browse/HIVE-13207 > Project: Hive > Issue Type: Wish >Reporter: Marco barbieri > > Hi, > I want to know if hive supports acid transaction with Parquet File and if i > can have an example about how i can perform it. > Thank you. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13207) HIVE Acid Transaction with Parquet
[ https://issues.apache.org/jira/browse/HIVE-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180270#comment-15180270 ] Alan Gates commented on HIVE-13207: --- See HIVE-8123 which is the umbrella JIRA for the work to put make Parquet work with ACID. > HIVE Acid Transaction with Parquet > -- > > Key: HIVE-13207 > URL: https://issues.apache.org/jira/browse/HIVE-13207 > Project: Hive > Issue Type: Wish >Reporter: Marco barbieri > > Hi, > I want to know if hive supports acid transaction with Parquet File and if i > can have an example about how i can perform it. > Thank you. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180193#comment-15180193 ] Ashutosh Chauhan commented on HIVE-11160: - It will be good to have test run on this with config value to set. Can you set that to true in data/conf/hive-site.xml ? > Auto-gather column stats > > > Key: HIVE-11160 > URL: https://issues.apache.org/jira/browse/HIVE-11160 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch, > HIVE-11160.03.patch, HIVE-11160.04.patch > > > Hive will collect table stats when set hive.stats.autogather=true during the > INSERT OVERWRITE command. And then the users need to collect the column stats > themselves using "Analyze" command. In this patch, the column stats will also > be collected automatically. More specifically, INSERT OVERWRITE will > automatically create new column stats. INSERT INTO will automatically merge > new column stats with existing ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13096) Cost to choose side table in MapJoin conversion based on cumulative cardinality
[ https://issues.apache.org/jira/browse/HIVE-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13096: --- Attachment: HIVE-13096.04.patch Thanks [~ashutoshc], I will run a new QA before committing, as latest was run almost a week ago. > Cost to choose side table in MapJoin conversion based on cumulative > cardinality > --- > > Key: HIVE-13096 > URL: https://issues.apache.org/jira/browse/HIVE-13096 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.0.0, 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13096.01.patch, HIVE-13096.02.patch, > HIVE-13096.03.patch, HIVE-13096.04.patch, HIVE-13096.patch > > > HIVE-11954 changed the logic to choose the side table in the MapJoin > conversion algorithm. Initial heuristic for the cost was based on number of > heavyweight operators. > This extends that work so the heuristic is based on accumulate cardinality. > In the future, we should choose the side based on total latency for the input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13096) Cost to choose side table in MapJoin conversion based on cumulative cardinality
[ https://issues.apache.org/jira/browse/HIVE-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13096: --- Status: Patch Available (was: Open) > Cost to choose side table in MapJoin conversion based on cumulative > cardinality > --- > > Key: HIVE-13096 > URL: https://issues.apache.org/jira/browse/HIVE-13096 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.0.0, 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13096.01.patch, HIVE-13096.02.patch, > HIVE-13096.03.patch, HIVE-13096.04.patch, HIVE-13096.patch > > > HIVE-11954 changed the logic to choose the side table in the MapJoin > conversion algorithm. Initial heuristic for the cost was based on number of > heavyweight operators. > This extends that work so the heuristic is based on accumulate cardinality. > In the future, we should choose the side based on total latency for the input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13096) Cost to choose side table in MapJoin conversion based on cumulative cardinality
[ https://issues.apache.org/jira/browse/HIVE-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13096: --- Status: Open (was: Patch Available) > Cost to choose side table in MapJoin conversion based on cumulative > cardinality > --- > > Key: HIVE-13096 > URL: https://issues.apache.org/jira/browse/HIVE-13096 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.0.0, 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13096.01.patch, HIVE-13096.02.patch, > HIVE-13096.03.patch, HIVE-13096.patch > > > HIVE-11954 changed the logic to choose the side table in the MapJoin > conversion algorithm. Initial heuristic for the cost was based on number of > heavyweight operators. > This extends that work so the heuristic is based on accumulate cardinality. > In the future, we should choose the side based on total latency for the input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13072) ROW_NUMBER() function creates wrong results
[ https://issues.apache.org/jira/browse/HIVE-13072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180176#comment-15180176 ] Ashutosh Chauhan commented on HIVE-13072: - HIVE-11583 may have fixed this too. > ROW_NUMBER() function creates wrong results > --- > > Key: HIVE-13072 > URL: https://issues.apache.org/jira/browse/HIVE-13072 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Philipp Brandl >Assignee: Yongzhi Chen > > When using ROW_NUMBER() on tables with more than 25000 rows, the function > ROW_NUMBER() duplicates rows with separate row numbers. > Reproduce by using a large table with more than 25000 rows with distinct > values and then using a query involving ROW_NUMBER(). It will then result in > getting the same distinct values twice with separate row numbers apart by > 25000. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13072) ROW_NUMBER() function creates wrong results
[ https://issues.apache.org/jira/browse/HIVE-13072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180168#comment-15180168 ] Yongzhi Chen commented on HIVE-13072: - I tried hive version 1.1 for CDH, still can not reproduce the issue. > ROW_NUMBER() function creates wrong results > --- > > Key: HIVE-13072 > URL: https://issues.apache.org/jira/browse/HIVE-13072 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Philipp Brandl >Assignee: Yongzhi Chen > > When using ROW_NUMBER() on tables with more than 25000 rows, the function > ROW_NUMBER() duplicates rows with separate row numbers. > Reproduce by using a large table with more than 25000 rows with distinct > values and then using a query involving ROW_NUMBER(). It will then result in > getting the same distinct values twice with separate row numbers apart by > 25000. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13096) Cost to choose side table in MapJoin conversion based on cumulative cardinality
[ https://issues.apache.org/jira/browse/HIVE-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180164#comment-15180164 ] Ashutosh Chauhan commented on HIVE-13096: - Ok, that makes sense. +1 > Cost to choose side table in MapJoin conversion based on cumulative > cardinality > --- > > Key: HIVE-13096 > URL: https://issues.apache.org/jira/browse/HIVE-13096 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.0.0, 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13096.01.patch, HIVE-13096.02.patch, > HIVE-13096.03.patch, HIVE-13096.patch > > > HIVE-11954 changed the logic to choose the side table in the MapJoin > conversion algorithm. Initial heuristic for the cost was based on number of > heavyweight operators. > This extends that work so the heuristic is based on accumulate cardinality. > In the future, we should choose the side based on total latency for the input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6162) "multiple SLF4J bindings" warning messages when running hive CLI on Hadoop 2.0
[ https://issues.apache.org/jira/browse/HIVE-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180125#comment-15180125 ] raja commented on HIVE-6162: Hello All, I am installing Hive 2.0.0 with Hadoop 2.7.2 in ubuntu 15.10, I am getting an error while i start Hive through CLI. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Logging initialized using configuration in jar:file:/usr/local/hive/lib/hive-common-2.0.0.jar!/hive-log4j2.properties Exception in thread "main" java.lang.RuntimeException: Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql) I tried to apply this patch but failed to understand , Please help me. > "multiple SLF4J bindings" warning messages when running hive CLI on Hadoop 2.0 > -- > > Key: HIVE-6162 > URL: https://issues.apache.org/jira/browse/HIVE-6162 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 0.12.0 >Reporter: shanyu zhao >Assignee: shanyu zhao > Fix For: 0.13.0 > > Attachments: HIVE-6162.patch > > > On Hadoop 2.0, when running hive command line, we saw warnings like this: > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/C:/myhdp/hadoop-2.1.2.2.0.6.0-/share/hado > op/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/C:/myhdp/hive-0.12.0.2.0.6.0-/lib/slf4j-l > og4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13206) Create a test-sources.jar when -Psources profile is invoked
[ https://issues.apache.org/jira/browse/HIVE-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-13206: --- Status: Patch Available (was: Open) > Create a test-sources.jar when -Psources profile is invoked > --- > > Key: HIVE-13206 > URL: https://issues.apache.org/jira/browse/HIVE-13206 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Minor > Attachments: HIVE-13206.1.patch > > > It'd be nice to attach a test-sources jar alongside the others as part of the > build, to provide test resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13206) Create a test-sources.jar when -Psources profile is invoked
[ https://issues.apache.org/jira/browse/HIVE-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-13206: --- Attachment: HIVE-13206.1.patch > Create a test-sources.jar when -Psources profile is invoked > --- > > Key: HIVE-13206 > URL: https://issues.apache.org/jira/browse/HIVE-13206 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Minor > Attachments: HIVE-13206.1.patch > > > It'd be nice to attach a test-sources jar alongside the others as part of the > build, to provide test resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13129:
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks Chaoyu and Naveen to review the code.
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13129.2.patch, HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12045) ClassNotFoundException for GenericUDF [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179956#comment-15179956
]
Zsolt Tóth commented on HIVE-12045:
---
I've just noticed that this was not included in Hive 2.0. Is there any specific
reason for that? [~xuefuz] Is it possible that Cloudera will include this patch
soon in a CDH release?
> ClassNotFoundException for GenericUDF [Spark Branch]
>
>
> Key: HIVE-12045
> URL: https://issues.apache.org/jira/browse/HIVE-12045
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Environment: Cloudera QuickStart VM - CDH5.4.2
> beeline
>Reporter: Zsolt Tóth
>Assignee: Rui Li
> Fix For: spark-branch, 2.1.0
>
> Attachments: HIVE-12045.1-spark.patch, HIVE-12045.2-spark.patch,
> HIVE-12045.2-spark.patch, HIVE-12045.3-spark.patch, HIVE-12045.4-spark.patch,
> HIVE-12045.patch, example.jar, genUDF.patch, hive.log.gz
>
>
> If I execute the following query in beeline, I get ClassNotFoundException for
> the UDF class.
> {code}
> drop function myGenericUdf;
> create function myGenericUdf as 'org.example.myGenericUdf' using jar
> 'hdfs:///tmp/myudf.jar';
> select distinct myGenericUdf(1,2,1) from mytable;
> {code}
> In my example, myGenericUdf just looks for the 1st argument's value in the
> others and returns the index. I don't think this is related to the actual
> GenericUDF function.
> Note that:
> "select myGenericUdf(1,2,1) from mytable;" succeeds
> If I use the non-generic implementation of the same UDF, the select distinct
> call succeeds.
> StackTrace:
> {code}
> 15/10/06 05:20:25 ERROR exec.Utilities: Failed to load plan:
> hdfs://quickstart.cloudera:8020/tmp/hive/hive/f9de3f09-c12d-4528-9ee6-1f12932a14ae/hive_2015-10-06_05-20-07_438_6519207588897968406-20/-mr-10003/27cd7226-3e22-46f4-bddd-fb8fd4aa4b8d/map.xml:
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
> class: org.example.myGenericUDF
> Serialization trace:
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.GroupByOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
> class: org.example.myGenericUDF
> Serialization trace:
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.GroupByOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
[jira] [Commented] (HIVE-13171) Add unit test for hs2 webui
[ https://issues.apache.org/jira/browse/HIVE-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179953#comment-15179953 ] Aihua Xu commented on HIVE-13171: - Another thought: We expose the functions in OperationManager as public which allows us to call directly (seems unnecessary). The correct way probably is to call HiveSessionImpl.executeStatementAsync(). > Add unit test for hs2 webui > --- > > Key: HIVE-13171 > URL: https://issues.apache.org/jira/browse/HIVE-13171 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13171.2.patch, HIVE-13171.patch > > > With more complex changes going into webui, it is hard to manually verify all > the kinds of cases. > With HIVE-12952, HS2 webui now uses jamon, which should be more unit-testable > than plain old jsp. We can perhaps add unit test for the jamon servlets, or > test the new OperationDisplay classes queried by the servlets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13096) Cost to choose side table in MapJoin conversion based on cumulative cardinality
[
https://issues.apache.org/jira/browse/HIVE-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179951#comment-15179951
]
Jesus Camacho Rodriguez commented on HIVE-13096:
[~ashutoshc], I finally could take a look back at this one.
The heuristic change impacts the selection of table chosen for streaming, and
it might change the shape of the DAG too e.g. in the presence of GB + Join.
For instance, consider {{bucket_map_join_tez1.q}}.
- Previously, the shape was:
{noformat}
Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (CUSTOM_SIMPLE_EDGE)
{noformat}
Reducer2 contains a GB on the input from Map1 (TS on table1), followed by a
Join.
In this case, Map3 (TS on table2) is broadcasted for the Join execution that is
done in Reduce2.
- With the patch, the shape is:
{noformat}
Map 3 <- Reducer 2 (CUSTOM_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE)
{noformat}
Reducer2 contains a GB on the input from Map1 (TS on table1).
In this case, the output of GB is broadcasted for the Join execution that is
done in Map3.
> Cost to choose side table in MapJoin conversion based on cumulative
> cardinality
> ---
>
> Key: HIVE-13096
> URL: https://issues.apache.org/jira/browse/HIVE-13096
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13096.01.patch, HIVE-13096.02.patch,
> HIVE-13096.03.patch, HIVE-13096.patch
>
>
> HIVE-11954 changed the logic to choose the side table in the MapJoin
> conversion algorithm. Initial heuristic for the cost was based on number of
> heavyweight operators.
> This extends that work so the heuristic is based on accumulate cardinality.
> In the future, we should choose the side based on total latency for the input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12270: --- Attachment: HIVE-12270.2.patch Uploaded initial patch with testDBTokenStore NPE fixed. > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.1.nothrift.patch, HIVE-12270.1.patch, > HIVE-12270.2.patch, HIVE-12270.nothrift.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179939#comment-15179939
]
Aihua Xu commented on HIVE-13160:
-
[~prasanth_j] Can you clarify the incompatibility issue a little bit more? Does
this only happen to CLI? Does it only happen to old HMS?
With the patch hive will start when we have a functional HMS, allowing us to
load the functions. Before, we will ignore the failure and continue but we
would have issues later when we executing the queries since HMS is not ready.
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list
[jira] [Commented] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[
https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179798#comment-15179798
]
Hive QA commented on HIVE-12270:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791392/HIVE-12270.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9789 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.thrift.TestDBTokenStore.testDBTokenStore
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7160/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7160/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7160/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12791392 - PreCommit-HIVE-TRUNK-Build
> Add DBTokenStore support to HS2 delegation token
>
>
> Key: HIVE-12270
> URL: https://issues.apache.org/jira/browse/HIVE-12270
> Project: Hive
> Issue Type: New Feature
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-12270.1.nothrift.patch, HIVE-12270.1.patch,
> HIVE-12270.nothrift.patch
>
>
> DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is
> mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token
> support was introduced by HIVE-5155 but it used MemoryTokenStore as token
> store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access
> the token/keys information in HMS DB directly from HS2 seems not the right
> approach to support DBTokenStore in HS2. I think we should use
> HiveMetaStoreClient in HS2 instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-4662) first_value can't have more than one order by column
[
https://issues.apache.org/jira/browse/HIVE-4662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-4662:
--
Attachment: HIVE-4662.patch
> first_value can't have more than one order by column
>
>
> Key: HIVE-4662
> URL: https://issues.apache.org/jira/browse/HIVE-4662
> Project: Hive
> Issue Type: Bug
> Components: SQL
>Affects Versions: 0.11.0
>Reporter: Frans Drijver
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-4662.patch
>
>
> In the current implementation of the first_value function, it's not allowed
> to have more than one (1) order by column, as so:
> {quote}
> select distinct
> first_value(kastr.DEWNKNR) over ( partition by kastr.DEKTRNR order by
> kastr.DETRADT, kastr.DEVPDNR )
> from RTAVP_DRKASTR kastr
> ;
> {quote}
> Error given:
> {quote}
> FAILED: SemanticException Range based Window Frame can have only 1 Sort Key
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-4662) first_value can't have more than one order by column
[
https://issues.apache.org/jira/browse/HIVE-4662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-4662 started by Jesus Camacho Rodriguez.
-
> first_value can't have more than one order by column
>
>
> Key: HIVE-4662
> URL: https://issues.apache.org/jira/browse/HIVE-4662
> Project: Hive
> Issue Type: Bug
> Components: SQL
>Affects Versions: 0.11.0
>Reporter: Frans Drijver
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-4662.patch
>
>
> In the current implementation of the first_value function, it's not allowed
> to have more than one (1) order by column, as so:
> {quote}
> select distinct
> first_value(kastr.DEWNKNR) over ( partition by kastr.DEKTRNR order by
> kastr.DETRADT, kastr.DEVPDNR )
> from RTAVP_DRKASTR kastr
> ;
> {quote}
> Error given:
> {quote}
> FAILED: SemanticException Range based Window Frame can have only 1 Sort Key
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-4662) first_value can't have more than one order by column
[
https://issues.apache.org/jira/browse/HIVE-4662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-4662:
-
Assignee: Jesus Camacho Rodriguez
> first_value can't have more than one order by column
>
>
> Key: HIVE-4662
> URL: https://issues.apache.org/jira/browse/HIVE-4662
> Project: Hive
> Issue Type: Bug
> Components: SQL
>Affects Versions: 0.11.0
>Reporter: Frans Drijver
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-4662.patch
>
>
> In the current implementation of the first_value function, it's not allowed
> to have more than one (1) order by column, as so:
> {quote}
> select distinct
> first_value(kastr.DEWNKNR) over ( partition by kastr.DEKTRNR order by
> kastr.DETRADT, kastr.DEVPDNR )
> from RTAVP_DRKASTR kastr
> ;
> {quote}
> Error given:
> {quote}
> FAILED: SemanticException Range based Window Frame can have only 1 Sort Key
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-4662) first_value can't have more than one order by column
[
https://issues.apache.org/jira/browse/HIVE-4662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-4662:
--
Status: Patch Available (was: In Progress)
> first_value can't have more than one order by column
>
>
> Key: HIVE-4662
> URL: https://issues.apache.org/jira/browse/HIVE-4662
> Project: Hive
> Issue Type: Bug
> Components: SQL
>Affects Versions: 0.11.0
>Reporter: Frans Drijver
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-4662.patch
>
>
> In the current implementation of the first_value function, it's not allowed
> to have more than one (1) order by column, as so:
> {quote}
> select distinct
> first_value(kastr.DEWNKNR) over ( partition by kastr.DEKTRNR order by
> kastr.DETRADT, kastr.DEVPDNR )
> from RTAVP_DRKASTR kastr
> ;
> {quote}
> Error given:
> {quote}
> FAILED: SemanticException Range based Window Frame can have only 1 Sort Key
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[
https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179666#comment-15179666
]
Gopal V commented on HIVE-13204:
h2. With
{code}
Status: Running (Executing on YARN cluster with App id
application_1455662455106_2535)
--
VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
FAILED KILLED
--
Map 1 .. llap SUCCEEDED27027000
0 0
Reducer 2 .. llap SUCCEEDED 30 3000
0 0
--
VERTICES: 02/02 [==>>] 100% ELAPSED TIME: 73.26 s
--
Status: DAG finished successfully in 73.26 seconds
{code}
h2. Without
{code}
Status: Running (Executing on YARN cluster with App id
application_1455662455106_2535)
--
VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
FAILED KILLED
--
Map 1 .. llap SUCCEEDED27027000
0 0
Reducer 2 .. llap SUCCEEDED 30 3000
0 0
--
VERTICES: 02/02 [==>>] 100% ELAPSED TIME: 378.36 s
--
Status: DAG finished successfully in 378.36 seconds
{code}
> Vectorization: Add ChainedCheckerFactory for LIKE
> --
>
> Key: HIVE-13204
> URL: https://issues.apache.org/jira/browse/HIVE-13204
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-13204.1.patch, regex-cpu-like.png
>
>
> Currently, Vectorization runs through a UTF-8 decode to produce a String &
> then check for mildly complex patterns like "http://%.exe; using a Regex.
> Since this pattern doesn't need any backtracking patterns, using a full
> fledged Regex is too expensive.
> !regex-cpu-like.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13204: --- Attachment: HIVE-13204.1.patch > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13204.1.patch, regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13204: --- Attachment: (was: HIVE-13204.1.patch) > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13204: --- Attachment: (was: HIVE-13204.1.patch) > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13204.1.patch, regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13204: --- Attachment: HIVE-13204.1.patch > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13204.1.patch, regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13204: --- Attachment: HIVE-13204.1.patch > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13204.1.patch, regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13203) STRUCT type not allowing more than 266 fields. Table get created with more fields but throws error while querying against the table. SHOW CREATE TABLE skips having some
[ https://issues.apache.org/jira/browse/HIVE-13203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] VISHNU S NAIR reassigned HIVE-13203: Assignee: VISHNU S NAIR > STRUCT type not allowing more than 266 fields. Table get created with more > fields but throws error while querying against the table. SHOW CREATE TABLE > skips having some fields in STRUCT. > -- > > Key: HIVE-13203 > URL: https://issues.apache.org/jira/browse/HIVE-13203 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 1.2.0 >Reporter: Satyajit varma >Assignee: VISHNU S NAIR > > I am trying to create a table with multiple STRUCT column types, one of which > having more than 300 fields. > I get the table created with no problem, but when i load i face few errors in > loading up the data.when i check SHOW CREATE TABLE TABLENAME, i get fields > only till 266th column and rest of them do not appear in the table DDL.(check > the issue at StateOrProvinc, search for StateOrProvinc in the second DDL). > Would like to know, if there is any restriction on the number of fields to be > using in STRUCT datatype. > EXAMPLE: > BELOW IS THE DDL i run in hive and the second below DLL is the TABLE DDL i > get from SHOW CREATE TABLE. > CREATE TABLE property_feature.property_feature_new( > `_id` STRUCT<`mls_sysid`:String,`type`:String>, > all_attrs STRUCT<`1`: String ,`10`: String ,`100`: String ,`105`: String > ,`113`: String ,`118`: String ,`126`: String ,`1299`: String ,`1301`: String > ,`1305`: String ,`1307`: String ,`1319`: String ,`1325`: String ,`1327`: > String ,`1329`: String ,`1339`: String ,`1341`: String ,`1345`: String > ,`1351`: String ,`1353`: String ,`1359`: String ,`1363`: String ,`1365`: > String ,`1367`: String ,`1373`: String ,`1377`: String ,`1397`: String > ,`1401`: String ,`1403`: String ,`1405`: String ,`1409`: String ,`1411`: > String ,`1413`: String ,`1417`: String ,`1419`: String ,`1421`: String > ,`1423`: String ,`1431`: String ,`1437`: String ,`1439`: String ,`144`: > String ,`1441`: String ,`145`: String ,`1451`: String ,`1453`: String ,`146`: > String ,`1467`: String ,`1469`: String ,`1483`: String ,`1485`: String > ,`1491`: String ,`1493`: String ,`15`: String ,`1505`: String ,`1519`: String > ,`1521`: String ,`1523`: String ,`155`: String ,`156`: String ,`158`: String > ,`1586`: String ,`1678`: String ,`169`: String ,`17`: String ,`177`: String > ,`1783`: String ,`1785`: String ,`1787`: String ,`1789`: String ,`1793`: > String ,`1795`: String ,`1797`: String ,`1799`: String ,`1801`: String > ,`1803`: String ,`1805`: String ,`1807`: String ,`1809`: String ,`1811`: > String ,`1815`: String ,`1818`: String ,`1826`: String ,`1829`: String > ,`1874`: String ,`1876`: String ,`1878`: String ,`1880`: String ,`1882`: > String ,`207`: String ,`215`: String ,`226`: String ,`2293`: String ,`232`: > String ,`239`: String ,`241`: String ,`245`: String ,`2456`: String ,`248`: > String ,`2482`: String ,`2490`: String ,`2491`: String ,`2492`: String > ,`2493`: String ,`2495`: String ,`25`: String ,`2502`: String ,`2503`: String > ,`2517`: String ,`2522`: String ,`255`: String ,`256`: String ,`2573`: String > ,`2574`: String ,`260`: String ,`263`: String ,`265`: String ,`276`: String > ,`278`: String ,`286`: String ,`293`: String ,`296`: String ,`299`: String > ,`3`: String ,`30`: String ,`307`: String ,`312`: String ,`320`: String > ,`328`: String ,`340`: String ,`348`: String ,`354`: String ,`363`: String > ,`364`: String ,`369`: String ,`375`: String ,`376`: String ,`383`: String > ,`401`: String ,`405`: String ,`409`: String ,`412`: String ,`42`: String > ,`423`: String ,`444`: String ,`445`: String ,`446`: String ,`447`: String > ,`450`: String ,`451`: String ,`452`: String ,`453`: String ,`455`: String > ,`458`: String ,`461`: String ,`463`: String ,`464`: String ,`51`: String > ,`543472980`: String ,`57`: String ,`75`: String ,`90`: String ,`92`: String > ,`AVMYN`: String ,`AccessibilityAmenities`: String ,`AddressPublic`: String > ,`Amenities`: String ,`AmountofWaterFront`: String ,`AppliancesIncluded`: > String ,`AppraiserInitials`: String ,`AppraiserName`: String ,`Basement`: > String ,`BathsFull`: String ,`BathsHalf`: String ,`BathsTotal`: String > ,`BedsTotal`: String ,`Block`: String ,`Book`: String ,`BuildingNumber`: > String ,`City`: String ,`CloseDate`: String ,`ClosePrice`: String , > `CoListAgentDirectWorkPhone`: String ,`CoListAgentEmail`: String > ,`CoListAgentFullName`: String ,`CoListAgentMLSID`: String > ,`CoListAgent_MUI`: String ,`CoListOfficeMLSID`: String ,`CoListOfficeName`: > String
[jira] [Work started] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13204 started by Gopal V. -- > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-13204: -- Assignee: Gopal V > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] VISHNU S NAIR updated HIVE-13204: - Assignee: (was: VISHNU S NAIR) > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V > Attachments: regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] VISHNU S NAIR reassigned HIVE-13204: Assignee: VISHNU S NAIR (was: Gopal V) > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: VISHNU S NAIR > Attachments: regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[
https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179585#comment-15179585
]
Hive QA commented on HIVE-13178:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791255/HIVE-13178.02.patch
{color:green}SUCCESS:{color} +1 due to 26 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 9701 tests
executed
*Failed tests:*
{noformat}
TestPerfCliDriver - did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_orc_replace_columns2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_orc_replace_columns2_acid
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_orc_replace_columns3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_orc_replace_columns3_acid
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_orc_type_promotion1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_orc_type_promotion1_acid
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_orc_type_promotion2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_orc_type_promotion2_acid
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_orc_type_promotion3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_orc_type_promotion3_acid
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7159/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7159/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7159/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12791255 - PreCommit-HIVE-TRUNK-Build
> Enhance ORC Schema Evolution to handle more standard data type conversions
> --
>
> Key: HIVE-13178
> URL: https://issues.apache.org/jira/browse/HIVE-13178
> Project: Hive
> Issue Type: Bug
> Components: Hive, ORC
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch
>
>
> Currently, SHORT -> INT -> BIGINT is supported.
> Handle ORC data type conversions permitted by Implicit conversion allowed by
> TypeIntoUtils.implicitConvertible method.
>* STRING_GROUP -> DOUBLE
>* STRING_GROUP -> DECIMAL
>* DATE_GROUP -> STRING
>* NUMERIC_GROUP -> STRING
>* STRING_GROUP -> STRING_GROUP
>*
>* // Upward from "lower" type to "higher" numeric type:
>* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13189) Consider using Joda DateTimeFormatter instead of SimpleDateFormat in GenericUDFDateAdd
[
https://issues.apache.org/jira/browse/HIVE-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179583#comment-15179583
]
Gopal V commented on HIVE-13189:
[~rajesh.balamohan]: can you add a case with a prefixed space? Not sure if that
works with regular DateTime, but the indexOf() looks rather odd.
> Consider using Joda DateTimeFormatter instead of SimpleDateFormat in
> GenericUDFDateAdd
> --
>
> Key: HIVE-13189
> URL: https://issues.apache.org/jira/browse/HIVE-13189
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: varun a kumar
> Attachments: HIVE-13189.1.patch
>
>
> Quite an amount was spent by tasks in trying to parse date string in
> GenericUDFDateAdd.
> {noformat}
> java.lang.Thread.State: RUNNABLE
> at java.text.DecimalFormat.subparse(DecimalFormat.java:1467)
> at java.text.DecimalFormat.parse(DecimalFormat.java:1268)
> at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:2088)
> at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1455)
> at java.text.DateFormat.parse(DateFormat.java:355)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFDateAdd.evaluate(GenericUDFDateAdd.java:172)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:87)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPGreaterThan.evaluate(GenericUDFOPGreaterThan.java:80)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:108)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.internalForward(CommonJoinOperator.java:644)
> {noformat}
> Joda DateTimeFormatter can be considered for better performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13189) Consider using Joda DateTimeFormatter instead of SimpleDateFormat in GenericUDFDateAdd
[
https://issues.apache.org/jira/browse/HIVE-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179584#comment-15179584
]
Gopal V commented on HIVE-13189:
[~rajesh.balamohan]: can you add a case with a prefixed space? Not sure if that
works with regular DateTime, but the indexOf() looks rather odd.
> Consider using Joda DateTimeFormatter instead of SimpleDateFormat in
> GenericUDFDateAdd
> --
>
> Key: HIVE-13189
> URL: https://issues.apache.org/jira/browse/HIVE-13189
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: varun a kumar
> Attachments: HIVE-13189.1.patch
>
>
> Quite an amount was spent by tasks in trying to parse date string in
> GenericUDFDateAdd.
> {noformat}
> java.lang.Thread.State: RUNNABLE
> at java.text.DecimalFormat.subparse(DecimalFormat.java:1467)
> at java.text.DecimalFormat.parse(DecimalFormat.java:1268)
> at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:2088)
> at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1455)
> at java.text.DateFormat.parse(DateFormat.java:355)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFDateAdd.evaluate(GenericUDFDateAdd.java:172)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:87)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPGreaterThan.evaluate(GenericUDFOPGreaterThan.java:80)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:108)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.internalForward(CommonJoinOperator.java:644)
> {noformat}
> Joda DateTimeFormatter can be considered for better performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
