[jira] [Commented] (HIVE-14728) Redundant orig files
[
https://issues.apache.org/jira/browse/HIVE-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15479211#comment-15479211
]
Hive QA commented on HIVE-14728:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827870/HIVE-14728.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10517 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1147/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1147/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1147/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12827870 - PreCommit-HIVE-MASTER-Build
> Redundant orig files
>
>
> Key: HIVE-14728
> URL: https://issues.apache.org/jira/browse/HIVE-14728
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Minor
> Attachments: HIVE-14728.1.patch
>
>
> I find some orig files in master, e.g. SemanticAnalyzer.java.orig. Wondering
> if they are added by mistake?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14727) llap-server may case file descriptor leak in BuddyAllocator class
[ https://issues.apache.org/jira/browse/HIVE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yechao Chen updated HIVE-14727: --- Fix Version/s: 2.2.0 > llap-server may case file descriptor leak in BuddyAllocator class > - > > Key: HIVE-14727 > URL: https://issues.apache.org/jira/browse/HIVE-14727 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Yechao Chen >Assignee: Yechao Chen > Fix For: 2.2.0 > > Attachments: HIVE-14727.patch > > > llap-server,the method preallocate(int) of BuddyAllocator may case file > descriptor leak when FileChannel map allocate memory error. > the code: > //here if failed > ByteBuffer rwbuf = rwf.getChannel().map(MapMode.READ_WRITE, 0, > arenaSize); > // A mapping, once established, is not dependent upon the file > channel that was used to > // create it. delete file and hold onto the map >//can not close() and delete file > rwf.close(); > rf.delete(); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14719) ASTNode rootNode is not maintained properly when changing child/parent relation
[
https://issues.apache.org/jira/browse/HIVE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15479085#comment-15479085
]
Hive QA commented on HIVE-14719:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827869/HIVE-14719.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10547 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1146/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1146/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1146/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12827869 - PreCommit-HIVE-MASTER-Build
> ASTNode rootNode is not maintained properly when changing child/parent
> relation
> ---
>
> Key: HIVE-14719
> URL: https://issues.apache.org/jira/browse/HIVE-14719
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14719.1.patch, HIVE-14719.2.patch
>
>
> When I run some query like:
> {code}
> set hive.cbo.enable=false;
> select * from A where exists (select * from B where B.k1=A.k1 and B.k2=A.k2);
> {code}
> It gets error like:
> {noformat}
> FAILED: SemanticException Line 0:-1 Invalid table alias or column reference
> 'sq_1': (possible column names are: _table_or_col b) k2) sq_corr_1)) (tok, (.
> (tok_table_or_col sq_1) sq_corr_1))
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6517) Improvement on HiveServer2 JDBC: in Kerberos auth mode, accept username / password passed from the client's SessionConf
[
https://issues.apache.org/jira/browse/HIVE-6517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15479051#comment-15479051
]
Muthuraj Ramasamy commented on HIVE-6517:
-
Can HiveDriver class be modified to use preauthenticated subject creation in
this scenario. Below has the sample code which i used for testing purpose.
Reference : https://issues.apache.org/jira/browse/HIVE-6486
Sample Connection String :
jdbc:hive2://%HIVESERVER2%:1/default;krb5_kdc=%KDC%;principal=hive/%HIVESERVER2%@%REALM%;auth=kerberos;kerberosAuthType=fromSubject
public Connection connect(final String url, final Properties info)
throws SQLException {
if (url.contains("kerberosAuthType=fromSubject")) {
return getConnection(url, info);
} else {
return acceptsURL(url) ? new HiveConnection(url, info)
: null;
}
}
private Connection getConnection(final String url, final Properties
info) throws SQLException {
synchronized (SYNCHRONIZED_OBJ) {
String[] urlParams = url.split(";");
String krb5_realm = null;
String krb5_kdc = null;
JaasConfiguration jaasConf = new JaasConfiguration();
Configuration.setConfiguration(jaasConf);
for (String _urlParam : urlParams) {
if (_urlParam.contains("principal=")) {
krb5_realm =
_urlParam.replace("krb5_realm=", "");
krb5_realm =
krb5_realm.substring(krb5_realm.indexOf("@") + 1);
}
if (_urlParam.contains("krb5_kdc=")) {
krb5_kdc =
_urlParam.replace("krb5_kdc=", "");
}
}
System.setProperty("java.security.krb5.realm",
krb5_realm);
System.setProperty("java.security.krb5.kdc", krb5_kdc);
String subjectName = "SampleClient" + (new
Date().getTime());
Subject signedOnUserSubject =
getSubject(info.getProperty("user"), info.getProperty("password"), krb5_realm,
krb5_kdc, subjectName);
Connection conn = null;
try {
conn = (Connection)
Subject.doAs(signedOnUserSubject, new PrivilegedExceptionAction() {
public Object run() {
Connection con = null;
try {
con = acceptsURL(url) ?
new HiveConnection(url, info) : null;
} catch (SQLException e) {
e.printStackTrace();
}
return con;
}
});
} catch (PrivilegedActionException e) {
new SQLException(e);
}
return conn;
}
}
private static class JaasConfiguration extends Configuration {
@Override
public AppConfigurationEntry[] getAppConfigurationEntry(String
appName) {
Map krbOptions = new HashMap();
krbOptions.put("refreshKrb5Config", "true");
krbOptions.put("storeKey", "true");
AppConfigurationEntry testClientEntry = new
AppConfigurationEntry("com.sun.security.auth.module.Krb5LoginModule",
LoginModuleControlFlag.REQUIRED, krbOptions);
return new AppConfigurationEntry[] { testClientEntry };
}
}
public class MyCallbackHandler implements CallbackHandler {
String username = null;
String krb5_realm = null;
String krb5_kdc = null;
String password = null;
public MyCallbackHandler(String username, String password,
String krb5_realm, String krb5_kdc) {
this.username = username;
this.krb5_realm = krb5_realm;
this.krb5_kdc = krb5_kdc;
this.password = password;
}
public void handle(Callback[] callbacks) throws IOException,
UnsupportedCallbackException {
for (int i = 0; i < callbacks.length; i++) {
if (callbacks[i] instanceof
[jira] [Commented] (HIVE-14728) Redundant orig files
[ https://issues.apache.org/jira/browse/HIVE-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15479025#comment-15479025 ] Pengcheng Xiong commented on HIVE-14728: +1. Sorry that I added some of them by mistake. > Redundant orig files > > > Key: HIVE-14728 > URL: https://issues.apache.org/jira/browse/HIVE-14728 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HIVE-14728.1.patch > > > I find some orig files in master, e.g. SemanticAnalyzer.java.orig. Wondering > if they are added by mistake? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[
https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478974#comment-15478974
]
Hive QA commented on HIVE-14731:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827862/HIVE-14731.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 124 failed/errored test(s), 10547 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[archive_excludeHadoop20]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[archive_multi]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_explain]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_filters]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_const]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_cross_product_check_2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_join1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[complex_alias]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constantPropagateForSubQuery]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constant_prop_1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constant_prop_3]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog3]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_partitioner]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cross_join]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cross_join_merge]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cross_prod_1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cross_product_check_1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cross_product_check_2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_5]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_basic2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynamic_partition_pruning]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input23]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join23]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join42]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_alt_syntax]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_3]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_unqual1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_unqual3]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_filters]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_nulls]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_view]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lineage3]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mergejoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[no_hooks]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonblock_op_deduplicate]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[optimize_nullscan]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parallel_join0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pointlookup2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pointlookup3]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_outer_join5]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_repeated_alias]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_udf_case]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[print_header]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reduce_deduplicate_extended2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_25]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_multiinsert]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_notin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_notin_having]
[jira] [Assigned] (HIVE-1362) Optimizer statistics on columns in tables and partitions
[ https://issues.apache.org/jira/browse/HIVE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang reassigned HIVE-1362: -- Assignee: Zhenhua Wang (was: Shreepadma Venugopalan) > Optimizer statistics on columns in tables and partitions > > > Key: HIVE-1362 > URL: https://issues.apache.org/jira/browse/HIVE-1362 > Project: Hive > Issue Type: Sub-task > Components: Statistics >Reporter: Ning Zhang >Assignee: Zhenhua Wang > Fix For: 0.10.0 > > Attachments: HIVE-1362-gen_thrift.1.patch.txt, > HIVE-1362-gen_thrift.2.patch.txt, HIVE-1362-gen_thrift.3.patch.txt, > HIVE-1362-gen_thrift.4.patch.txt, HIVE-1362-gen_thrift.5.patch.txt, > HIVE-1362-gen_thrift.6.patch.txt, HIVE-1362.1.patch.txt, > HIVE-1362.10.patch.txt, HIVE-1362.11.patch.txt, HIVE-1362.2.patch.txt, > HIVE-1362.3.patch.txt, HIVE-1362.4.patch.txt, HIVE-1362.5.patch.txt, > HIVE-1362.6.patch.txt, HIVE-1362.7.patch.txt, HIVE-1362.8.patch.txt, > HIVE-1362.9.patch.txt, HIVE-1362.D6339.1.patch, > HIVE-1362_gen-thrift.10.patch.txt, HIVE-1362_gen-thrift.7.patch.txt, > HIVE-1362_gen-thrift.8.patch.txt, HIVE-1362_gen-thrift.9.patch.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14700) clean up file/txn information via a metastore thread similar to compactor
[ https://issues.apache.org/jira/browse/HIVE-14700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14700: Attachment: HIVE-14700.WIP.patch WIP patch here for now... untested > clean up file/txn information via a metastore thread similar to compactor > - > > Key: HIVE-14700 > URL: https://issues.apache.org/jira/browse/HIVE-14700 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14700.WIP.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14728) Redundant orig files
[ https://issues.apache.org/jira/browse/HIVE-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478898#comment-15478898 ] Rui Li commented on HIVE-14728: --- Seems the files came in with HIVE-14362. [~pxiong] could you take a look? Thanks. > Redundant orig files > > > Key: HIVE-14728 > URL: https://issues.apache.org/jira/browse/HIVE-14728 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HIVE-14728.1.patch > > > I find some orig files in master, e.g. SemanticAnalyzer.java.orig. Wondering > if they are added by mistake? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14728) Redundant orig files
[ https://issues.apache.org/jira/browse/HIVE-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14728: -- Status: Patch Available (was: Open) > Redundant orig files > > > Key: HIVE-14728 > URL: https://issues.apache.org/jira/browse/HIVE-14728 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HIVE-14728.1.patch > > > I find some orig files in master, e.g. SemanticAnalyzer.java.orig. Wondering > if they are added by mistake? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14728) Redundant orig files
[ https://issues.apache.org/jira/browse/HIVE-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14728: -- Attachment: HIVE-14728.1.patch > Redundant orig files > > > Key: HIVE-14728 > URL: https://issues.apache.org/jira/browse/HIVE-14728 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HIVE-14728.1.patch > > > I find some orig files in master, e.g. SemanticAnalyzer.java.orig. Wondering > if they are added by mistake? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14728) Redundant orig files
[ https://issues.apache.org/jira/browse/HIVE-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-14728: - Assignee: Rui Li > Redundant orig files > > > Key: HIVE-14728 > URL: https://issues.apache.org/jira/browse/HIVE-14728 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HIVE-14728.1.patch > > > I find some orig files in master, e.g. SemanticAnalyzer.java.orig. Wondering > if they are added by mistake? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14719) ASTNode rootNode is not maintained properly when changing child/parent relation
[
https://issues.apache.org/jira/browse/HIVE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-14719:
--
Attachment: HIVE-14719.2.patch
Thank you [~ashutoshc] for your comments. Update patch to add test. It clears
rootNode only for {{setChild}}. Ideally maybe we should also do it for
{{deleteChild}}, {{replaceChildren}}, {{addChild}} etc. But I'm afraid that's
bad for performance.
> ASTNode rootNode is not maintained properly when changing child/parent
> relation
> ---
>
> Key: HIVE-14719
> URL: https://issues.apache.org/jira/browse/HIVE-14719
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14719.1.patch, HIVE-14719.2.patch
>
>
> When I run some query like:
> {code}
> set hive.cbo.enable=false;
> select * from A where exists (select * from B where B.k1=A.k1 and B.k2=A.k2);
> {code}
> It gets error like:
> {noformat}
> FAILED: SemanticException Line 0:-1 Invalid table alias or column reference
> 'sq_1': (possible column names are: _table_or_col b) k2) sq_corr_1)) (tok, (.
> (tok_table_or_col sq_1) sq_corr_1))
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Status: Patch Available (was: Open) > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14726) delete statement fails when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478709#comment-15478709 ] Eugene Koifman commented on HIVE-14726: --- every update/delete statement in dynpart_sort_optimization_acid.q has "key" which is the bucketing column in the WHERE clause. Perhaps that's why this works. For ACID data only needs to be sorted in each bucket by ROW__ID. Thus if we sort globally by partition/bucket/ROW__ID, acid would still work. > delete statement fails when spdo is on > -- > > Key: HIVE-14726 > URL: https://issues.apache.org/jira/browse/HIVE-14726 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Deepesh Khandelwal >Assignee: Ashutosh Chauhan > Attachments: HIVE-14726.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14558) Add support for listing views similar to "show tables"
[
https://issues.apache.org/jira/browse/HIVE-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478707#comment-15478707
]
Hive QA commented on HIVE-14558:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827822/HIVE-14558.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10547 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1143/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1143/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1143/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12827822 - PreCommit-HIVE-MASTER-Build
> Add support for listing views similar to "show tables"
> --
>
> Key: HIVE-14558
> URL: https://issues.apache.org/jira/browse/HIVE-14558
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14558.1.patch
>
>
> Users have been asking for such feature where they can get a lists of views
> separately.
> So perhaps a syntax similar to "show tables" command?
> show views [in/from ] []
> Does it make sense to add such command? or is it not worth the effort?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.1.patch Here is the patch based on initial patch provided by [~hagleitn]. It modifies root pom so that we can test it against tez 0.9.0-SNAPSHOT. A qtest has been added into mini llap qtest. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14726) delete statement fails when spdo is on
[
https://issues.apache.org/jira/browse/HIVE-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478698#comment-15478698
]
Ashutosh Chauhan edited comment on HIVE-14726 at 9/9/16 11:52 PM:
--
Thanks [~alangates] for looking into it. So, we begin with {{ROW__ID}} in sort
and shuffle key and partition column as value of RS. In this optimization if we
change RS sort and shuffle key to ROW__ID and partition column, we need not to
keep orc buffer open and thus avoid OOM case. But is that ok with ACID writer
in FileSink or it assumes sort and shuffle keys as {{ROW__ID}} and nothing else?
was (Author: ashutoshc):
Thanks [~alangates] for looking into it. So, we begin with ROW__ID in sort and
shuffle key and partition column as value of RS. In this optimization if we
change RS sort and shuffle key to ROW__ID and partition column, we need not to
keep orc buffer open and thus avoid OOM case. But is that ok with ACID writer
in FileSink or it assumes sort and shuffle keys as ROW__ID and nothing else?
> delete statement fails when spdo is on
> --
>
> Key: HIVE-14726
> URL: https://issues.apache.org/jira/browse/HIVE-14726
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 2.1.0
>Reporter: Deepesh Khandelwal
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-14726.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14726) delete statement fails when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478698#comment-15478698 ] Ashutosh Chauhan commented on HIVE-14726: - Thanks [~alangates] for looking into it. So, we begin with ROW__ID in sort and shuffle key and partition column as value of RS. In this optimization if we change RS sort and shuffle key to ROW__ID and partition column, we need not to keep orc buffer open and thus avoid OOM case. But is that ok with ACID writer in FileSink or it assumes sort and shuffle keys as ROW__ID and nothing else? > delete statement fails when spdo is on > -- > > Key: HIVE-14726 > URL: https://issues.apache.org/jira/browse/HIVE-14726 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Deepesh Khandelwal >Assignee: Ashutosh Chauhan > Attachments: HIVE-14726.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14729) analyze table compute stats for columns should collect basic stats as well
[
https://issues.apache.org/jira/browse/HIVE-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478679#comment-15478679
]
Ashutosh Chauhan commented on HIVE-14729:
-
yeah.. with auto gather stats they never need to run any analyze statements..
without it it will be good to run just 1, instead of current 2.
> analyze table compute stats for columns should collect basic stats as well
> --
>
> Key: HIVE-14729
> URL: https://issues.apache.org/jira/browse/HIVE-14729
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Ashutosh Chauhan
>
> Currently user need to run following two statements to collect stats:
> {code}
> analyze table T compute statistics;
> analyze table T compute statistics for columns;
> {code}
> Its little confusing for users that they need to collect both stats. Since
> basic stats are cheap to compute, we should collect them even in case of
> column stats, so that by running single statement users can get all stats.
> This leaves with the possibility of fast basic stats collection with first
> statement which will be good to have.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14726) delete statement fails when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478671#comment-15478671 ] Alan Gates commented on HIVE-14726: --- I think the issue is that SortDynPartitionOptimizer is using the bucket column numbers to index into the array of columns. UpdateDeleteSemanticAnalyzer is rewriting a delete query into an insert that contains only partition columns (see around line 150ff). This means the bucket columns aren't in the rewritten query. The bucket id is present since it's part of the ROW__ID, but we may need to write special code in SDPO to fetch it from there rather than look for the column. > delete statement fails when spdo is on > -- > > Key: HIVE-14726 > URL: https://issues.apache.org/jira/browse/HIVE-14726 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Deepesh Khandelwal >Assignee: Ashutosh Chauhan > Attachments: HIVE-14726.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13589) beeline - support prompt for password with '-u' option
[ https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478651#comment-15478651 ] Vihang Karajgaonkar commented on HIVE-13589: Hi [~Ke Jia] I think I have some ideas to work around the issue which we discussed in HIVE-14717. Since I have already spend some time on this issue and I think I have a patch which is almost ready, is it okay if I work on this JIRA? Please let me know. Thanks! > beeline - support prompt for password with '-u' option > -- > > Key: HIVE-13589 > URL: https://issues.apache.org/jira/browse/HIVE-13589 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Thejas M Nair >Assignee: Ke Jia > Fix For: 2.2.0 > > Attachments: HIVE-13589.1.patch, HIVE-13589.2.patch, > HIVE-13589.3.patch, HIVE-13589.4.patch, HIVE-13589.5.patch, HIVE-13589.6.patch > > > Specifying connection string using commandline options in beeline is > convenient, as it gets saved in shell command history, and it is easy to > retrieve it from there. > However, specifying the password in command prompt is not secure as it gets > displayed on screen and saved in the history. > It should be possible to specify '-p' without an argument to make beeline > prompt for password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-14717) Beeline tests failing
[
https://issues.apache.org/jira/browse/HIVE-14717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar resolved HIVE-14717.
Resolution: Fixed
Change which caused the failure was reverted. Tests are working now. See
HIVE-13589 for more details.
> Beeline tests failing
> -
>
> Key: HIVE-14717
> URL: https://issues.apache.org/jira/browse/HIVE-14717
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Critical
> Attachments: HIVE-14717.01.patch
>
>
> If you run mvn clean test in beeline you see the following errors:
> {noformat}
> ---
> T E S T S
> ---
> Running org.apache.hive.beeline.cli.TestHiveCli
> Running org.apache.hive.beeline.TestBeelineArgParsing
> Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.584 sec -
> in org.apache.hive.beeline.TestBeelineArgParsing
> Running org.apache.hive.beeline.TestBeeLineHistory
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.198 sec -
> in org.apache.hive.beeline.TestBeeLineHistory
> Running org.apache.hive.beeline.TestBeeLineOpts
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.338 sec -
> in org.apache.hive.beeline.TestBeeLineOpts
> Running org.apache.hive.beeline.TestBufferedRows
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.382 sec -
> in org.apache.hive.beeline.TestBufferedRows
> Running org.apache.hive.beeline.TestClientCommandHookFactory
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.546 sec -
> in org.apache.hive.beeline.TestClientCommandHookFactory
> Running org.apache.hive.beeline.TestIncrementalRowsWithNormalization
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.379 sec -
> in org.apache.hive.beeline.TestIncrementalRowsWithNormalization
> Running org.apache.hive.beeline.TestTableOutputFormat
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.402 sec -
> in org.apache.hive.beeline.TestTableOutputFormat
> Results :
> Tests run: 44, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 01:03 min
> [INFO] Finished at: 2016-09-07T10:57:28-07:00
> [INFO] Final Memory: 65M/949M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on
> project hive-beeline: ExecutionException The forked VM terminated without
> properly saying goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd
> /Users/vihang/work/src/upstream/hive/beeline &&
> /Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/jre/bin/java
> -Xmx2048m -XX:MaxPermSize=512m -jar
> /Users/vihang/work/src/upstream/hive/beeline/target/surefire/surefirebooter8133887423099901737.jar
>
> /Users/vihang/work/src/upstream/hive/beeline/target/surefire/surefire6528142441948588259tmp
>
> /Users/vihang/work/src/upstream/hive/beeline/target/surefire/surefire_05570572112194455658tmp
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> {noformat}
> My guess is this is related to recent fix for HIVE-13589 but needs more
> investigation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14717) Beeline tests failing
[
https://issues.apache.org/jira/browse/HIVE-14717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478612#comment-15478612
]
Vihang Karajgaonkar commented on HIVE-14717:
Closing this issue now since the change which caused the test failure has been
reverted. We can continue this discussion on the original JIRA.
> Beeline tests failing
> -
>
> Key: HIVE-14717
> URL: https://issues.apache.org/jira/browse/HIVE-14717
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Critical
> Attachments: HIVE-14717.01.patch
>
>
> If you run mvn clean test in beeline you see the following errors:
> {noformat}
> ---
> T E S T S
> ---
> Running org.apache.hive.beeline.cli.TestHiveCli
> Running org.apache.hive.beeline.TestBeelineArgParsing
> Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.584 sec -
> in org.apache.hive.beeline.TestBeelineArgParsing
> Running org.apache.hive.beeline.TestBeeLineHistory
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.198 sec -
> in org.apache.hive.beeline.TestBeeLineHistory
> Running org.apache.hive.beeline.TestBeeLineOpts
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.338 sec -
> in org.apache.hive.beeline.TestBeeLineOpts
> Running org.apache.hive.beeline.TestBufferedRows
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.382 sec -
> in org.apache.hive.beeline.TestBufferedRows
> Running org.apache.hive.beeline.TestClientCommandHookFactory
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.546 sec -
> in org.apache.hive.beeline.TestClientCommandHookFactory
> Running org.apache.hive.beeline.TestIncrementalRowsWithNormalization
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.379 sec -
> in org.apache.hive.beeline.TestIncrementalRowsWithNormalization
> Running org.apache.hive.beeline.TestTableOutputFormat
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.402 sec -
> in org.apache.hive.beeline.TestTableOutputFormat
> Results :
> Tests run: 44, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 01:03 min
> [INFO] Finished at: 2016-09-07T10:57:28-07:00
> [INFO] Final Memory: 65M/949M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on
> project hive-beeline: ExecutionException The forked VM terminated without
> properly saying goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd
> /Users/vihang/work/src/upstream/hive/beeline &&
> /Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/jre/bin/java
> -Xmx2048m -XX:MaxPermSize=512m -jar
> /Users/vihang/work/src/upstream/hive/beeline/target/surefire/surefirebooter8133887423099901737.jar
>
> /Users/vihang/work/src/upstream/hive/beeline/target/surefire/surefire6528142441948588259tmp
>
> /Users/vihang/work/src/upstream/hive/beeline/target/surefire/surefire_05570572112194455658tmp
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> {noformat}
> My guess is this is related to recent fix for HIVE-13589 but needs more
> investigation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14726) delete statement fails when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478607#comment-15478607 ] Alan Gates commented on HIVE-14726: --- I don't know what's causing dynpart_sort_optimization_acid.q to fail, but I don't think it's the same thing since we don't see the IndexOutOfBounds exception. > delete statement fails when spdo is on > -- > > Key: HIVE-14726 > URL: https://issues.apache.org/jira/browse/HIVE-14726 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Deepesh Khandelwal >Assignee: Ashutosh Chauhan > Attachments: HIVE-14726.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14726) delete statement fails when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478501#comment-15478501 ] Eugene Koifman commented on HIVE-14726: --- Does Jenkins print Commit hash for each run? This test passed in run 1122 but failed in 1123. cc [~sershe] > delete statement fails when spdo is on > -- > > Key: HIVE-14726 > URL: https://issues.apache.org/jira/browse/HIVE-14726 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Deepesh Khandelwal >Assignee: Ashutosh Chauhan > Attachments: HIVE-14726.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14729) analyze table compute stats for columns should collect basic stats as well
[
https://issues.apache.org/jira/browse/HIVE-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478495#comment-15478495
]
Pengcheng Xiong commented on HIVE-14729:
turn auto column stats collection flag on :)
> analyze table compute stats for columns should collect basic stats as well
> --
>
> Key: HIVE-14729
> URL: https://issues.apache.org/jira/browse/HIVE-14729
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Ashutosh Chauhan
>
> Currently user need to run following two statements to collect stats:
> {code}
> analyze table T compute statistics;
> analyze table T compute statistics for columns;
> {code}
> Its little confusing for users that they need to collect both stats. Since
> basic stats are cheap to compute, we should collect them even in case of
> column stats, so that by running single statement users can get all stats.
> This leaves with the possibility of fast basic stats collection with first
> statement which will be good to have.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14726) delete statement fails when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478489#comment-15478489 ] Eugene Koifman commented on HIVE-14726: --- as far as I can tell it started failing here https://builds.apache.org/view/H-L/view/Hive/job/PreCommit-HIVE-MASTER-Build/1123/ which was for HIVE-14710, though I don't know how that could cause it > delete statement fails when spdo is on > -- > > Key: HIVE-14726 > URL: https://issues.apache.org/jira/browse/HIVE-14726 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Deepesh Khandelwal >Assignee: Ashutosh Chauhan > Attachments: HIVE-14726.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14011) MessageFactory is not pluggable
[
https://issues.apache.org/jira/browse/HIVE-14011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478477#comment-15478477
]
Hive QA commented on HIVE-14011:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12810694/HIVE-14011.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10546 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1142/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1142/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1142/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12810694 - PreCommit-HIVE-MASTER-Build
> MessageFactory is not pluggable
> ---
>
> Key: HIVE-14011
> URL: https://issues.apache.org/jira/browse/HIVE-14011
> Project: Hive
> Issue Type: Bug
>Reporter: Sravya Tirukkovalur
>Assignee: Sravya Tirukkovalur
> Attachments: HIVE-14011.patch
>
>
> Property "hcatalog.message.factory.impl.json" is available to use a custom
> message factory implementation. Although it is not pluggable as
> MessageFatcory is hardcoded to use JSONMessageFactory.
> https://github.com/apache/hive/blob/26b5c7b56a4f28ce3eabc0207566cce46b29b558/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/MessageFactory.java#L39
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14726) delete statement fails when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478455#comment-15478455 ] Alan Gates commented on HIVE-14726: --- There's no info on which query the stack trace came from. [~deepesh], do you have the query and any information on the table? Agreed that we don't want to blindly turn spdo off here. > delete statement fails when spdo is on > -- > > Key: HIVE-14726 > URL: https://issues.apache.org/jira/browse/HIVE-14726 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Deepesh Khandelwal >Assignee: Ashutosh Chauhan > Attachments: HIVE-14726.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14726) delete statement fails when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478456#comment-15478456 ] Eugene Koifman commented on HIVE-14726: --- I think hive.optimize.sort.dynamic.partition was made to work with ACID update/delete in HIVE-13849 and HIVE-13750 > delete statement fails when spdo is on > -- > > Key: HIVE-14726 > URL: https://issues.apache.org/jira/browse/HIVE-14726 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Deepesh Khandelwal >Assignee: Ashutosh Chauhan > Attachments: HIVE-14726.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14726) delete statement fails when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478442#comment-15478442 ] Eugene Koifman edited comment on HIVE-14726 at 9/9/16 10:23 PM: Update/Delete can certainly write many partitions at the same time, e.g. Update T set c1=c1+1 where c2 > 100. This may match 1 row in each partition and thus the Insert statement that is actually executed will write to each partition. Same for Delete. Since what is actually executed is an Insert statement, why is Update/Delete different from any other Insert? was (Author: ekoifman): Update/Delete can certainly write many partitions at the same time, e.g. Update T set c1=c1+1 where c2 > 100. This may match 1 row in each partition and thus the Insert statement that is actually executed will write to each partition. Same for Delete. > delete statement fails when spdo is on > -- > > Key: HIVE-14726 > URL: https://issues.apache.org/jira/browse/HIVE-14726 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Deepesh Khandelwal >Assignee: Ashutosh Chauhan > Attachments: HIVE-14726.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14578) Fixes to TestEncryptedHDFSCliDriver
[ https://issues.apache.org/jira/browse/HIVE-14578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-14578: - Status: Patch Available (was: Open) > Fixes to TestEncryptedHDFSCliDriver > --- > > Key: HIVE-14578 > URL: https://issues.apache.org/jira/browse/HIVE-14578 > Project: Hive > Issue Type: Sub-task >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-14578.1.patch > > > 900s runtime. One of the stragglers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14578) Fixes to TestEncryptedHDFSCliDriver
[ https://issues.apache.org/jira/browse/HIVE-14578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-14578: - Attachment: HIVE-14578.1.patch > Fixes to TestEncryptedHDFSCliDriver > --- > > Key: HIVE-14578 > URL: https://issues.apache.org/jira/browse/HIVE-14578 > Project: Hive > Issue Type: Sub-task >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-14578.1.patch > > > 900s runtime. One of the stragglers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13770) HiveServer2: Improve Thrift result set streaming when serializing thrift ResultSets in tasks
[ https://issues.apache.org/jira/browse/HIVE-13770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13770: Assignee: Ziyang Zhao > HiveServer2: Improve Thrift result set streaming when serializing thrift > ResultSets in tasks > > > Key: HIVE-13770 > URL: https://issues.apache.org/jira/browse/HIVE-13770 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Holman Lan >Assignee: Ziyang Zhao > > When serializing the Thrift result set in final task, i.e. the > hive.server2.thrift.resultset.serialize.in.tasks property is set to true, HS2 > does not start sending the results until the entire result set has been > written to HDFS. > This is not efficient and we should find a way for HS2 to start sending the > results as soon as a block of result becomes available. The advantage for > this is two folds. One, the client can start consuming the results much > sooner. Two, we can start reclaiming the storage space in HDFS used by a > particular result set block as soon as the result set block has been > successfully sent to the client. > It's worth checking if this is also the case when not serializing the Thrift > result set in final task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14726) delete statement fails when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478321#comment-15478321 ] Ashutosh Chauhan commented on HIVE-14726: - On taking a second look it seems that delete/update statements can also be affected by wrath of orc buffers. e.g., for the query in test case RS sort and shuffle key is ROW__ID and partition column is in value of RS. If I understand correctly in Reducers FileSink operator need to have an orc buffer open for each partition. This may result in similar OOM cases. However, I am not sure if its possible to solve that problem through this optimization for delete/update statement. [~ekoifman] [~prasanth_j] any suggestions? > delete statement fails when spdo is on > -- > > Key: HIVE-14726 > URL: https://issues.apache.org/jira/browse/HIVE-14726 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Deepesh Khandelwal >Assignee: Ashutosh Chauhan > Attachments: HIVE-14726.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14715) Hive throws NumberFormatException with query with Null value
[ https://issues.apache.org/jira/browse/HIVE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478278#comment-15478278 ] Ashutosh Chauhan commented on HIVE-14715: - I see. Thanks for explanation. > Hive throws NumberFormatException with query with Null value > > > Key: HIVE-14715 > URL: https://issues.apache.org/jira/browse/HIVE-14715 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-14715.1.patch, HIVE-14715.2.patch > > > The java.lang.NumberFormatException will throw with following reproduce: > set hive.cbo.enable=false; > CREATE TABLE `paqtest`( > `c1` int, > `s1` string, > `s2` string, > `bn1` bigint) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert into paqtest values (58, '', 'ABC', 0); > SELECT > 'PM' AS cy, > c1, > NULL AS iused, > NULL AS itp, > s2, > NULL AS cvg, > NULL AS acavg, > sum(bn1) AS cca > FROM paqtest > WHERE (s1 IS NULL OR length(s1) = 0) > GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL; > The stack like following: > java.lang.NumberFormatException: ABC > GroupByOperator.process(Object, int) line: 773 > ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236 > ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, > RawKeyValueIterator, RawComparator, Class, Class) > line: 444 > ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392 > LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319 > Executors$RunnableAdapter.call() line: 471 > It works fine when hive.cbo.enable = true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12540) Create function failed, but show functions display it
[
https://issues.apache.org/jira/browse/HIVE-12540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478271#comment-15478271
]
Naveen Gangam commented on HIVE-12540:
--
[~Sephiroth-Lin] I have tried this a few times in a few different ways. I
thought I had this reproduced the first time but before I could take a deeper
look I got diverted. After a refresh and a rebuild, I can no longer reproduce
this. Perhaps I was using older code the first time but this does not appear to
be a problem in the current code.
Could you please retry and let me know if you can still reproduce this? If so,
can you please post the git hash of the commit you are running at? Thanks
> Create function failed, but show functions display it
> -
>
> Key: HIVE-12540
> URL: https://issues.apache.org/jira/browse/HIVE-12540
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Weizhong
>Assignee: Naveen Gangam
>Priority: Minor
>
> {noformat}
> 0: jdbc:hive2://vm119:1> create function udfTest as
> 'hive.udf.UDFArrayNotE';
> ERROR : Failed to register default.udftest using class hive.udf.UDFArrayNotE
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.FunctionTask (state=08S01,code=1)
> 0: jdbc:hive2://vm119:1> show functions;
> +-+--+
> |tab_name |
> +-+--+
> | ... |
> | default.udftest |
> | ... |
> +-+--+
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14715) Hive throws NumberFormatException with query with Null value
[
https://issues.apache.org/jira/browse/HIVE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478254#comment-15478254
]
Yongzhi Chen commented on HIVE-14715:
-
[~ashutoshc],
In cbo mode, the reduce output operator treat each null as different cols
(known from the query plan), so there is no column removed in reduce keys,
therefore it works. Following is the plan(partial) from cbo:
{noformat}
Group By Operator
aggregations: sum(_col6)
keys: _col0 (type: int), null (type: void), null (type:
void), _col3 (type: string), null (type: void), null (type: void)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6
Statistics: Num rows: 4 Data size: 1019 Basic stats:
COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int), _col1 (type: void),
_col2 (type: void), _col3 (type: string), _col4 (type: void), _col5 (type:
void)
sort order: ++
Map-reduce partition columns: _col0 (type: int), _col1
(type: void), _col2 (type: void), _col3 (type: string), _col4 (type: void),
_col5 (type: void)
Statistics: Num rows: 4 Data size: 1019 Basic stats:
COMPLETE Column stats: NONE
value expressions: _col6 (type: bigint)
{noformat}
And following is the corresponding plan without fix for non-cbo mode:
{noformat}
Group By Operator
aggregations: sum(bn1)
keys: 'Pricing mismatch' (type: string), c1 (type: int),
null (type: void), null (type: void), s2 (type: string), null (type: void),
null (type: void)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col7
Statistics: Num rows: 4 Data size: 1019 Basic stats:
COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: 'Pricing mismatch' (type: string), _col1
(type: int), null (type: void), _col4 (type: string)
sort order:
Map-reduce partition columns: 'Pricing mismatch' (type:
string), _col1 (type: int), null (type: void), _col4 (type: string)
Statistics: Num rows: 4 Data size: 1019 Basic stats:
COMPLETE Column stats: NONE
value expressions: _col4 (type: string)
{noformat}
You can see _col4 is wrong, it should be _col7.
> Hive throws NumberFormatException with query with Null value
>
>
> Key: HIVE-14715
> URL: https://issues.apache.org/jira/browse/HIVE-14715
> Project: Hive
> Issue Type: Bug
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-14715.1.patch, HIVE-14715.2.patch
>
>
> The java.lang.NumberFormatException will throw with following reproduce:
> set hive.cbo.enable=false;
> CREATE TABLE `paqtest`(
> `c1` int,
> `s1` string,
> `s2` string,
> `bn1` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat';
> insert into paqtest values (58, '', 'ABC', 0);
> SELECT
> 'PM' AS cy,
> c1,
> NULL AS iused,
> NULL AS itp,
> s2,
> NULL AS cvg,
> NULL AS acavg,
> sum(bn1) AS cca
> FROM paqtest
> WHERE (s1 IS NULL OR length(s1) = 0)
> GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL;
> The stack like following:
> java.lang.NumberFormatException: ABC
> GroupByOperator.process(Object, int) line: 773
> ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236
> ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter,
> RawKeyValueIterator, RawComparator, Class, Class)
> line: 444
> ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392
> LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319
> Executors$RunnableAdapter.call() line: 471
> It works fine when hive.cbo.enable = true
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13770) HiveServer2: Improve Thrift result set streaming when serializing thrift ResultSets in tasks
[ https://issues.apache.org/jira/browse/HIVE-13770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13770: Component/s: JDBC HiveServer2 > HiveServer2: Improve Thrift result set streaming when serializing thrift > ResultSets in tasks > > > Key: HIVE-13770 > URL: https://issues.apache.org/jira/browse/HIVE-13770 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Holman Lan > > When serializing the Thrift result set in final task, i.e. the > hive.server2.thrift.resultset.serialize.in.tasks property is set to true, HS2 > does not start sending the results until the entire result set has been > written to HDFS. > This is not efficient and we should find a way for HS2 to start sending the > results as soon as a block of result becomes available. The advantage for > this is two folds. One, the client can start consuming the results much > sooner. Two, we can start reclaiming the storage space in HDFS used by a > particular result set block as soon as the result set block has been > successfully sent to the client. > It's worth checking if this is also the case when not serializing the Thrift > result set in final task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13593) HiveServer2: Performance instrumentation for HIVE-12049 (serializing thrift ResultSets in tasks)
[ https://issues.apache.org/jira/browse/HIVE-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta resolved HIVE-13593. - Resolution: Fixed Preliminary results attached. Resolving this task. Thanks [~ziyangz]. > HiveServer2: Performance instrumentation for HIVE-12049 (serializing thrift > ResultSets in tasks) > > > Key: HIVE-13593 > URL: https://issues.apache.org/jira/browse/HIVE-13593 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Ziyang Zhao > Attachments: HiveServer2 With Option to Write Serialized Thrift > Objects in Final Tasks Performance Test Results.pdf, raw_data.zip > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13593) HiveServer2: Performance instrumentation for HIVE-12049 (serializing thrift ResultSets in tasks)
[ https://issues.apache.org/jira/browse/HIVE-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13593: Fix Version/s: 2.1.1 2.2.0 > HiveServer2: Performance instrumentation for HIVE-12049 (serializing thrift > ResultSets in tasks) > > > Key: HIVE-13593 > URL: https://issues.apache.org/jira/browse/HIVE-13593 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Ziyang Zhao > Fix For: 2.2.0, 2.1.1 > > Attachments: HiveServer2 With Option to Write Serialized Thrift > Objects in Final Tasks Performance Test Results.pdf, raw_data.zip > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14558) Add support for listing views similar to "show tables"
[ https://issues.apache.org/jira/browse/HIVE-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-14558: - Attachment: HIVE-14558.1.patch > Add support for listing views similar to "show tables" > -- > > Key: HIVE-14558 > URL: https://issues.apache.org/jira/browse/HIVE-14558 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14558.1.patch > > > Users have been asking for such feature where they can get a lists of views > separately. > So perhaps a syntax similar to "show tables" command? > show views [in/from ] [] > Does it make sense to add such command? or is it not worth the effort? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14558) Add support for listing views similar to "show tables"
[ https://issues.apache.org/jira/browse/HIVE-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-14558: - Status: Patch Available (was: Open) > Add support for listing views similar to "show tables" > -- > > Key: HIVE-14558 > URL: https://issues.apache.org/jira/browse/HIVE-14558 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14558.1.patch > > > Users have been asking for such feature where they can get a lists of views > separately. > So perhaps a syntax similar to "show tables" command? > show views [in/from ] [] > Does it make sense to add such command? or is it not worth the effort? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14558) Add support for listing views similar to "show tables"
[
https://issues.apache.org/jira/browse/HIVE-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478146#comment-15478146
]
Naveen Gangam commented on HIVE-14558:
--
Based on some quick searches, appears most databases have some form of support
for getting a list of views. I think it makes sense to support a "show views"
command in hive, with a syntax similar to "show tables". I am uploading a
patch that will support commands like
{code}
show views; // list all views in the current database
show views 'test_*'; // list all views that start with "test_"
show views '*view'; // list all views that end in "view"
show views in ; // list all views in database
show views from ; // same as IN
show views in/from [LIKE] 'test_*'; // show views in database that
start with "test_"
{code}
The syntax for this command is derived from SHOW TABLES command.
General syntax is
SHOW VIEWS [IN/FROM DBNAME] [LIKE] [PATTERN]
> Add support for listing views similar to "show tables"
> --
>
> Key: HIVE-14558
> URL: https://issues.apache.org/jira/browse/HIVE-14558
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>
> Users have been asking for such feature where they can get a lists of views
> separately.
> So perhaps a syntax similar to "show tables" command?
> show views [in/from ] []
> Does it make sense to add such command? or is it not worth the effort?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13593) HiveServer2: Performance instrumentation for HIVE-12049 (serializing thrift ResultSets in tasks)
[ https://issues.apache.org/jira/browse/HIVE-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziyang Zhao updated HIVE-13593: --- Attachment: raw_data.zip > HiveServer2: Performance instrumentation for HIVE-12049 (serializing thrift > ResultSets in tasks) > > > Key: HIVE-13593 > URL: https://issues.apache.org/jira/browse/HIVE-13593 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Ziyang Zhao > Attachments: HiveServer2 With Option to Write Serialized Thrift > Objects in Final Tasks Performance Test Results.pdf, raw_data.zip > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13593) HiveServer2: Performance instrumentation for HIVE-12049 (serializing thrift ResultSets in tasks)
[ https://issues.apache.org/jira/browse/HIVE-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziyang Zhao updated HIVE-13593: --- Attachment: HiveServer2 With Option to Write Serialized Thrift Objects in Final Tasks Performance Test Results.pdf > HiveServer2: Performance instrumentation for HIVE-12049 (serializing thrift > ResultSets in tasks) > > > Key: HIVE-13593 > URL: https://issues.apache.org/jira/browse/HIVE-13593 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Ziyang Zhao > Attachments: HiveServer2 With Option to Write Serialized Thrift > Objects in Final Tasks Performance Test Results.pdf > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13593) HiveServer2: Performance instrumentation for HIVE-12049 (serializing thrift ResultSets in tasks)
[ https://issues.apache.org/jira/browse/HIVE-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziyang Zhao reassigned HIVE-13593: -- Assignee: Ziyang Zhao (was: Holman Lan) > HiveServer2: Performance instrumentation for HIVE-12049 (serializing thrift > ResultSets in tasks) > > > Key: HIVE-13593 > URL: https://issues.apache.org/jira/browse/HIVE-13593 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Ziyang Zhao > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14011) MessageFactory is not pluggable
[ https://issues.apache.org/jira/browse/HIVE-14011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sravya Tirukkovalur updated HIVE-14011: --- Status: Patch Available (was: Open) > MessageFactory is not pluggable > --- > > Key: HIVE-14011 > URL: https://issues.apache.org/jira/browse/HIVE-14011 > Project: Hive > Issue Type: Bug >Reporter: Sravya Tirukkovalur >Assignee: Sravya Tirukkovalur > Attachments: HIVE-14011.patch > > > Property "hcatalog.message.factory.impl.json" is available to use a custom > message factory implementation. Although it is not pluggable as > MessageFatcory is hardcoded to use JSONMessageFactory. > https://github.com/apache/hive/blob/26b5c7b56a4f28ce3eabc0207566cce46b29b558/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/MessageFactory.java#L39 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13593) HiveServer2: Performance instrumentation for HIVE-12049 (serializing thrift ResultSets in tasks)
[ https://issues.apache.org/jira/browse/HIVE-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13593: Summary: HiveServer2: Performance instrumentation for HIVE-12049 (serializing thrift ResultSets in tasks) (was: HiveServer2: Better performance instrumentation for HIVE-12049 (serializing thrift ResultSets in tasks)) > HiveServer2: Performance instrumentation for HIVE-12049 (serializing thrift > ResultSets in tasks) > > > Key: HIVE-13593 > URL: https://issues.apache.org/jira/browse/HIVE-13593 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Holman Lan > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13830) Hive on spark driver crash with Spark 1.6.1
[
https://issues.apache.org/jira/browse/HIVE-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Liew updated HIVE-13830:
--
Description:
With Hive 1.2.1 I was able to use Hive on successfully with the use of the
-assembly "-assembly-1.4.1-hadoop2.7.1.jar".
Today with Hive 2.0.0, I'm unable to use Hive on whether it be with the
-assembly "-assembly-1.4.1-hadoop2.7.1.jar" or the -assembly
"-assembly-1.6.1-hadoop2.7.2.jar".
My configuration is the following:
* -. available in HIVE_DIR/
* assembly available in HIVE_DIR/lib
I gathered several logs below:
- HQL commands
{noformat}
$ hive -v --database shfs3453
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/application/Hive/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/application/Hive/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/application//-1.6.1/assembly/target/scala-2.10/-assembly-1.6.1-hadoop2.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in
file:/opt/application/Hive/apache-hive-2.0.0-bin/conf/hive-log4j2.properties
use shfs3453
OK
Time taken: 1.425 seconds
Hive-on-MR is deprecated in Hive 2 and may not be available in the future
versions. Consider using a different execution engine (i.e. tez, ) or using
Hive 1.X releases.
hive (shfs3453)> set hive.execution.engine=;
set hive.execution.engine=
hive (shfs3453)> set .master=yarn-client;
set .master=yarn-client
hive (shfs3453)> CREATE TABLE chicagoCrimes2 (ID BIGINT, CaseNumber STRING, Day
STRING, Block STRING, IUCR INT, PrimaryType STRING, Description STRING,
LocationDescription STRING, Arrest BOOLEAN, Domestic BOOLEAN, Beat INT,
District INT, Ward INT, CommunityArea INT, FBICode INT, XCoordinate BIGINT,
YCoordinate BIGINT, Year INT, UpdatedOn STRING, Latitude FLOAT, Longitude
FLOAT, Location STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS
TEXTFILE;
CREATE TABLE chicagoCrimes2 (ID BIGINT, CaseNumber STRING, Day STRING, Block
STRING, IUCR INT, PrimaryType STRING, Description STRING, LocationDescription
STRING, Arrest BOOLEAN, Domestic BOOLEAN, Beat INT, District INT, Ward INT,
CommunityArea INT, FBICode INT, XCoordinate BIGINT, YCoordinate BIGINT, Year
INT, UpdatedOn STRING, Latitude FLOAT, Longitude FLOAT, Location STRING) ROW
FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE
OK
Time taken: 0.408 seconds
hive (shfs3453)> INSERT OVERWRITE TABLE chicagocrimes2 SELECT * FROM
chicagocrimes WHERE Description = 'FIRST DEGREE MURDER';
INSERT OVERWRITE TABLE chicagocrimes2 SELECT * FROM chicagocrimes WHERE
Description = 'FIRST DEGREE MURDER'
Query ID = shfs3453_20160524092714_41c89aec-2c6f-49e9-98c7-d227ca144f73
Total jobs = 1
Launching Job 1 out of 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapreduce.job.reduces=
Starting Job = 79484279-8e75-4b13-8e71-7de463f4d51e
Status: SENT
Failed to execute task, with exception 'java.lang.IllegalStateException(RPC
channel is closed.)'
FAILED: Execution , return code 1 from org.apache.hadoop.hive.ql.exec..SparkTask
{noformat}
- Client logs
{noformat}
May 24 09:32:19 hive-cli - org.apache.hive..client.rpc.RpcDispatcherReceived
message:io.netty.handler.codec.DecoderException:
java.lang.NoClassDefFoundError: org/apache/hive//client/Job
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:358)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:230)
at
io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at
[jira] [Commented] (HIVE-14727) llap-server may case file descriptor leak in BuddyAllocator class
[ https://issues.apache.org/jira/browse/HIVE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477875#comment-15477875 ] Sergey Shelukhin commented on HIVE-14727: - cc [~gopalv] +1 from my side. What was the original failure? > llap-server may case file descriptor leak in BuddyAllocator class > - > > Key: HIVE-14727 > URL: https://issues.apache.org/jira/browse/HIVE-14727 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Yechao Chen >Assignee: Yechao Chen > Attachments: HIVE-14727.patch > > > llap-server,the method preallocate(int) of BuddyAllocator may case file > descriptor leak when FileChannel map allocate memory error. > the code: > //here if failed > ByteBuffer rwbuf = rwf.getChannel().map(MapMode.READ_WRITE, 0, > arenaSize); > // A mapping, once established, is not dependent upon the file > channel that was used to > // create it. delete file and hold onto the map >//can not close() and delete file > rwf.close(); > rf.delete(); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14719) ASTNode rootNode is not maintained properly when changing child/parent relation
[
https://issues.apache.org/jira/browse/HIVE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477660#comment-15477660
]
Ashutosh Chauhan commented on HIVE-14719:
-
Thanks [~lirui] for detailed explanation. Your patch is in right direction. If
we remove a node and use it elsewhere then we should clear its root node before
removing it. Can you also add a test case in your patch?
> ASTNode rootNode is not maintained properly when changing child/parent
> relation
> ---
>
> Key: HIVE-14719
> URL: https://issues.apache.org/jira/browse/HIVE-14719
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14719.1.patch
>
>
> When I run some query like:
> {code}
> set hive.cbo.enable=false;
> select * from A where exists (select * from B where B.k1=A.k1 and B.k2=A.k2);
> {code}
> It gets error like:
> {noformat}
> FAILED: SemanticException Line 0:-1 Invalid table alias or column reference
> 'sq_1': (possible column names are: _table_or_col b) k2) sq_corr_1)) (tok, (.
> (tok_table_or_col sq_1) sq_corr_1))
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14715) Hive throws NumberFormatException with query with Null value
[ https://issues.apache.org/jira/browse/HIVE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477436#comment-15477436 ] Ashutosh Chauhan commented on HIVE-14715: - Do you happen to know why this doesnt show up when cbo is enabled, seems like it should happen even in that case. > Hive throws NumberFormatException with query with Null value > > > Key: HIVE-14715 > URL: https://issues.apache.org/jira/browse/HIVE-14715 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-14715.1.patch, HIVE-14715.2.patch > > > The java.lang.NumberFormatException will throw with following reproduce: > set hive.cbo.enable=false; > CREATE TABLE `paqtest`( > `c1` int, > `s1` string, > `s2` string, > `bn1` bigint) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert into paqtest values (58, '', 'ABC', 0); > SELECT > 'PM' AS cy, > c1, > NULL AS iused, > NULL AS itp, > s2, > NULL AS cvg, > NULL AS acavg, > sum(bn1) AS cca > FROM paqtest > WHERE (s1 IS NULL OR length(s1) = 0) > GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL; > The stack like following: > java.lang.NumberFormatException: ABC > GroupByOperator.process(Object, int) line: 773 > ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236 > ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, > RawKeyValueIterator, RawComparator, Class, Class) > line: 444 > ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392 > LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319 > Executors$RunnableAdapter.call() line: 471 > It works fine when hive.cbo.enable = true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14726) delete statement fails when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477432#comment-15477432 ] Ashutosh Chauhan commented on HIVE-14726: - Reason where this optimization is useful when there are large number of partitions being written. That time there are many orc writers which are buffering data. Update and delete statement gets rewritten as insert statement but they won't have this issue of many dynamic partitions being written into at same time. So, I am not sure why there is a handling of Update/Delete at place you pointed out, we should simply skip this optimization step in those cases. I will take another look and update the patch accordingly. > delete statement fails when spdo is on > -- > > Key: HIVE-14726 > URL: https://issues.apache.org/jira/browse/HIVE-14726 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Deepesh Khandelwal >Assignee: Ashutosh Chauhan > Attachments: HIVE-14726.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14715) Hive throws NumberFormatException with query with Null value
[ https://issues.apache.org/jira/browse/HIVE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477383#comment-15477383 ] Aihua Xu commented on HIVE-14715: - The change looks good to me. +1. > Hive throws NumberFormatException with query with Null value > > > Key: HIVE-14715 > URL: https://issues.apache.org/jira/browse/HIVE-14715 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-14715.1.patch, HIVE-14715.2.patch > > > The java.lang.NumberFormatException will throw with following reproduce: > set hive.cbo.enable=false; > CREATE TABLE `paqtest`( > `c1` int, > `s1` string, > `s2` string, > `bn1` bigint) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert into paqtest values (58, '', 'ABC', 0); > SELECT > 'PM' AS cy, > c1, > NULL AS iused, > NULL AS itp, > s2, > NULL AS cvg, > NULL AS acavg, > sum(bn1) AS cca > FROM paqtest > WHERE (s1 IS NULL OR length(s1) = 0) > GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL; > The stack like following: > java.lang.NumberFormatException: ABC > GroupByOperator.process(Object, int) line: 773 > ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236 > ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, > RawKeyValueIterator, RawComparator, Class, Class) > line: 444 > ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392 > LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319 > Executors$RunnableAdapter.call() line: 471 > It works fine when hive.cbo.enable = true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14726) delete statement fails when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477367#comment-15477367 ] Jesus Camacho Rodriguez commented on HIVE-14726: [~ashutoshc], could you elaborate why there is no advantage of running the optimization for update and delete? It seems we used to optimize these cases (l187 in the original SortedDynPartitionOptimizer class). If we apply the patch attached, we should remove the code in l187. > delete statement fails when spdo is on > -- > > Key: HIVE-14726 > URL: https://issues.apache.org/jira/browse/HIVE-14726 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Deepesh Khandelwal >Assignee: Ashutosh Chauhan > Attachments: HIVE-14726.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14726) delete statement fails when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477342#comment-15477342 ] Ashutosh Chauhan commented on HIVE-14726: - [~jcamachorodriguez] Can you take a quick look? Seemingly related dynpart_sort_optimization_acid.q is failing on master since last 15 builds. > delete statement fails when spdo is on > -- > > Key: HIVE-14726 > URL: https://issues.apache.org/jira/browse/HIVE-14726 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Deepesh Khandelwal >Assignee: Ashutosh Chauhan > Attachments: HIVE-14726.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp
[
https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477327#comment-15477327
]
Hive QA commented on HIVE-14412:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827735/HIVE-14412.4.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10524 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1141/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1141/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1141/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12827735 - PreCommit-HIVE-MASTER-Build
> Add a timezone-aware timestamp
> --
>
> Key: HIVE-14412
> URL: https://issues.apache.org/jira/browse/HIVE-14412
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14412.1.patch, HIVE-14412.2.patch,
> HIVE-14412.3.patch, HIVE-14412.4.patch
>
>
> Java's Timestamp stores the time elapsed since the epoch. While it's by
> itself unambiguous, ambiguity comes when we parse a string into timestamp, or
> convert a timestamp to string, causing problems like HIVE-14305.
> To solve the issue, I think we should make timestamp aware of timezone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14626) Support Trash in Truncate Table
[ https://issues.apache.org/jira/browse/HIVE-14626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477287#comment-15477287 ] Chaoyu Tang commented on HIVE-14626: [~leftylev] I updated the wiki to reflect this change. As for encryption, it is about HDFS encryption zone and you can find the details in HIVE-8065. > Support Trash in Truncate Table > --- > > Key: HIVE-14626 > URL: https://issues.apache.org/jira/browse/HIVE-14626 > Project: Hive > Issue Type: Sub-task > Components: Query Processor >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14626.1.patch, HIVE-14626.patch > > > Currently Truncate Table (or Partition) is implemented using > FileSystem.delete and then recreate the directory, so > 1. it does not support HDFS Trash > 2. if the table/partition directory is initially encryption protected, after > being deleted and recreated, it is no more protected. > The new implementation is to clean the contents of directory using > multi-threaded trashFiles. If Trash is enabled and has a lower encryption > level than the data directory, the files under it will be deleted. Otherwise, > they will be Trashed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477265#comment-15477265
]
Aihua Xu edited comment on HIVE-13160 at 9/9/16 3:03 PM:
-
To me, it may not be reasonable to keep the backward compatibility, such as HS2
2.1 to HMS1.2.1 since HMS1.2.1 may miss some APIs which HS2 2.1.0 are using.
Even if we let HS2 start, we still can't use UDFs e.g.
So I think you need to upgrade both HMS and HS2.
was (Author: aihuaxu):
To me, it may not be reasonable to keep the backward compatibility, such as HS2
2.1 to HMS1.2.1 since HMS1.2.1 may miss some APIs which HS2 2.1.0 are using.
Even if we let HS2 start, we still can't use UDFs e.g.
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477265#comment-15477265
]
Aihua Xu commented on HIVE-13160:
-
To me, it may not be reasonable to keep the backward compatibility, such as HS2
2.1 to HMS1.2.1 since HMS1.2.1 may miss some APIs which HS2 2.1.0 are using.
Even if we let HS2 start, we still can't use UDFs e.g.
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from
[jira] [Commented] (HIVE-14098) Logging task properties, and environment variables might contain passwords
[ https://issues.apache.org/jira/browse/HIVE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477261#comment-15477261 ] Peter Vary commented on HIVE-14098: --- To tell the truth, not sure about this result. There is no previous result to compare to, but since the patch on the master branch did not created these errors, I think these failures are not introduced by this patch. Most of it is query error, some of them are easy to explain (ordering of the hashmap toString), some of them are easy to understand that not introduced by this patch - different execution plans. [~spena], [~ctang.ma] what should be my next steps? What I could do is, create a patch with a minimal modification, which changes nothing, and submit as a patch. This way we could see, what are the already existing errors on the Hive 2.1 branch. So we could validate that this patch does not make things worst. What do you think? Thanks, Peter > Logging task properties, and environment variables might contain passwords > -- > > Key: HIVE-14098 > URL: https://issues.apache.org/jira/browse/HIVE-14098 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Logging, Spark >Affects Versions: 2.1.0 >Reporter: Peter Vary >Assignee: Peter Vary > Fix For: 2.2.0 > > Attachments: HIVE-14098-branch-2.1.patch, HIVE-14098.2.patch, > HIVE-14098.patch > > > Hive MapredLocalTask Can Print Environment Passwords, like > -Djavax.net.ssl.trustStorePassword. > The same could happen, when logging spark properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14098) Logging task properties, and environment variables might contain passwords
[
https://issues.apache.org/jira/browse/HIVE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477133#comment-15477133
]
Hive QA commented on HIVE-14098:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827585/HIVE-14098-branch-2.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 254 failed/errored test(s), 10353 tests
executed
*Failed tests:*
{noformat}
TestDruidSerDe - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestHiveDruidQueryBasedInputFormat - did not produce a TEST-*.xml file
TestHs2HooksWithMiniKdc - did not produce a TEST-*.xml file
TestJdbcNonKrbSASLWithMiniKdc - did not produce a TEST-*.xml file
TestJdbcWithDBTokenStore - did not produce a TEST-*.xml file
TestJdbcWithMiniKdc - did not produce a TEST-*.xml file
TestJdbcWithMiniKdcCookie - did not produce a TEST-*.xml file
TestJdbcWithMiniKdcSQLAuthBinary - did not produce a TEST-*.xml file
TestJdbcWithMiniKdcSQLAuthHttp - did not produce a TEST-*.xml file
TestMsgBusConnection - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binary_output_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_outer_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_udf1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnStatsUpdateForStatsOptimizer_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnStatsUpdateForStatsOptimizer_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_describe_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_full
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial_ndv
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fouter_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input42
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join35
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_json_serde1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
[jira] [Commented] (HIVE-14727) llap-server may case file descriptor leak in BuddyAllocator class
[
https://issues.apache.org/jira/browse/HIVE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476892#comment-15476892
]
Hive QA commented on HIVE-14727:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827725/HIVE-14727.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10546 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1140/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1140/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1140/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12827725 - PreCommit-HIVE-MASTER-Build
> llap-server may case file descriptor leak in BuddyAllocator class
> -
>
> Key: HIVE-14727
> URL: https://issues.apache.org/jira/browse/HIVE-14727
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Yechao Chen
>Assignee: Yechao Chen
> Attachments: HIVE-14727.patch
>
>
> llap-server,the method preallocate(int) of BuddyAllocator may case file
> descriptor leak when FileChannel map allocate memory error.
> the code:
> //here if failed
> ByteBuffer rwbuf = rwf.getChannel().map(MapMode.READ_WRITE, 0,
> arenaSize);
> // A mapping, once established, is not dependent upon the file
> channel that was used to
> // create it. delete file and hold onto the map
>//can not close() and delete file
> rwf.close();
> rf.delete();
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14412) Add a timezone-aware timestamp
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14412: -- Attachment: HIVE-14412.4.patch Update some golden files. > Add a timezone-aware timestamp > -- > > Key: HIVE-14412 > URL: https://issues.apache.org/jira/browse/HIVE-14412 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-14412.1.patch, HIVE-14412.2.patch, > HIVE-14412.3.patch, HIVE-14412.4.patch > > > Java's Timestamp stores the time elapsed since the epoch. While it's by > itself unambiguous, ambiguity comes when we parse a string into timestamp, or > convert a timestamp to string, causing problems like HIVE-14305. > To solve the issue, I think we should make timestamp aware of timezone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-14686:
--
Affects Version/s: (was: 1.1.0)
1.2.1
2.1.0
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch, HIVE-14686.5.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-14686:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to master. Thanks Yunbo for your contribution.
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch, HIVE-14686.5.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476757#comment-15476757
]
Hive QA commented on HIVE-14686:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827726/HIVE-14686.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10544 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1139/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1139/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1139/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12827726 - PreCommit-HIVE-MASTER-Build
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch, HIVE-14686.5.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14039) HiveServer2: Make the usage of server with JDBC thirft serde enabled, backward compatible for older clients
[ https://issues.apache.org/jira/browse/HIVE-14039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-14039: Committed to master and branch-2.1. Thanks [~ziyangz]! > HiveServer2: Make the usage of server with JDBC thirft serde enabled, > backward compatible for older clients > --- > > Key: HIVE-14039 > URL: https://issues.apache.org/jira/browse/HIVE-14039 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.0.1 >Reporter: Vaibhav Gumashta >Assignee: Ziyang Zhao > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14039.5.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14039) HiveServer2: Make the usage of server with JDBC thirft serde enabled, backward compatible for older clients
[ https://issues.apache.org/jira/browse/HIVE-14039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-14039: Fix Version/s: 2.1.1 2.2.0 > HiveServer2: Make the usage of server with JDBC thirft serde enabled, > backward compatible for older clients > --- > > Key: HIVE-14039 > URL: https://issues.apache.org/jira/browse/HIVE-14039 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.0.1 >Reporter: Vaibhav Gumashta >Assignee: Ziyang Zhao > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14039.5.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14039) HiveServer2: Make the usage of server with JDBC thirft serde enabled, backward compatible for older clients
[ https://issues.apache.org/jira/browse/HIVE-14039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476489#comment-15476489 ] Vaibhav Gumashta commented on HIVE-14039: - Test failures unrelated. Will commit shortly. > HiveServer2: Make the usage of server with JDBC thirft serde enabled, > backward compatible for older clients > --- > > Key: HIVE-14039 > URL: https://issues.apache.org/jira/browse/HIVE-14039 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.0.1 >Reporter: Vaibhav Gumashta >Assignee: Ziyang Zhao > Attachments: HIVE-14039.5.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14591) HS2 is shut down unexpectedly during the startup time
[ https://issues.apache.org/jira/browse/HIVE-14591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-14591: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to master and branch-2.1. Thanks [~taoli-hwx] > HS2 is shut down unexpectedly during the startup time > - > > Key: HIVE-14591 > URL: https://issues.apache.org/jira/browse/HIVE-14591 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.1.0 >Reporter: Tao Li >Assignee: Tao Li > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14591.1.patch > > > If there is issue with Zookeeper (e.g. connection issues), then it takes HS2 > some time to connect. During this time, Ambari could issue health checks > against HS2 and the CloseSession call will trigger the shutdown of HS2, which > is not expected. That triggering should happen only when the HS2 has been > deregistered with Zookeeper, not during the startup time when HS2 is not > registered with ZK yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14591) HS2 is shut down unexpectedly during the startup time
[ https://issues.apache.org/jira/browse/HIVE-14591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-14591: Affects Version/s: 1.2.1 2.1.0 > HS2 is shut down unexpectedly during the startup time > - > > Key: HIVE-14591 > URL: https://issues.apache.org/jira/browse/HIVE-14591 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.1.0 >Reporter: Tao Li >Assignee: Tao Li > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14591.1.patch > > > If there is issue with Zookeeper (e.g. connection issues), then it takes HS2 > some time to connect. During this time, Ambari could issue health checks > against HS2 and the CloseSession call will trigger the shutdown of HS2, which > is not expected. That triggering should happen only when the HS2 has been > deregistered with Zookeeper, not during the startup time when HS2 is not > registered with ZK yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14591) HS2 is shut down unexpectedly during the startup time
[ https://issues.apache.org/jira/browse/HIVE-14591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-14591: Fix Version/s: 2.1.1 2.2.0 > HS2 is shut down unexpectedly during the startup time > - > > Key: HIVE-14591 > URL: https://issues.apache.org/jira/browse/HIVE-14591 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.1.0 >Reporter: Tao Li >Assignee: Tao Li > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14591.1.patch > > > If there is issue with Zookeeper (e.g. connection issues), then it takes HS2 > some time to connect. During this time, Ambari could issue health checks > against HS2 and the CloseSession call will trigger the shutdown of HS2, which > is not expected. That triggering should happen only when the HS2 has been > deregistered with Zookeeper, not during the startup time when HS2 is not > registered with ZK yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14098) Logging task properties, and environment variables might contain passwords
[ https://issues.apache.org/jira/browse/HIVE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-14098: -- Affects Version/s: 2.1.0 Target Version/s: 2.1.1 Status: Patch Available (was: Reopened) The precommit test did not run on the branch. Trying to submit the patch > Logging task properties, and environment variables might contain passwords > -- > > Key: HIVE-14098 > URL: https://issues.apache.org/jira/browse/HIVE-14098 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Logging, Spark >Affects Versions: 2.1.0 >Reporter: Peter Vary >Assignee: Peter Vary > Fix For: 2.2.0 > > Attachments: HIVE-14098-branch-2.1.patch, HIVE-14098.2.patch, > HIVE-14098.patch > > > Hive MapredLocalTask Can Print Environment Passwords, like > -Djavax.net.ssl.trustStorePassword. > The same could happen, when logging spark properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476462#comment-15476462
]
Rui Li commented on HIVE-14686:
---
+1 pending test
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch, HIVE-14686.5.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476439#comment-15476439
]
Fan Yunbo commented on HIVE-14686:
--
[~ruili] Thanks for your advice again.
The braces of if statement have been added.
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch, HIVE-14686.5.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fan Yunbo updated HIVE-14686:
-
Status: Patch Available (was: Open)
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch, HIVE-14686.5.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fan Yunbo updated HIVE-14686:
-
Attachment: HIVE-14686.5.patch
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch, HIVE-14686.5.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14727) llap-server may case file descriptor leak in BuddyAllocator class
[ https://issues.apache.org/jira/browse/HIVE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yechao Chen updated HIVE-14727: --- Status: Patch Available (was: In Progress) [~prasanth_j] [~sershe] may someone please review this small patch? > llap-server may case file descriptor leak in BuddyAllocator class > - > > Key: HIVE-14727 > URL: https://issues.apache.org/jira/browse/HIVE-14727 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Yechao Chen >Assignee: Yechao Chen > Attachments: HIVE-14727.patch > > > llap-server,the method preallocate(int) of BuddyAllocator may case file > descriptor leak when FileChannel map allocate memory error. > the code: > //here if failed > ByteBuffer rwbuf = rwf.getChannel().map(MapMode.READ_WRITE, 0, > arenaSize); > // A mapping, once established, is not dependent upon the file > channel that was used to > // create it. delete file and hold onto the map >//can not close() and delete file > rwf.close(); > rf.delete(); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14727) llap-server may case file descriptor leak in BuddyAllocator class
[ https://issues.apache.org/jira/browse/HIVE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yechao Chen updated HIVE-14727: --- Attachment: HIVE-14727.patch > llap-server may case file descriptor leak in BuddyAllocator class > - > > Key: HIVE-14727 > URL: https://issues.apache.org/jira/browse/HIVE-14727 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Yechao Chen >Assignee: Yechao Chen > Attachments: HIVE-14727.patch > > > llap-server,the method preallocate(int) of BuddyAllocator may case file > descriptor leak when FileChannel map allocate memory error. > the code: > //here if failed > ByteBuffer rwbuf = rwf.getChannel().map(MapMode.READ_WRITE, 0, > arenaSize); > // A mapping, once established, is not dependent upon the file > channel that was used to > // create it. delete file and hold onto the map >//can not close() and delete file > rwf.close(); > rf.delete(); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-14727) llap-server may case file descriptor leak in BuddyAllocator class
[ https://issues.apache.org/jira/browse/HIVE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-14727 started by Yechao Chen. -- > llap-server may case file descriptor leak in BuddyAllocator class > - > > Key: HIVE-14727 > URL: https://issues.apache.org/jira/browse/HIVE-14727 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Yechao Chen >Assignee: Yechao Chen > > llap-server,the method preallocate(int) of BuddyAllocator may case file > descriptor leak when FileChannel map allocate memory error. > the code: > //here if failed > ByteBuffer rwbuf = rwf.getChannel().map(MapMode.READ_WRITE, 0, > arenaSize); > // A mapping, once established, is not dependent upon the file > channel that was used to > // create it. delete file and hold onto the map >//can not close() and delete file > rwf.close(); > rf.delete(); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476386#comment-15476386
]
Rui Li commented on HIVE-14686:
---
Thanks Yunbo for the update. Just a nit:
{code}
+if (command_type == CREATE_TABLE || command_type == CTLT)
+queryState.setCommandType(HiveOperation.CREATETABLE);
+else if (command_type == CTAS)
+queryState.setCommandType(HiveOperation.CREATETABLE_AS_SELECT);
+else
+throw new SemanticException("Unrecognized command.");
{code}
Please don't omit the braces of if statement.
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476379#comment-15476379
]
Hive QA commented on HIVE-14686:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827716/HIVE-14686.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10544 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1138/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1138/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1138/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12827716 - PreCommit-HIVE-MASTER-Build
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476358#comment-15476358
]
Fan Yunbo commented on HIVE-14686:
--
[~ruili]Thanks for apply so quickly.
Sorry for misunderstanding.
Update again.
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fan Yunbo updated HIVE-14686:
-
Attachment: HIVE-14686.4.patch
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fan Yunbo updated HIVE-14686:
-
Attachment: (was: HIVE-14686.4.patch)
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fan Yunbo updated HIVE-14686:
-
Status: Open (was: Patch Available)
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476295#comment-15476295
]
Rui Li commented on HIVE-14686:
---
Sorry if I wasn't being clear enough. I think we should set the cmd type after
the for loop. For example, in the for loop we may firstly decide
{{command_type}} is CTAS. But later we may also find CTLT so this is an error.
In this case, we shouldn't modify the cmd type of the query state.
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476187#comment-15476187
]
Fan Yunbo commented on HIVE-14686:
--
[~ruili]
I also thank so and I update the patch.
It's really nice of you to review it again.
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fan Yunbo updated HIVE-14686:
-
Status: Patch Available (was: Open)
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fan Yunbo updated HIVE-14686:
-
Attachment: HIVE-14686.4.patch
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch, HIVE-14686.4.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fan Yunbo updated HIVE-14686:
-
Status: Open (was: Patch Available)
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14706) Lineage information not set properly
[
https://issues.apache.org/jira/browse/HIVE-14706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476102#comment-15476102
]
Vimal Sharma commented on HIVE-14706:
-
cc [~madhan.neethiraj] [~shwethags] [~rhbutani]
> Lineage information not set properly
>
>
> Key: HIVE-14706
> URL: https://issues.apache.org/jira/browse/HIVE-14706
> Project: Hive
> Issue Type: Bug
>Reporter: Vimal Sharma
>Priority: Critical
>
> I am trying to fetch column level lineage after a CTAS query in a Post
> Execution hook in Hive. Below are the queries:
> {code}
> create table t1(id int, name string);
> create table t2 as select * from t1;
> {code}
> The lineage information is retrieved using the following sample piece of code:
> {code}
> lInfo = hookContext.getLinfo()
> for(Map.Entry e :
> lInfo.entrySet()) {
> System.out.println("Col Lineage Key : " + e.getKey());
> System.out.println("Col Lineage Value: " + e.getValue());
> }
> {code}
> The Dependency field(i.e Col Lineage Value) is coming in as null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[
https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476051#comment-15476051
]
Rui Li commented on HIVE-14686:
---
Hi Yunbo, how about setting the command type after the for loop
{code}
for (int num = 1; num < numCh; num++)
{code}
Because in the loop we can hit errors like CTAS_CTLT_COEXISTENCE. So let's not
change the original behaviour in that case. I guess it also means you don't
have to set a default value.
> Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS
> ... AS"
> --
>
> Key: HIVE-14686
> URL: https://issues.apache.org/jira/browse/HIVE-14686
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Fan Yunbo
>Assignee: Fan Yunbo
> Fix For: 2.2.0
>
> Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch,
> HIVE-14686.3.patch
>
>
> See the query:
> {code}
> create table if not exists DST as select * from SRC;
> {code}
> if the table DST doesn't exist, SessionState.get().getHiveOperation() will
> return HiveOperation.CREATETABLE_AS_SELECT;
> But if the table DST already exists, it will return HiveOperation.CREATETABLE;
> It really makes some trouble for those who judge operation type by
> SessionState.get().getHiveOperation().
> The reason I find out is that the function analyzeCreateTable in
> SemanticAnalyzer.java will return null and won't set the correct command type
> if the table already exists.
> Here is the related code:
> {code}
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> } catch (HiveException e) {
> // should not occur since second parameter to getTableWithQN is false
> throw new IllegalStateException("Unxpected Exception thrown: " +
> e.getMessage(), e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
