[jira] [Work started] (HIVE-24092) Implement additional JDBC methods required by JDBC storage handler
[
https://issues.apache.org/jira/browse/HIVE-24092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-24092 started by Kishen Das.
-
> Implement additional JDBC methods required by JDBC storage handler
> --
>
> Key: HIVE-24092
> URL: https://issues.apache.org/jira/browse/HIVE-24092
> Project: Hive
> Issue Type: Bug
> Components: JDBC storage handler
>Reporter: Jesus Camacho Rodriguez
>Assignee: Kishen Das
>Priority: Major
>
> Calcite may rely on the following JDBC methods to generate SQL queries for
> Hive JDBC storage handler, which in the case of Hive itself, return a
> {{Method not supported}} exception. We should implement such methods:
> {code}
> nullsAreSortedAtEnd
> nullsAreSortedAtStart
> nullsAreSortedLow
> nullsAreSortedHigh
> storesLowerCaseIdentifiers
> storesLowerCaseQuotedIdentifiers
> storesMixedCaseIdentifiers
> storesMixedCaseQuotedIdentifiers
> storesUpperCaseIdentifiers
> storesUpperCaseQuotedIdentifiers
> supportsMixedCaseIdentifiers
> supportsMixedCaseQuotedIdentifiers
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24092) Implement additional JDBC methods required by JDBC storage handler
[
https://issues.apache.org/jira/browse/HIVE-24092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kishen Das reassigned HIVE-24092:
-
Assignee: Kishen Das
> Implement additional JDBC methods required by JDBC storage handler
> --
>
> Key: HIVE-24092
> URL: https://issues.apache.org/jira/browse/HIVE-24092
> Project: Hive
> Issue Type: Bug
> Components: JDBC storage handler
>Reporter: Jesus Camacho Rodriguez
>Assignee: Kishen Das
>Priority: Major
>
> Calcite may rely on the following JDBC methods to generate SQL queries for
> Hive JDBC storage handler, which in the case of Hive itself, return a
> {{Method not supported}} exception. We should implement such methods:
> {code}
> nullsAreSortedAtEnd
> nullsAreSortedAtStart
> nullsAreSortedLow
> nullsAreSortedHigh
> storesLowerCaseIdentifiers
> storesLowerCaseQuotedIdentifiers
> storesMixedCaseIdentifiers
> storesMixedCaseQuotedIdentifiers
> storesUpperCaseIdentifiers
> storesUpperCaseQuotedIdentifiers
> supportsMixedCaseIdentifiers
> supportsMixedCaseQuotedIdentifiers
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22782) Consolidate metastore call to fetch constraints
[
https://issues.apache.org/jira/browse/HIVE-22782?focusedWorklogId=476147&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476147
]
ASF GitHub Bot logged work on HIVE-22782:
-
Author: ASF GitHub Bot
Created on: 29/Aug/20 18:53
Start Date: 29/Aug/20 18:53
Worklog Time Spent: 10m
Work Description: ashish-kumar-sharma commented on a change in pull
request #1419:
URL: https://github.com/apache/hive/pull/1419#discussion_r479678899
##
File path:
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestGetAllTableConstraints.java
##
@@ -0,0 +1,382 @@
+package org.apache.hadoop.hive.metastore.client;
+
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.MetaStoreTestUtils;
+import org.apache.hadoop.hive.metastore.annotation.MetastoreCheckinTest;
+import org.apache.hadoop.hive.metastore.api.AllTableConstraintsRequest;
+import org.apache.hadoop.hive.metastore.api.Catalog;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.PrimaryKeysRequest;
+import org.apache.hadoop.hive.metastore.api.SQLAllTableConstraints;
+import org.apache.hadoop.hive.metastore.api.SQLCheckConstraint;
+import org.apache.hadoop.hive.metastore.api.SQLDefaultConstraint;
+import org.apache.hadoop.hive.metastore.api.SQLForeignKey;
+import org.apache.hadoop.hive.metastore.api.SQLNotNullConstraint;
+import org.apache.hadoop.hive.metastore.api.SQLPrimaryKey;
+import org.apache.hadoop.hive.metastore.api.SQLUniqueConstraint;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.client.builder.CatalogBuilder;
+import org.apache.hadoop.hive.metastore.client.builder.DatabaseBuilder;
+import
org.apache.hadoop.hive.metastore.client.builder.SQLCheckConstraintBuilder;
+import
org.apache.hadoop.hive.metastore.client.builder.SQLDefaultConstraintBuilder;
+import org.apache.hadoop.hive.metastore.client.builder.SQLForeignKeyBuilder;
+import
org.apache.hadoop.hive.metastore.client.builder.SQLNotNullConstraintBuilder;
+import org.apache.hadoop.hive.metastore.client.builder.SQLPrimaryKeyBuilder;
+import
org.apache.hadoop.hive.metastore.client.builder.SQLUniqueConstraintBuilder;
+import org.apache.hadoop.hive.metastore.client.builder.TableBuilder;
+import org.apache.hadoop.hive.metastore.minihms.AbstractMetaStoreService;
+import org.apache.thrift.TException;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.util.List;
+
+import static org.apache.hadoop.hive.metastore.Warehouse.DEFAULT_DATABASE_NAME;
+
+@RunWith(Parameterized.class)
+@Category(MetastoreCheckinTest.class)
+public class TestGetAllTableConstraints
+extends MetaStoreClientTest {
+ private static final String OTHER_DATABASE =
"test_constraints_other_database";
+ private static final String OTHER_CATALOG = "test_constraints_other_catalog";
+ private static final String DATABASE_IN_OTHER_CATALOG =
"test_constraints_database_in_other_catalog";
+ private final AbstractMetaStoreService metaStore;
+ private IMetaStoreClient client;
+ private Table[] testTables = new Table[3];
+ private Database inOtherCatalog;
+
+ public TestGetAllTableConstraints(String name, AbstractMetaStoreService
metaStore) throws Exception {
+this.metaStore = metaStore;
+ }
+
+ @Before
+ public void setUp() throws Exception {
+// Get new client
+client = metaStore.getClient();
+
+// Clean up the database
+client.dropDatabase(OTHER_DATABASE, true, true, true);
+// Drop every table in the default database
+for (String tableName : client.getAllTables(DEFAULT_DATABASE_NAME)) {
+ client.dropTable(DEFAULT_DATABASE_NAME, tableName, true, true, true);
+}
+
+client.dropDatabase(OTHER_CATALOG, DATABASE_IN_OTHER_CATALOG, true, true,
true);
+try {
+ client.dropCatalog(OTHER_CATALOG);
+} catch (NoSuchObjectException e) {
+ // NOP
+}
+
+// Clean up trash
+metaStore.cleanWarehouseDirs();
+
+new DatabaseBuilder().setName(OTHER_DATABASE).create(client,
metaStore.getConf());
+
+Catalog cat =
+new
CatalogBuilder().setName(OTHER_CATALOG).setLocation(MetaStoreTestUtils.getTestWarehouseDir(OTHER_CATALOG))
+.build();
+client.createCatalog(cat);
+
+// For this one don't specify a location to make sure it gets put in the
catalog directory
+inOtherCatalog = new
DatabaseBuilder().setName(DATABASE_IN_OTHER_CATALOG).setCatalogName(OTHER_CATALOG)
+.create(client, metaStore.getConf());
+
+testTables[0] = new
TableBuilder().setTableName("test_table
[jira] [Comment Edited] (HIVE-24092) Implement additional JDBC methods required by JDBC storage handler
[
https://issues.apache.org/jira/browse/HIVE-24092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187068#comment-17187068
]
Jesus Camacho Rodriguez edited comment on HIVE-24092 at 8/29/20, 5:29 PM:
--
An attempt off the top of my head at the return values for those calls:
{code}
nullsAreSortedAtEnd -> false
nullsAreSortedAtStart -> false
nullsAreSortedLow -> depends on session configuration : !HIVE_DEFAULT_NULLS_LAST
nullsAreSortedHigh -> depends on session configuration : HIVE_DEFAULT_NULLS_LAST
storesLowerCaseIdentifiers -> true
storesLowerCaseQuotedIdentifiers -> true
storesMixedCaseIdentifiers -> false
storesMixedCaseQuotedIdentifiers -> false
storesUpperCaseIdentifiers -> false
storesUpperCaseQuotedIdentifiers -> false
supportsMixedCaseIdentifiers -> false
supportsMixedCaseQuotedIdentifiers -> false
{code}
was (Author: jcamachorodriguez):
An attempt from the top of my head at the return values for those calls:
{code}
nullsAreSortedAtEnd -> false
nullsAreSortedAtStart -> false
nullsAreSortedLow -> depends on session configuration : !HIVE_DEFAULT_NULLS_LAST
nullsAreSortedHigh -> depends on session configuration : HIVE_DEFAULT_NULLS_LAST
storesLowerCaseIdentifiers -> true
storesLowerCaseQuotedIdentifiers -> true
storesMixedCaseIdentifiers -> false
storesMixedCaseQuotedIdentifiers -> false
storesUpperCaseIdentifiers -> false
storesUpperCaseQuotedIdentifiers -> false
supportsMixedCaseIdentifiers -> false
supportsMixedCaseQuotedIdentifiers -> false
{code}
> Implement additional JDBC methods required by JDBC storage handler
> --
>
> Key: HIVE-24092
> URL: https://issues.apache.org/jira/browse/HIVE-24092
> Project: Hive
> Issue Type: Bug
> Components: JDBC storage handler
>Reporter: Jesus Camacho Rodriguez
>Priority: Major
>
> Calcite may rely on the following JDBC methods to generate SQL queries for
> Hive JDBC storage handler, which in the case of Hive itself, return a
> {{Method not supported}} exception. We should implement such methods:
> {code}
> nullsAreSortedAtEnd
> nullsAreSortedAtStart
> nullsAreSortedLow
> nullsAreSortedHigh
> storesLowerCaseIdentifiers
> storesLowerCaseQuotedIdentifiers
> storesMixedCaseIdentifiers
> storesMixedCaseQuotedIdentifiers
> storesUpperCaseIdentifiers
> storesUpperCaseQuotedIdentifiers
> supportsMixedCaseIdentifiers
> supportsMixedCaseQuotedIdentifiers
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24092) Implement additional JDBC methods required by JDBC storage handler
[
https://issues.apache.org/jira/browse/HIVE-24092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187068#comment-17187068
]
Jesus Camacho Rodriguez commented on HIVE-24092:
An attempt from the top of my head at the return values for those calls:
{code}
nullsAreSortedAtEnd -> false
nullsAreSortedAtStart -> false
nullsAreSortedLow -> depends on session configuration : !HIVE_DEFAULT_NULLS_LAST
nullsAreSortedHigh -> depends on session configuration : HIVE_DEFAULT_NULLS_LAST
storesLowerCaseIdentifiers -> true
storesLowerCaseQuotedIdentifiers -> true
storesMixedCaseIdentifiers -> false
storesMixedCaseQuotedIdentifiers -> false
storesUpperCaseIdentifiers -> false
storesUpperCaseQuotedIdentifiers -> false
supportsMixedCaseIdentifiers -> false
supportsMixedCaseQuotedIdentifiers -> false
{code}
> Implement additional JDBC methods required by JDBC storage handler
> --
>
> Key: HIVE-24092
> URL: https://issues.apache.org/jira/browse/HIVE-24092
> Project: Hive
> Issue Type: Bug
> Components: JDBC storage handler
>Reporter: Jesus Camacho Rodriguez
>Priority: Major
>
> Calcite may rely on the following JDBC methods to generate SQL queries for
> Hive JDBC storage handler, which in the case of Hive itself, return a
> {{Method not supported}} exception. We should implement such methods:
> {code}
> nullsAreSortedAtEnd
> nullsAreSortedAtStart
> nullsAreSortedLow
> nullsAreSortedHigh
> storesLowerCaseIdentifiers
> storesLowerCaseQuotedIdentifiers
> storesMixedCaseIdentifiers
> storesMixedCaseQuotedIdentifiers
> storesUpperCaseIdentifiers
> storesUpperCaseQuotedIdentifiers
> supportsMixedCaseIdentifiers
> supportsMixedCaseQuotedIdentifiers
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24090) NPE while SJ reduction due to missing null check for col stats
[
https://issues.apache.org/jira/browse/HIVE-24090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186973#comment-17186973
]
Vipin Vishvkarma commented on HIVE-24090:
-
[~zabetak] Yes, sorry the stacktrace is from a private branch. For master NPE
will be at this line
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L2071]
> NPE while SJ reduction due to missing null check for col stats
> --
>
> Key: HIVE-24090
> URL: https://issues.apache.org/jira/browse/HIVE-24090
> Project: Hive
> Issue Type: Bug
>Reporter: Vipin Vishvkarma
>Assignee: Vipin Vishvkarma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hitting NPE while SJ reduction due to missing col stats
> {code:java}
> Error(1647)) - FAILED: NullPointerException null
> java.lang.NullPointerException at
> org.apache.hadoop.hive.ql.stats.StatsUtils.updateStats(StatsUtils.java:2111)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1629)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:498)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:209)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:144)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12642)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11960)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24090) NPE while SJ reduction due to missing null check for col stats
[
https://issues.apache.org/jira/browse/HIVE-24090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186962#comment-17186962
]
Stamatis Zampetakis commented on HIVE-24090:
The NPE stacktrace does not seem to match current master. Is the problem still
reproducible?
> NPE while SJ reduction due to missing null check for col stats
> --
>
> Key: HIVE-24090
> URL: https://issues.apache.org/jira/browse/HIVE-24090
> Project: Hive
> Issue Type: Bug
>Reporter: Vipin Vishvkarma
>Assignee: Vipin Vishvkarma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hitting NPE while SJ reduction due to missing col stats
> {code:java}
> Error(1647)) - FAILED: NullPointerException null
> java.lang.NullPointerException at
> org.apache.hadoop.hive.ql.stats.StatsUtils.updateStats(StatsUtils.java:2111)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1629)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:498)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:209)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:144)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12642)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11960)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23930) Upgrade to tez 0.10.x
[ https://issues.apache.org/jira/browse/HIVE-23930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23930: Description: Tez 0.10.0 is not yet released, but this ticket is for tracking the effort and the needed hive changes. Currently, Hive depends on 0.9.1 Hadoop dependencies: Hive/master: *3.1.0* Tez/master: *3.1.3* Tez/branch-0.9: *2.7.2* TODOs: - check why HIVE-23689 broke some unit tests intermittently (0.9.2 ->0.9.3 bump), because a 0.10.x upgrade will also contain those tez changes which could be related - maintain the needed hive changes (reflecting tez api changes): HIVE-23190: LLAP: modify IndexCache to pass filesystem object to TezSpillRecord was: Tez 0.10.1 is not yet released, but this ticket is for tracking the effort and the needed hive changes. Currently, Hive depends on 0.9.1 Hadoop dependencies: Hive/master: *3.1.0* Tez/master: *3.1.3* Tez/branch-0.9: *2.7.2* TODOs: - check why HIVE-23689 broke some unit tests intermittently (0.9.2 ->0.9.3 bump), because a 0.10.x upgrade will also contain those tez changes which could be related - maintain the needed hive changes (reflecting tez api changes): HIVE-23190: LLAP: modify IndexCache to pass filesystem object to TezSpillRecord > Upgrade to tez 0.10.x > - > > Key: HIVE-23930 > URL: https://issues.apache.org/jira/browse/HIVE-23930 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Tez 0.10.0 is not yet released, but this ticket is for tracking the effort > and the needed hive changes. > Currently, Hive depends on 0.9.1 > Hadoop dependencies: > Hive/master: *3.1.0* > Tez/master: *3.1.3* > Tez/branch-0.9: *2.7.2* > TODOs: > - check why HIVE-23689 broke some unit tests intermittently (0.9.2 ->0.9.3 > bump), because a 0.10.x upgrade will also contain those tez changes which > could be related > - maintain the needed hive changes (reflecting tez api changes): > HIVE-23190: LLAP: modify IndexCache to pass filesystem object to > TezSpillRecord -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22782) Consolidate metastore call to fetch constraints
[ https://issues.apache.org/jira/browse/HIVE-22782?focusedWorklogId=476091&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476091 ] ASF GitHub Bot logged work on HIVE-22782: - Author: ASF GitHub Bot Created on: 29/Aug/20 09:53 Start Date: 29/Aug/20 09:53 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma commented on a change in pull request #1419: URL: https://github.com/apache/hive/pull/1419#discussion_r479632452 ## File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java ## @@ -2811,6 +2811,26 @@ public GetFieldsResponse getFieldsRequest(GetFieldsRequest req) return client.get_check_constraints(req).getCheckConstraints(); } + @Override + public SQLAllTableConstraints getAllTableConstraints(AllTableConstraintsRequest req) Review comment: https://issues.apache.org/jira/browse/HIVE-22782 ## File path: standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestGetAllTableConstraints.java ## @@ -0,0 +1,382 @@ +package org.apache.hadoop.hive.metastore.client; + +import org.apache.hadoop.hive.metastore.IMetaStoreClient; +import org.apache.hadoop.hive.metastore.MetaStoreTestUtils; +import org.apache.hadoop.hive.metastore.annotation.MetastoreCheckinTest; +import org.apache.hadoop.hive.metastore.api.AllTableConstraintsRequest; +import org.apache.hadoop.hive.metastore.api.Catalog; +import org.apache.hadoop.hive.metastore.api.Database; +import org.apache.hadoop.hive.metastore.api.NoSuchObjectException; +import org.apache.hadoop.hive.metastore.api.PrimaryKeysRequest; +import org.apache.hadoop.hive.metastore.api.SQLAllTableConstraints; +import org.apache.hadoop.hive.metastore.api.SQLCheckConstraint; +import org.apache.hadoop.hive.metastore.api.SQLDefaultConstraint; +import org.apache.hadoop.hive.metastore.api.SQLForeignKey; +import org.apache.hadoop.hive.metastore.api.SQLNotNullConstraint; +import org.apache.hadoop.hive.metastore.api.SQLPrimaryKey; +import org.apache.hadoop.hive.metastore.api.SQLUniqueConstraint; +import org.apache.hadoop.hive.metastore.api.Table; +import org.apache.hadoop.hive.metastore.client.builder.CatalogBuilder; +import org.apache.hadoop.hive.metastore.client.builder.DatabaseBuilder; +import org.apache.hadoop.hive.metastore.client.builder.SQLCheckConstraintBuilder; +import org.apache.hadoop.hive.metastore.client.builder.SQLDefaultConstraintBuilder; +import org.apache.hadoop.hive.metastore.client.builder.SQLForeignKeyBuilder; +import org.apache.hadoop.hive.metastore.client.builder.SQLNotNullConstraintBuilder; +import org.apache.hadoop.hive.metastore.client.builder.SQLPrimaryKeyBuilder; +import org.apache.hadoop.hive.metastore.client.builder.SQLUniqueConstraintBuilder; +import org.apache.hadoop.hive.metastore.client.builder.TableBuilder; +import org.apache.hadoop.hive.metastore.minihms.AbstractMetaStoreService; +import org.apache.thrift.TException; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; + +import java.util.List; + +import static org.apache.hadoop.hive.metastore.Warehouse.DEFAULT_DATABASE_NAME; + +@RunWith(Parameterized.class) +@Category(MetastoreCheckinTest.class) Review comment: Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 476091) Time Spent: 1h 20m (was: 1h 10m) > Consolidate metastore call to fetch constraints > --- > > Key: HIVE-22782 > URL: https://issues.apache.org/jira/browse/HIVE-22782 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently separate calls are made to metastore to fetch constraints like Pk, > fk, not null etc. Since planner always retrieve these constraints we should > retrieve all of them in one call. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-24091) Replace multiple constraints call with getAllTableConstraints api call in query planner
[ https://issues.apache.org/jira/browse/HIVE-24091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-24091 started by Ashish Sharma. > Replace multiple constraints call with getAllTableConstraints api call in > query planner > --- > > Key: HIVE-24091 > URL: https://issues.apache.org/jira/browse/HIVE-24091 > Project: Hive > Issue Type: Improvement >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Major > > Inorder get all the constraints of table i.e. PrimaryKey, ForeignKey, > UniqueConstraint ,NotNullConstraint ,DefaultConstraint ,CheckConstraint. We > have to do 6 different metastore call. Replace these call with one > getAllTableConstraints api which provide all the constraints at once -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24091) Replace multiple constraints call with getAllTableConstraints api call in query planner
[ https://issues.apache.org/jira/browse/HIVE-24091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Sharma reassigned HIVE-24091: > Replace multiple constraints call with getAllTableConstraints api call in > query planner > --- > > Key: HIVE-24091 > URL: https://issues.apache.org/jira/browse/HIVE-24091 > Project: Hive > Issue Type: Improvement >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Major > > Inorder get all the constraints of table i.e. PrimaryKey, ForeignKey, > UniqueConstraint ,NotNullConstraint ,DefaultConstraint ,CheckConstraint. We > have to do 6 different metastore call. Replace these call with one > getAllTableConstraints api which provide all the constraints at once -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22622) Hive allows to create a struct with duplicate attribute names
[
https://issues.apache.org/jira/browse/HIVE-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186923#comment-17186923
]
Stamatis Zampetakis commented on HIVE-22622:
If the queries do not reference the column with the problematic struct there is
no exception. This means that a lot of time can pass before realizing that
there is a problem which can make the problem hard to debug.
> Hive allows to create a struct with duplicate attribute names
> -
>
> Key: HIVE-22622
> URL: https://issues.apache.org/jira/browse/HIVE-22622
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Krisztian Kasa
>Priority: Major
>
> When you create at table with a struct with twice the same attribute name,
> hive allow you to create it.
> create table test_struct( duplicateColumn struct);
> You can insert data into it :
> insert into test_struct select named_struct("id",1,"id",1);
> But you can not read it :
> select * from test_struct;
> Return : java.io.IOException: java.io.IOException: Error reading file:
> hdfs://.../test_struct/delta_001_001_/bucket_0 ,
> We can create and insert. but fail on read the Struct part of the tables. We
> can still read all other columns (if we have more than one) but not the
> struct anymore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22622) Hive allows to create a struct with duplicate attribute names
[
https://issues.apache.org/jira/browse/HIVE-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186922#comment-17186922
]
Stamatis Zampetakis commented on HIVE-22622:
I just tried the following test on master and it fails.
{code:sql}
CREATE TABLE person
(
`id` int,
`address` struct
)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat';
INSERT INTO person
VALUES (1, named_struct('number', 61, 'street', 'Terrasse', 'number', 62));
INSERT INTO person
VALUES (2, named_struct('number', 51, 'street', 'Terrasse', 'number', 52));
SELECT address.number FROM person;
{code}
And it fails with the following exception when performing the SELECT statement:
{noformat}
java.io.IOException: java.io.IOException: Error reading file:
file:/home/stamatis/Projects/Apache/hive/itests/qtest/target/localfs/warehouse/person/00_0
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:638)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:545)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:556)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:243)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:279)
at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:740)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:710)
at
org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170)
at
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
at
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver(TestMiniLlapLocalCliDriver.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:135)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at
org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:95)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.ma
[jira] [Work logged] (HIVE-24090) NPE while SJ reduction due to missing null check for col stats
[
https://issues.apache.org/jira/browse/HIVE-24090?focusedWorklogId=476086&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476086
]
ASF GitHub Bot logged work on HIVE-24090:
-
Author: ASF GitHub Bot
Created on: 29/Aug/20 08:59
Start Date: 29/Aug/20 08:59
Worklog Time Spent: 10m
Work Description: vpnvishv commented on pull request #1442:
URL: https://github.com/apache/hive/pull/1442#issuecomment-683260899
@jcamachor Can you please review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 476086)
Time Spent: 20m (was: 10m)
> NPE while SJ reduction due to missing null check for col stats
> --
>
> Key: HIVE-24090
> URL: https://issues.apache.org/jira/browse/HIVE-24090
> Project: Hive
> Issue Type: Bug
>Reporter: Vipin Vishvkarma
>Assignee: Vipin Vishvkarma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hitting NPE while SJ reduction due to missing col stats
> {code:java}
> Error(1647)) - FAILED: NullPointerException null
> java.lang.NullPointerException at
> org.apache.hadoop.hive.ql.stats.StatsUtils.updateStats(StatsUtils.java:2111)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1629)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:498)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:209)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:144)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12642)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11960)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24090) NPE while SJ reduction due to missing null check for col stats
[
https://issues.apache.org/jira/browse/HIVE-24090?focusedWorklogId=476085&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476085
]
ASF GitHub Bot logged work on HIVE-24090:
-
Author: ASF GitHub Bot
Created on: 29/Aug/20 08:55
Start Date: 29/Aug/20 08:55
Worklog Time Spent: 10m
Work Description: vpnvishv opened a new pull request #1442:
URL: https://github.com/apache/hive/pull/1442
### What changes were proposed in this pull request?
Added null check
### Why are the changes needed?
To avoid NPE
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 476085)
Remaining Estimate: 0h
Time Spent: 10m
> NPE while SJ reduction due to missing null check for col stats
> --
>
> Key: HIVE-24090
> URL: https://issues.apache.org/jira/browse/HIVE-24090
> Project: Hive
> Issue Type: Bug
>Reporter: Vipin Vishvkarma
>Assignee: Vipin Vishvkarma
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hitting NPE while SJ reduction due to missing col stats
> {code:java}
> Error(1647)) - FAILED: NullPointerException null
> java.lang.NullPointerException at
> org.apache.hadoop.hive.ql.stats.StatsUtils.updateStats(StatsUtils.java:2111)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1629)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:498)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:209)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:144)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12642)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11960)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24090) NPE while SJ reduction due to missing null check for col stats
[
https://issues.apache.org/jira/browse/HIVE-24090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24090:
--
Labels: pull-request-available (was: )
> NPE while SJ reduction due to missing null check for col stats
> --
>
> Key: HIVE-24090
> URL: https://issues.apache.org/jira/browse/HIVE-24090
> Project: Hive
> Issue Type: Bug
>Reporter: Vipin Vishvkarma
>Assignee: Vipin Vishvkarma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hitting NPE while SJ reduction due to missing col stats
> {code:java}
> Error(1647)) - FAILED: NullPointerException null
> java.lang.NullPointerException at
> org.apache.hadoop.hive.ql.stats.StatsUtils.updateStats(StatsUtils.java:2111)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1629)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:498)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:209)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:144)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12642)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11960)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24090) NPE while SJ reduction due to missing null check for col stats
[
https://issues.apache.org/jira/browse/HIVE-24090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vipin Vishvkarma reassigned HIVE-24090:
---
> NPE while SJ reduction due to missing null check for col stats
> --
>
> Key: HIVE-24090
> URL: https://issues.apache.org/jira/browse/HIVE-24090
> Project: Hive
> Issue Type: Bug
>Reporter: Vipin Vishvkarma
>Assignee: Vipin Vishvkarma
>Priority: Major
>
> Hitting NPE while SJ reduction due to missing col stats
> {code:java}
> Error(1647)) - FAILED: NullPointerException null
> java.lang.NullPointerException at
> org.apache.hadoop.hive.ql.stats.StatsUtils.updateStats(StatsUtils.java:2111)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1629)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:498)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:209)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:144)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12642)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11960)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
