[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778596#comment-13778596 ] Hudson commented on HIVE-4914: -- FAILURE: Integrated in Hive-trunk-h0.21 #2358 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2358/]) HIVE-4914 : filtering via partition name should be done inside metastore server (implementation) (Sergey Shelukhin via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1526106) * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/metastore/if/hive_metastore.thrift * /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp * /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h * /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp * /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp * /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsByExprRequest.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsByExprResult.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java * /hive/trunk/metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php * /hive/trunk/metastore/src/gen/thrift/gen-php/metastore/Types.php * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py * /hive/trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb * /hive/trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/PartitionExpressionProxy.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionExpressionForMetastore.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/metastore/VerifyingObjectStore.java filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: D12561.5.patch, D12561.6.patch, D12561.7.patch, HIVE-4914.01.patch, HIVE-4914.02.patch, HIVE-4914.03.patch, HIVE-4914.04.patch, HIVE-4914.05.patch, HIVE-4914.06.patch, HIVE-4914.07.patch, HIVE-4914.D12561.1.patch, HIVE-4914.D12561.2.patch, HIVE-4914.D12561.3.patch, HIVE-4914.D12561.4.patch, HIVE-4914.D12645.1.patch, HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch, HIVE-4914.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see:
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778623#comment-13778623 ] Hudson commented on HIVE-4914: -- ABORTED: Integrated in Hive-trunk-hadoop2 #457 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/457/]) HIVE-4914 : filtering via partition name should be done inside metastore server (implementation) (Sergey Shelukhin via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1526106) * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/metastore/if/hive_metastore.thrift * /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp * /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h * /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp * /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp * /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsByExprRequest.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsByExprResult.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java * /hive/trunk/metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php * /hive/trunk/metastore/src/gen/thrift/gen-php/metastore/Types.php * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py * /hive/trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb * /hive/trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/PartitionExpressionProxy.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionExpressionForMetastore.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/metastore/VerifyingObjectStore.java filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: D12561.5.patch, D12561.6.patch, D12561.7.patch, HIVE-4914.01.patch, HIVE-4914.02.patch, HIVE-4914.03.patch, HIVE-4914.04.patch, HIVE-4914.05.patch, HIVE-4914.06.patch, HIVE-4914.07.patch, HIVE-4914.D12561.1.patch, HIVE-4914.D12561.2.patch, HIVE-4914.D12561.3.patch, HIVE-4914.D12561.4.patch, HIVE-4914.D12645.1.patch, HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch, HIVE-4914.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see:
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777394#comment-13777394 ] Hudson commented on HIVE-4914: -- FAILURE: Integrated in Hive-trunk-hadoop1-ptest #181 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/181/]) HIVE-4914 : filtering via partition name should be done inside metastore server (implementation) (Sergey Shelukhin via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1526106) * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/metastore/if/hive_metastore.thrift * /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp * /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h * /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp * /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp * /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsByExprRequest.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsByExprResult.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java * /hive/trunk/metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php * /hive/trunk/metastore/src/gen/thrift/gen-php/metastore/Types.php * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py * /hive/trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb * /hive/trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/PartitionExpressionProxy.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionExpressionForMetastore.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/metastore/VerifyingObjectStore.java filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: D12561.5.patch, D12561.6.patch, D12561.7.patch, HIVE-4914.01.patch, HIVE-4914.02.patch, HIVE-4914.03.patch, HIVE-4914.04.patch, HIVE-4914.05.patch, HIVE-4914.06.patch, HIVE-4914.07.patch, HIVE-4914.D12561.1.patch, HIVE-4914.D12561.2.patch, HIVE-4914.D12561.3.patch, HIVE-4914.D12561.4.patch, HIVE-4914.D12645.1.patch, HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch, HIVE-4914.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see:
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777410#comment-13777410 ] Hudson commented on HIVE-4914: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #115 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/115/]) HIVE-4914 : filtering via partition name should be done inside metastore server (implementation) (Sergey Shelukhin via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1526106) * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/metastore/if/hive_metastore.thrift * /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp * /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h * /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp * /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp * /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsByExprRequest.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsByExprResult.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java * /hive/trunk/metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php * /hive/trunk/metastore/src/gen/thrift/gen-php/metastore/Types.php * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py * /hive/trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb * /hive/trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/PartitionExpressionProxy.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionExpressionForMetastore.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/metastore/VerifyingObjectStore.java filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: D12561.5.patch, D12561.6.patch, D12561.7.patch, HIVE-4914.01.patch, HIVE-4914.02.patch, HIVE-4914.03.patch, HIVE-4914.04.patch, HIVE-4914.05.patch, HIVE-4914.06.patch, HIVE-4914.07.patch, HIVE-4914.D12561.1.patch, HIVE-4914.D12561.2.patch, HIVE-4914.D12561.3.patch, HIVE-4914.D12561.4.patch, HIVE-4914.D12645.1.patch, HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch, HIVE-4914.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see:
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[
https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13776174#comment-13776174
]
Hive QA commented on HIVE-4914:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12604648/HIVE-4914.06.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 3149 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.templeton.tool.TestJobIDParser.testParseHive
org.apache.hive.hcatalog.templeton.tool.TestJobIDParser.testParseJar
org.apache.hive.hcatalog.templeton.tool.TestJobIDParser.testParsePig
org.apache.hive.hcatalog.templeton.tool.TestJobIDParser.testParseStreaming
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/865/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/865/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
filtering via partition name should be done inside metastore server
(implementation)
Key: HIVE-4914
URL: https://issues.apache.org/jira/browse/HIVE-4914
Project: Hive
Issue Type: Improvement
Components: Metastore
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: D12561.5.patch, D12561.6.patch, HIVE-4914.01.patch,
HIVE-4914.02.patch, HIVE-4914.03.patch, HIVE-4914.04.patch,
HIVE-4914.05.patch, HIVE-4914.06.patch, HIVE-4914.D12561.1.patch,
HIVE-4914.D12561.2.patch, HIVE-4914.D12561.3.patch, HIVE-4914.D12561.4.patch,
HIVE-4914.D12645.1.patch, HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch,
HIVE-4914.patch, HIVE-4914.patch, HIVE-4914.patch
Currently, if the filter pushdown is impossible (which is most cases), the
client gets all partition names from metastore, filters them, and asks for
partitions by names for the filtered set.
Metastore server code should do that instead; it should check if pushdown is
possible and do it if so; otherwise it should do name-based filtering.
Saves the roundtrip with all partition names from the server to client, and
also removes the need to have pushdown viability checking on both sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13776498#comment-13776498 ] Phabricator commented on HIVE-4914: --- ashutoshc has requested changes to the revision HIVE-4914 [jira] filtering via partition name should be done inside metastore server (implementation). INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:21 Why do we need to this? Simply use runtimeSerializationKryo. ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:68 Did you encounter into any issue while serializing timestamps? We have other tests which serialize timestamp via Kryo, we haven't encountered any problems there yet. metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:32 I don't see any error handling for method not found and failure to deserialize expression exceptions. Are you planning to add them? REVISION DETAIL https://reviews.facebook.net/D12561 To: JIRA, ashutoshc, sershe filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: D12561.5.patch, D12561.6.patch, HIVE-4914.01.patch, HIVE-4914.02.patch, HIVE-4914.03.patch, HIVE-4914.04.patch, HIVE-4914.05.patch, HIVE-4914.06.patch, HIVE-4914.D12561.1.patch, HIVE-4914.D12561.2.patch, HIVE-4914.D12561.3.patch, HIVE-4914.D12561.4.patch, HIVE-4914.D12645.1.patch, HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch, HIVE-4914.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13776680#comment-13776680 ] Phabricator commented on HIVE-4914: --- sershe has commented on the revision HIVE-4914 [jira] filtering via partition name should be done inside metastore server (implementation). INLINE COMMENTS metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:32 It is not here, this class exposes specific methods so it just gives you the error. IncompatibleMetastoreException is thrown either directly by get_partitions_by_expr, from server, or by listPartitionsByExpr. Hive client cannot do handling as it doesn't know about expressions, filters, etc. PartitionPruner does the handling. metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1988 the code is simply moved; jira for this is HIVE-5190 ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java:179 right now, native-ness is implied when person calls the appropriate method, there's already isNative call. Annotation will duplicate this information, and require people to explicitly maintain it... REVISION DETAIL https://reviews.facebook.net/D12561 To: JIRA, ashutoshc, sershe filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: D12561.5.patch, D12561.6.patch, HIVE-4914.01.patch, HIVE-4914.02.patch, HIVE-4914.03.patch, HIVE-4914.04.patch, HIVE-4914.05.patch, HIVE-4914.06.patch, HIVE-4914.D12561.1.patch, HIVE-4914.D12561.2.patch, HIVE-4914.D12561.3.patch, HIVE-4914.D12561.4.patch, HIVE-4914.D12645.1.patch, HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch, HIVE-4914.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[
https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777133#comment-13777133
]
Hive QA commented on HIVE-4914:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12604885/HIVE-4914.07.patch
{color:green}SUCCESS:{color} +1 3149 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/878/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/878/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
filtering via partition name should be done inside metastore server
(implementation)
Key: HIVE-4914
URL: https://issues.apache.org/jira/browse/HIVE-4914
Project: Hive
Issue Type: Improvement
Components: Metastore
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: D12561.5.patch, D12561.6.patch, D12561.7.patch,
HIVE-4914.01.patch, HIVE-4914.02.patch, HIVE-4914.03.patch,
HIVE-4914.04.patch, HIVE-4914.05.patch, HIVE-4914.06.patch,
HIVE-4914.07.patch, HIVE-4914.D12561.1.patch, HIVE-4914.D12561.2.patch,
HIVE-4914.D12561.3.patch, HIVE-4914.D12561.4.patch, HIVE-4914.D12645.1.patch,
HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch,
HIVE-4914.patch, HIVE-4914.patch
Currently, if the filter pushdown is impossible (which is most cases), the
client gets all partition names from metastore, filters them, and asks for
partitions by names for the filtered set.
Metastore server code should do that instead; it should check if pushdown is
possible and do it if so; otherwise it should do name-based filtering.
Saves the roundtrip with all partition names from the server to client, and
also removes the need to have pushdown viability checking on both sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13774793#comment-13774793 ] Sergey Shelukhin commented on HIVE-4914: I think this is caused by error handling (ExceptionListener) I added to deserializeExpression, previously XML deserialization was just silently ignoring some exceptions. Since I switched to Kryo, the changes to original methods are no longer necessary. filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: D12561.5.patch, HIVE-4914.01.patch, HIVE-4914.02.patch, HIVE-4914.03.patch, HIVE-4914.04.patch, HIVE-4914.05.patch, HIVE-4914.D12561.1.patch, HIVE-4914.D12561.2.patch, HIVE-4914.D12561.3.patch, HIVE-4914.D12561.4.patch, HIVE-4914.D12645.1.patch, HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch, HIVE-4914.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[
https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13773758#comment-13773758
]
Hive QA commented on HIVE-4914:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12604372/HIVE-4914.05.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 3129 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.io.sarg.TestSearchArgumentImpl.testExpression3
org.apache.hadoop.hive.ql.io.sarg.TestSearchArgumentImpl.testExpression5
org.apache.hadoop.hive.ql.io.sarg.TestSearchArgumentImpl.testExpression9
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/852/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/852/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
filtering via partition name should be done inside metastore server
(implementation)
Key: HIVE-4914
URL: https://issues.apache.org/jira/browse/HIVE-4914
Project: Hive
Issue Type: Improvement
Components: Metastore
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: D12561.5.patch, HIVE-4914.01.patch, HIVE-4914.02.patch,
HIVE-4914.03.patch, HIVE-4914.04.patch, HIVE-4914.05.patch,
HIVE-4914.D12561.1.patch, HIVE-4914.D12561.2.patch, HIVE-4914.D12561.3.patch,
HIVE-4914.D12561.4.patch, HIVE-4914.D12645.1.patch,
HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch,
HIVE-4914.patch, HIVE-4914.patch
Currently, if the filter pushdown is impossible (which is most cases), the
client gets all partition names from metastore, filters them, and asks for
partitions by names for the filtered set.
Metastore server code should do that instead; it should check if pushdown is
possible and do it if so; otherwise it should do name-based filtering.
Saves the roundtrip with all partition names from the server to client, and
also removes the need to have pushdown viability checking on both sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761942#comment-13761942 ] Phabricator commented on HIVE-4914: --- ashutoshc has requested changes to the revision HIVE-4914 [jira] filtering via partition name should be done inside metastore server (implementation). Also now we anyways have client side code for filter expression evaluation, I think its easier to support backward compatibility. Client can catch thrift equivalent of method not function exception while trying this function and than degrade itself to old style client side evaluation. INLINE COMMENTS metastore/if/hive_metastore.thrift:289 Now that kryo support is checked in trunk, I think its better to send kryo serialized expression in binary format instead of xml serialized string. REVISION DETAIL https://reviews.facebook.net/D12561 To: JIRA, ashutoshc, sershe filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4914.01.patch, HIVE-4914.02.patch, HIVE-4914.03.patch, HIVE-4914.D12561.1.patch, HIVE-4914.D12561.2.patch, HIVE-4914.D12561.3.patch, HIVE-4914.D12645.1.patch, HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch, HIVE-4914.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. NO PRECOMMIT TESTS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13756817#comment-13756817 ] Sergey Shelukhin commented on HIVE-4914: somehow phabricator created the new review... not sure why filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4914.01.patch, HIVE-4914.02.patch, HIVE-4914.D12561.1.patch, HIVE-4914.D12645.1.patch, HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch, HIVE-4914.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. NO PRECOMMIT TESTS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754685#comment-13754685 ] Phabricator commented on HIVE-4914: --- ashutoshc has requested changes to the revision HIVE-4914 [jira] filtering via partition name should be done inside metastore server (implementation). Comments. INLINE COMMENTS metastore/if/hive_metastore.thrift:282 Can you add a comment here defining this boolean? metastore/if/hive_metastore.thrift:510 Instead of list of parameters, can you define a struct which is passed in as an argument. That way in future if we need to add another parameter for this function, it will still be back-compat. metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:252 Consider using MetaStoreUtils::newInstance() for this. metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:249 Can you add a comment here saying this class is created via reflection to avoid circular dependency on ql package? metastore/if/hive_metastore.thrift:281 SetPartition instead of listPartition ? ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:569 Why not enhance existing deserializeExpressions() to allow it to throw exception? Or, atleast reuse the common code. metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java:494 Add description of @param FilterBuilder in javadoc here. metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java:445 Update javadoc with new param. metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java:418 Update javadoc. metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java:113 ExpressionTree is getting too large. Better to put this class and FilterBuilder in another file? metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1887 With query.setRange() this is no longer required. metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1944 Didn't get this. With this patch, Hive client wont do any work, right? metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1971 This TODO is important to resolve. Can you follow up on this? metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1965 Is this just for tests? Or is it needed? Either way, can you add a comment for it. metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:5756 This class is getting too large. May be a good idea to put some of the helper inner classes and methods in MetaStoreUtils class. metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:496 This class is also growing in size. Probably, put it in a seprate file along with TreeVisitor. REVISION DETAIL https://reviews.facebook.net/D12561 BRANCH HIVE-4914-no-gen ARCANIST PROJECT hive To: JIRA, ashutoshc, sershe filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4914.01.patch, HIVE-4914.D12561.1.patch, HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch, HIVE-4914.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. NO PRECOMMIT TESTS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13755156#comment-13755156 ] Phabricator commented on HIVE-4914: --- sershe has commented on the revision HIVE-4914 [jira] filtering via partition name should be done inside metastore server (implementation). INLINE COMMENTS metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:496 TreeVisitor is part of the tree, and this is logically the part of metastoresql... I think it makes sense to not expose it, it's a private class metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1887 that's in separate patch that is not in yet. will adjust either way when resolving conflict metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1944 the old methods are still there; old versions of hive client, or some other clients, might call them. I will add this to the comment metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1965 this is old JDO code, it just moved metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1971 filed HIVE-5190 metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:5756 I think this should be separate JIRA to refactor. I can move JDO partition retrieval into separate class, but ideally it should be a refactoring patch without code changes, otherwise it's hard to understand what broke if something does metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java:113 This class needs access to internals, and both are very small by themselves... REVISION DETAIL https://reviews.facebook.net/D12561 BRANCH HIVE-4914-no-gen ARCANIST PROJECT hive To: JIRA, ashutoshc, sershe filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4914.01.patch, HIVE-4914.D12561.1.patch, HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch, HIVE-4914.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. NO PRECOMMIT TESTS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754201#comment-13754201 ] Ashutosh Chauhan commented on HIVE-4914: If there are udfs in expression we should still do expression eval on client because * Otherwise user jar is required on server. * It will be security concern to run user code in metastore server. filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4914.01.patch, HIVE-4914.D12561.1.patch, HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch, HIVE-4914.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. NO PRECOMMIT TESTS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749332#comment-13749332 ] Ashutosh Chauhan commented on HIVE-4914: Can you create a phabricator entry for this? filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4914.01.patch, HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch, HIVE-4914.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749444#comment-13749444 ] Sergey Shelukhin commented on HIVE-4914: Phabricator won't work for this patch (due to size probably). There's an RB link above filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4914.01.patch, HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch, HIVE-4914.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[
https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13742037#comment-13742037
]
Hive QA commented on HIVE-4914:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12598355/HIVE-4914.patch
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 2879 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testFilterSinglePartition
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testPartitionFilter
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testFilterSinglePartition
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testFilterLastPartition
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testPartitionFilter
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testPartitionFilter
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testFilterSinglePartition
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testFilterLastPartition
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testFilterLastPartition
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testPartitionFilter
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testFilterLastPartition
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testFilterSinglePartition
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testFilterSinglePartition
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported2
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testPartitionFilter
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testFilterLastPartition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/460/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/460/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
filtering via partition name should be done inside metastore server
(implementation)
Key: HIVE-4914
URL: https://issues.apache.org/jira/browse/HIVE-4914
Project: Hive
Issue Type: Improvement
Components: Metastore
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch,
HIVE-4914.patch, HIVE-4914.patch
Currently, if the filter pushdown is impossible (which is most cases), the
client gets all partition names from metastore, filters them, and asks for
partitions by names for the filtered set.
Metastore server code should do that instead; it should check if pushdown is
possible and do it if so; otherwise it should do name-based filtering.
Saves the roundtrip with all partition names from the server to client, and
also removes the need to have pushdown viability checking on both sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733747#comment-13733747 ] Sergey Shelukhin commented on HIVE-4914: Entire test suite passed, which is suspicious. Let me double check. Preliminarily, the patch is ready for review. filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[
https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733866#comment-13733866
]
Hive QA commented on HIVE-4914:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12596754/HIVE-4914.patch
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/339/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/339/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-339/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1511968.
At revision 1511968.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0 to p2
+ exit 1
'
{noformat}
This message is automatically generated.
filtering via partition name should be done inside metastore server
(implementation)
Key: HIVE-4914
URL: https://issues.apache.org/jira/browse/HIVE-4914
Project: Hive
Issue Type: Improvement
Components: Metastore
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch,
HIVE-4914.patch
Currently, if the filter pushdown is impossible (which is most cases), the
client gets all partition names from metastore, filters them, and asks for
partitions by names for the filtered set.
Metastore server code should do that instead; it should check if pushdown is
possible and do it if so; otherwise it should do name-based filtering.
Saves the roundtrip with all partition names from the server to client, and
also removes the need to have pushdown viability checking on both sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13734137#comment-13734137 ] Sergey Shelukhin commented on HIVE-4914: There are few minor bugs, I will update later today. There's also some code in JDO filter pushdown that never worked in the first place, but client didn't let it execute due to more restrictive pushdown checks. I will fix/remove it. filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733038#comment-13733038 ] Sergey Shelukhin commented on HIVE-4914: https://reviews.apache.org/r/13391/ is the RB link, description from RB is: Here's the first stab at the patch. It is on top of refactoring from HIVE-4985. Most of the changes are thrift code. It unifies get-by-name and get-by-filter methods with new get-by-expr method on metastore server side, also merging in the SQL stuff. On client, the former two are removed and partitions are retrieved via the latter. As described, client only makes one call to the server for partitions, and filtering of the names is done on the server side; to that effect, the wrapper around PartitionPruner is used. I will run full test suite overnight. Some tests have already passed. filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-4914-only-no-gen.patch, HIVE-4914-only.patch, HIVE-4914.patch Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4914) filtering via partition name should be done inside metastore server (implementation)
[ https://issues.apache.org/jira/browse/HIVE-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13728414#comment-13728414 ] Sergey Shelukhin commented on HIVE-4914: This will conflict massively with HIVE-4051, I will rebase whichever comes 2nd. filtering via partition name should be done inside metastore server (implementation) Key: HIVE-4914 URL: https://issues.apache.org/jira/browse/HIVE-4914 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Currently, if the filter pushdown is impossible (which is most cases), the client gets all partition names from metastore, filters them, and asks for partitions by names for the filtered set. Metastore server code should do that instead; it should check if pushdown is possible and do it if so; otherwise it should do name-based filtering. Saves the roundtrip with all partition names from the server to client, and also removes the need to have pushdown viability checking on both sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
