[
https://issues.apache.org/jira/browse/HIVE-24343?focusedWorklogId=510183&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-510183
]
ASF GitHub Bot logged work on HIVE-24343:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 11/Nov/20 10:50
Start Date: 11/Nov/20 10:50
Worklog Time Spent: 10m
Work Description: vnhive commented on a change in pull request #1640:
URL: https://github.com/apache/hive/pull/1640#discussion_r521273558
##########
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
##########
@@ -870,44 +870,88 @@ private boolean isViewTable(String catName, String
dbName, String tblName) throw
List<? extends Object> paramsForFilter, List<String> joinsForFilter,
Integer max)
throws MetaException {
boolean doTrace = LOG.isDebugEnabled();
+
+ // The JDBC client driver implementations of Derby, PostgreSQL, MySQL,
MariaDB and Oracle send the number of
+ // parameters being set in a executed statement as a 2 byte signed integer
to the servers. This indirectly
+ // sets s limit on the number of parameters that can be set to 32767. This
is also the number of parameters
+ // that can be passed to the JDO executeArray call.
+ final int JDBC_STMT_PARAM_LIMIT = 32767;
+
final String dbNameLcase = dbName.toLowerCase();
final String tblNameLcase = tblName.toLowerCase();
final String catNameLcase = normalizeSpace(catName).toLowerCase();
// We have to be mindful of order during filtering if we are not returning
all partitions.
+ // If the number of parameters is larger than 32767 and the query is split
into multiple
+ // parts, the order by would work within each query part.
String orderForFilter = (max != null) ? " order by \"PART_NAME\" asc" : "";
String queryText =
- "select " + PARTITIONS + ".\"PART_ID\" from " + PARTITIONS + ""
- + " inner join " + TBLS + " on " + PARTITIONS + ".\"TBL_ID\" = " + TBLS
+ ".\"TBL_ID\" "
- + " and " + TBLS + ".\"TBL_NAME\" = ? "
- + " inner join " + DBS + " on " + TBLS + ".\"DB_ID\" = " + DBS +
".\"DB_ID\" "
- + " and " + DBS + ".\"NAME\" = ? "
- + join(joinsForFilter, ' ')
- + " where " + DBS + ".\"CTLG_NAME\" = ? "
- + (StringUtils.isBlank(sqlFilter) ? "" : (" and " + sqlFilter)) +
orderForFilter;
- Object[] params = new Object[paramsForFilter.size() + 3];
- params[0] = tblNameLcase;
- params[1] = dbNameLcase;
- params[2] = catNameLcase;
- for (int i = 0; i < paramsForFilter.size(); ++i) {
- params[i + 3] = paramsForFilter.get(i);
- }
+ "select " + PARTITIONS + ".\"PART_ID\" from " + PARTITIONS + ""
Review comment:
Will do ! Will make the change and resubmit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 510183)
Time Spent: 40m (was: 0.5h)
> Table partition operations (create, drop, select) fail when the number of
> partitions is greater than 32767 (signed int)
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-24343
> URL: https://issues.apache.org/jira/browse/HIVE-24343
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Narayanan Venkateswaran
> Assignee: Narayanan Venkateswaran
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The table partition operations - create, drop, select access the underlying
> relation database using JDO, which internally routes the operations through
> the JDBC driver. Most of the underlying JDBC driver implementations place a
> limit on the number of parameters that can be passed through a statement
> implementation. The limitations are as follows,
> postgreSQL - 32767
> (https://www.postgresql.org/message-id/16832734.post%40talk.nabble.com)
> MySQL - 32767 - 2 Byte Integer - num of params
> (https://dev.mysql.com/doc/internals/en/com-stmt-prepare-response.html#packet-COM_STMT_PREPARE_OK)
> Oracle - 32767 -
> https://www.jooq.org/doc/3.12/manual/sql-building/dsl-context/custom-settings/settings-inline-threshold/
> Derby - 32767 - stored in an unsinged integer - Note the Prepared
> Statement implementation here -
> [https://svn.apache.org/repos/asf/db/derby/code/branches/10.1/java/client/org/apache/derby/client/am/PreparedStatement.java]
>
> These limits should be taken into account when querying the underlying
> metastore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
