Daniel Dai created HIVE-20720: --------------------------------- Summary: Add partition column option to JDBC handler Key: HIVE-20720 URL: https://issues.apache.org/jira/browse/HIVE-20720 Project: Hive Issue Type: New Feature Components: StorageHandler Reporter: Daniel Dai Assignee: Daniel Dai
Currently JdbcStorageHandler does not split input in Tez. The reason is numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" in Tez. And "mapreduce.job.maps" is not a valid param if ranger is in use. User ends up always use 1 split. We need to rely on this new feature if we want to support multi-splits. Here is my proposal: 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound in tblproperties if user want to split jdbc data source. In case lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min query to get this in planner. We can currently limit partitionColumn to only numeric/date/timestamp column for simplicity 2. If partitionColumn/numPartitions are not specified, don't split input 3. Splits are equal intervals without respect to data distribution 4. There is also a "hive.sql.query.split" flag vetos the split (can be set manually or automatically by calcite) 5. If partitionColumn is not defined, but numPartitions is defined, use original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)