[ https://issues.apache.org/jira/browse/HIVE-11229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627732#comment-14627732 ]
Elliot West commented on HIVE-11229: ------------------------------------ The failed test does not appear to be related to the patch. > Mutation API: Coordinator communication with meta store should be optional > -------------------------------------------------------------------------- > > Key: HIVE-11229 > URL: https://issues.apache.org/jira/browse/HIVE-11229 > Project: Hive > Issue Type: Bug > Components: HCatalog > Affects Versions: 2.0.0 > Reporter: Elliot West > Assignee: Elliot West > Labels: streaming_api > Attachments: HIVE-11229.1.patch > > > [~ekoifman] raised a theoretical issue with the streaming mutation API > (HIVE-10165) where worker nodes operating in a distributed cluster might > overwhelm a meta store while trying to obtain partition locks. Although this > does not happen in practice (see HIVE-11228), the API does communicate with > the meta store in this manner to obtain partition paths and create new > partitions. Therefore the issue described does in fact exist in the current > implementation, albeit in a different code path. I’d like to make such > communication optional like so: > * When the user chooses not to create partitions on demand, no meta store > connection will be created in the {{MutationCoordinators}}. Additionally, > partition paths will be resolved using > {{org.apache.hadoop.hive.metastore.Warehouse.getPartitionPath(Path, > LinkedHashMap<String, String>)}} which should be suitable so long as standard > Hive partition layouts are followed. > * If the user does choose to create partitions on demand then the system will > operate as is does currently; using the meta store to both issue > {{add_partition}} events and look up partition meta data. > * The documentation will be updated to describe these behaviours and outline > alternative approaches to collecting affected partition names and creating > partitions in a less intensive manner. > Side note for follow up: The parameter names {{tblName}} and {{dbName}} seem > to be the wrong way around on the method > {{org.apache.hadoop.hive.metastore.IMetaStoreClient.getPartition(String, > String, List<String>)}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)