[
https://issues.apache.org/jira/browse/HIVE-20534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-20534:
---------------------------------------
Description:
1. Check is there during create partition for existence of partition location
(in add partitions core method in metastore.java). It’s not required as we
would have created the directory and copied the required files to it.
2. Creating qualified directory name (convertAddSpecToMetaPartition method in
hive.java)– File system is access to check if the path provided is fully
qualified or not. Not sure why it’s taking 1-2 seconds.
was:Add notification for add partition and insert operation adds the list of
files added by the operation. The file listing is done at target side for
replication load. This takes 2-3 seconds as s3 is slow. This can be improved by
using the file list from the event directory and same can be used to populate
the notification table.
> File operation at target side during S3 replication slowing down the
> replication
> --------------------------------------------------------------------------------
>
> Key: HIVE-20534
> URL: https://issues.apache.org/jira/browse/HIVE-20534
> Project: Hive
> Issue Type: Sub-task
> Components: repl
> Affects Versions: 4.0.0
> Reporter: mahesh kumar behera
> Assignee: mahesh kumar behera
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> 1. Check is there during create partition for existence of partition location
> (in add partitions core method in metastore.java). It’s not required as we
> would have created the directory and copied the required files to it.
> 2. Creating qualified directory name (convertAddSpecToMetaPartition method in
> hive.java)– File system is access to check if the path provided is fully
> qualified or not. Not sure why it’s taking 1-2 seconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
