[jira] [Updated] (HIVE-20534) File operation at target side during S3 replication slowing down the replication
[ https://issues.apache.org/jira/browse/HIVE-20534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis updated HIVE-20534: --- Fix Version/s: (was: 4.0.0) I cleared the fixVersion field since this ticket is still open. Please review this ticket and if the fix is already committed to a specific version please set the version accordingly and mark the ticket as RESOLVED. According to the [JIRA guidelines|https://cwiki.apache.org/confluence/display/Hive/HowToContribute] the fixVersion should be set only when the issue is resolved/closed. > File operation at target side during S3 replication slowing down the > replication > > > Key: HIVE-20534 > URL: https://issues.apache.org/jira/browse/HIVE-20534 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > > 1. Check is there during create partition for existence of partition location > (in add partitions core method in metastore.java). It’s not required as we > would have created the directory and copied the required files to it. > 2. Creating qualified directory name (convertAddSpecToMetaPartition method in > hive.java)– File system is access to check if the path provided is fully > qualified or not. Not sure why it’s taking 1-2 seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-20534) File operation at target side during S3 replication slowing down the replication
[ https://issues.apache.org/jira/browse/HIVE-20534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20534: --- Description: 1. Check is there during create partition for existence of partition location (in add partitions core method in metastore.java). It’s not required as we would have created the directory and copied the required files to it. 2. Creating qualified directory name (convertAddSpecToMetaPartition method in hive.java)– File system is access to check if the path provided is fully qualified or not. Not sure why it’s taking 1-2 seconds. was:Add notification for add partition and insert operation adds the list of files added by the operation. The file listing is done at target side for replication load. This takes 2-3 seconds as s3 is slow. This can be improved by using the file list from the event directory and same can be used to populate the notification table. > File operation at target side during S3 replication slowing down the > replication > > > Key: HIVE-20534 > URL: https://issues.apache.org/jira/browse/HIVE-20534 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > 1. Check is there during create partition for existence of partition location > (in add partitions core method in metastore.java). It’s not required as we > would have created the directory and copied the required files to it. > 2. Creating qualified directory name (convertAddSpecToMetaPartition method in > hive.java)– File system is access to check if the path provided is fully > qualified or not. Not sure why it’s taking 1-2 seconds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
