[jira] [Updated] (HIVE-7389) Reduce number of metastore calls in MoveTask (when loading dynamic partitions)
[ https://issues.apache.org/jira/browse/HIVE-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-7389: - Resolution: Fixed Status: Resolved (was: Patch Available) committed to trunk and hive-14 Reduce number of metastore calls in MoveTask (when loading dynamic partitions) -- Key: HIVE-7389 URL: https://issues.apache.org/jira/browse/HIVE-7389 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: performance Fix For: 0.14.0 Attachments: HIVE-7389.1.patch, HIVE-7389.2.patch, local_vm_testcase.txt When the number of dynamic partitions to be loaded are high, the time taken for 'MoveTask' is greater than the actual job in some scenarios. It would be possible to reduce overall runtime by reducing the number of calls made to metastore from MoveTask operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7389) Reduce number of metastore calls in MoveTask (when loading dynamic partitions)
[ https://issues.apache.org/jira/browse/HIVE-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-7389: --- Attachment: HIVE-7389.2.patch rebasing the patch to trunk. Reduce number of metastore calls in MoveTask (when loading dynamic partitions) -- Key: HIVE-7389 URL: https://issues.apache.org/jira/browse/HIVE-7389 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Labels: performance Attachments: HIVE-7389.1.patch, HIVE-7389.2.patch, local_vm_testcase.txt When the number of dynamic partitions to be loaded are high, the time taken for 'MoveTask' is greater than the actual job in some scenarios. It would be possible to reduce overall runtime by reducing the number of calls made to metastore from MoveTask operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7389) Reduce number of metastore calls in MoveTask (when loading dynamic partitions)
[ https://issues.apache.org/jira/browse/HIVE-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-7389: - Priority: Critical (was: Major) Reduce number of metastore calls in MoveTask (when loading dynamic partitions) -- Key: HIVE-7389 URL: https://issues.apache.org/jira/browse/HIVE-7389 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical Labels: performance Fix For: 0.14.0 Attachments: HIVE-7389.1.patch, HIVE-7389.2.patch, local_vm_testcase.txt When the number of dynamic partitions to be loaded are high, the time taken for 'MoveTask' is greater than the actual job in some scenarios. It would be possible to reduce overall runtime by reducing the number of calls made to metastore from MoveTask operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7389) Reduce number of metastore calls in MoveTask (when loading dynamic partitions)
[ https://issues.apache.org/jira/browse/HIVE-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-7389: - Fix Version/s: 0.14.0 Reduce number of metastore calls in MoveTask (when loading dynamic partitions) -- Key: HIVE-7389 URL: https://issues.apache.org/jira/browse/HIVE-7389 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Labels: performance Fix For: 0.14.0 Attachments: HIVE-7389.1.patch, HIVE-7389.2.patch, local_vm_testcase.txt When the number of dynamic partitions to be loaded are high, the time taken for 'MoveTask' is greater than the actual job in some scenarios. It would be possible to reduce overall runtime by reducing the number of calls made to metastore from MoveTask operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7389) Reduce number of metastore calls in MoveTask (when loading dynamic partitions)
[ https://issues.apache.org/jira/browse/HIVE-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-7389: - Status: Patch Available (was: Open) Reduce number of metastore calls in MoveTask (when loading dynamic partitions) -- Key: HIVE-7389 URL: https://issues.apache.org/jira/browse/HIVE-7389 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Labels: performance Attachments: HIVE-7389.1.patch, local_vm_testcase.txt When the number of dynamic partitions to be loaded are high, the time taken for 'MoveTask' is greater than the actual job in some scenarios. It would be possible to reduce overall runtime by reducing the number of calls made to metastore from MoveTask operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7389) Reduce number of metastore calls in MoveTask (when loading dynamic partitions)
[ https://issues.apache.org/jira/browse/HIVE-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-7389: --- Affects Version/s: 0.14.0 Reduce number of metastore calls in MoveTask (when loading dynamic partitions) -- Key: HIVE-7389 URL: https://issues.apache.org/jira/browse/HIVE-7389 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Rajesh Balamohan Labels: performance When the number of dynamic partitions to be loaded are high, the time taken for 'MoveTask' is greater than the actual job in some scenarios. It would be possible to reduce overall runtime by reducing the number of calls made to metastore from MoveTask operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7389) Reduce number of metastore calls in MoveTask (when loading dynamic partitions)
[ https://issues.apache.org/jira/browse/HIVE-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-7389: --- Attachment: local_vm_testcase.txt Actual issue was observed in a larger cluster. Attaching here a simple test case which can be run on local vm. Reduce number of metastore calls in MoveTask (when loading dynamic partitions) -- Key: HIVE-7389 URL: https://issues.apache.org/jira/browse/HIVE-7389 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Rajesh Balamohan Labels: performance Attachments: local_vm_testcase.txt When the number of dynamic partitions to be loaded are high, the time taken for 'MoveTask' is greater than the actual job in some scenarios. It would be possible to reduce overall runtime by reducing the number of calls made to metastore from MoveTask operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7389) Reduce number of metastore calls in MoveTask (when loading dynamic partitions)
[ https://issues.apache.org/jira/browse/HIVE-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-7389: --- Attachment: HIVE-7389.1.patch Observed 10% improvement in overall runtime of the ETL workload (for a table with 1000 partitions). Reduce number of metastore calls in MoveTask (when loading dynamic partitions) -- Key: HIVE-7389 URL: https://issues.apache.org/jira/browse/HIVE-7389 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Rajesh Balamohan Labels: performance Attachments: HIVE-7389.1.patch, local_vm_testcase.txt When the number of dynamic partitions to be loaded are high, the time taken for 'MoveTask' is greater than the actual job in some scenarios. It would be possible to reduce overall runtime by reducing the number of calls made to metastore from MoveTask operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
