[
https://issues.apache.org/jira/browse/HCATALOG-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538645#comment-13538645
]
Sushanth Sowmyan commented on HCATALOG-538:
-------------------------------------------
Hi, I just created HCATALOG-580 that follows an e2e failure caused due to this
patch.
I want to make sure that any "fixes" we make there does not cause too
significant a performance degradation here, so could you please track that as
we solve it? Also, if you have any solutions, I'm all ears. :)
> HCatalogStorer fails for 100GB of data with dynamic partitioning (number of
> partition is 300)
> ---------------------------------------------------------------------------------------------
>
> Key: HCATALOG-538
> URL: https://issues.apache.org/jira/browse/HCATALOG-538
> Project: HCatalog
> Issue Type: Bug
> Affects Versions: 0.4, 0.5
> Environment: Hadoop 0.23.4
> HCatalog 0.4
> Reporter: Arup Malakar
> Assignee: Arup Malakar
> Fix For: 0.4.1
>
> Attachments: HCATALOG-538-branch0.4-0.patch,
> HCATALOG-538-trunk-0.patch
>
>
> A hadoop job with 100GB of data and 300 partitions fails. All the maps
> succeed fine but the commit job fails after that. This looks like a timeout
> issue as commitJob() takes more than 10 minutes. I am running this on
> hadoop-0.23.4. I am playing with yarn.nm.liveness-monitor.expiry-interval-ms,
> yarn.am.liveness-monitor.expiry-interval-ms etc to make it work.
> This JIRA is for optimizing the commitJob(), as 10 minutes is too long.
> On a side note for storing 100GB of data without partition takes ~12 minutes,
> same amount of data with 300 partitions fails after 45 minutes. These tests
> were run on a 10 node cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
