[
https://issues.apache.org/jira/browse/HIVE-9381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281253#comment-14281253
]
Lefty Leverenz commented on HIVE-9381:
--------------------------------------
Doc note: Sushanth documented *hcat.append.limit* in the "Storage directives"
section of HCatalog Config Properties.
* [HCatalog Config Properties |
https://cwiki.apache.org/confluence/display/Hive/HCatalog+Config+Properties]
> HCatalog hardcodes maximum append limit to 1000.
> ------------------------------------------------
>
> Key: HIVE-9381
> URL: https://issues.apache.org/jira/browse/HIVE-9381
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
> Reporter: Sushanth Sowmyan
> Assignee: Sushanth Sowmyan
> Fix For: 0.15.0
>
> Attachments: HIVE-9381.patch
>
>
> When HCatalog appends to a table, it tries to avoid fileclashes while moving
> by appending a "_a_NNN" suffix to a file if there is a clash, where the NNN
> is a number from 0 to 999.
> This limit is hardcoded and this causes issues with some users who have
> tables with a large number of files. Ideally, we should push back on those
> users to change their usecase, or run some manner of compactions on their
> table (a CREATE TABLE AS SELECT from this table to another table, followed by
> a drop and re creating as select from the other table would suffice, for
> example). But, for users where those are not viable solutions, we need to
> help them get unstuck. One way is by making that hardcoded parameter a
> configuration parameter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
