[
https://issues.apache.org/jira/browse/PHOENIX-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966902#comment-15966902
]
James Taylor edited comment on PHOENIX-3782 at 4/13/17 12:16 AM:
-----------------------------------------------------------------
The check for too many rows in the MutationState (and subsequent
throwIfTooBig() call) serves a different purpose. It protects against too much
memory due to too many rows being held on to in a single commit. We still need
to guard against this and throw and we can't split up the work yet. The client
needs to be able to decide which rows are submitted in a single commit. Once
the commit occurs, we're free to chunk them up however we want, though.
I'd suggest making the check byte-based like the others and keeping it as-is.
I'll file another JIRA to have a new check for the max number of rows in a
chunk. We could essentially transfer over which ever is the smaller chunk based
on both the byte-size and row-size chunk size. I've created PHOENIX-3784 for
this.
was (Author: jamestaylor):
The check for too many rows in the MutationState (and subsequent
throwIfTooBig() call) serves a different purpose. It protects against too much
memory due to too many rows being held on to in a single commit. We still need
to guard against this and throw and we can't split up the work yet. The client
needs to be able to decide which rows are submitted in a single commit. Once
the commit occurs, we're free to chunk them up however we want, though.
I'd suggest making the check byte-based like the others and keeping it as-is.
I'll file another JIRA to have a new check for the max number of rows in a
chunk. We could essentially transfer over which ever is the smaller chunk based
on both the byte-size and row-size chunk size.
> MutationStates with too many rows should transparently split requests instead
> of throwing exception
> ---------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-3782
> URL: https://issues.apache.org/jira/browse/PHOENIX-3782
> Project: Phoenix
> Issue Type: Improvement
> Affects Versions: 4.10.0
> Reporter: Geoffrey Jacoby
> Assignee: Geoffrey Jacoby
>
> For a long time the Phoenix client has had a throwIfTooBig() check in
> MutationState that throws an exception if the MutationState has more rows
> than a config value. Since the correct solution is always "split up the rows
> into manageable chunks and submit to HBase one chunk at a time", we should do
> that rather than throwing back to the user.
> In addition, we should lower the default for the max rows, which right now is
> quite high at 500K rows.
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
