[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009355#comment-15009355 ] Andrew Purtell commented on HBASE-14791: Comments on the v2 patch, basically LGTM: - BufferedHTable should be annotated as Private - I take back my earlier suggestion that buffering deletes should be disabled by default. FWIW > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > Attachments: HBASE-14791-0.98-v1.patch, HBASE-14791-0.98-v2.patch > > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue is with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, causing a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009558#comment-15009558 ] Alex Araujo commented on HBASE-14791: - Annotated BufferedHTable as Private and removed option to disable delete buffering in latest patch. Also dropped version number from patch name, to see if build bot runs it through tests this time. > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > Attachments: HBASE-14791-0.98-v1.patch, HBASE-14791-0.98-v2.patch, > HBASE-14791-0.98.patch > > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue is with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, causing a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007698#comment-15007698 ] Alex Araujo commented on HBASE-14791: - [~lhofhansl], v2 attached > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > Attachments: HBASE-14791-0.98-v1.patch, HBASE-14791-0.98-v2.patch > > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue is with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, causing a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004957#comment-15004957 ] Alex Araujo commented on HBASE-14791: - Thanks for reviewing [~lhofhansl]]! Changing HTable to HTableInterface does not break compatibility for subclasses of TableOutputFormat because HTable is private. Subclassing HTable instead of using delegation would save a fair amount of boilerplate in BufferedHTable, and allow both Mutation types to share the same buffer (addresses the ordering issue). I'll upload a v2 that subclasses HTable and preserves ordering. > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > Attachments: HBASE-14791-0.98-v1.patch > > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue is with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, causing a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003336#comment-15003336 ] Lars Hofhansl commented on HBASE-14791: --- Looks good! Two questions: # Would subclassing (as opposed to delegation as used in the patch) save us a bunch of code? # Would the change from HTable to HTableInterface break compatibility for folks subclasses TableOutputFormat? (would not be an issue too if we do the subclassing of #1) > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > Attachments: HBASE-14791-0.98-v1.patch > > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue is with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, causing a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001464#comment-15001464 ] Alex Araujo commented on HBASE-14791: - Attached a v1 that does size based buffering for Deletes, similar to how Puts are buffered. It does not change 0.98 HTable semantics, and is disabled by default as [~apurtell] suggested. I agree that two buffering mechanisms is not ideal. If correctness is an issue and/or we want to avoid separate buffers, we could try to use the Put buffering in HTable for both and disable it for Deletes by default. That would align 0.98 buffering more closely with BufferedMutator in 1.0+, but would also be more invasive. > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > Attachments: HBASE-14791-0.98-v1.patch > > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue is with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, causing a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999064#comment-14999064 ] Lars Hofhansl commented on HBASE-14791: --- Cool! > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, cause a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999028#comment-14999028 ] Vikas Vishwakarma commented on HBASE-14791: --- also thanks to [~sukuna...@gmail.com] for helping with debugging the issue > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, cause a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999243#comment-14999243 ] Alex Araujo commented on HBASE-14791: - The patch for HBASE-12728 moved buffering of Puts from HTable into BufferedMutator for 1.0+. Since BufferedMutator is not Put specific, it also made TableOutputFormat and MultiTableOutputFormat use BufferedMutator for all Mutation types. In 0.98 there is no buffering of Deletes in HTable or elsewhere as far as I can tell. Essentially, we'd need to implement a basic BufferedMutator and use it for both OutputFormat types. The downside is that we would be duplicating some of the buffering code in HTable. > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue is with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, causing a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999342#comment-14999342 ] Andrew Purtell commented on HBASE-14791: bq. Essentially, we'd need to implement a basic BufferedMutator and use it for both OutputFormat types. The downside is that we would be duplicating some of the buffering code in HTable. I think this is fine given the trade-offs here. We don't want to make a significant change in HTable semantics in 0.98. We do want to improve performance of TableOutputFormat substantially for Deletes. A reasonable option is some 0.98-specific buffering in TableOutputFormat. It would be more palatable a change if the new behavior is configurable, defaulting to off (backwards compatible FWIW) > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue is with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, causing a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers
[ https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1443#comment-1443 ] Lars Hofhansl commented on HBASE-14791: --- One tricky aspect of this is that we generally want to keep the order of the deletes w.r.t. puts as much as possible. If have one buffering mechanism for puts and another for deletes that is hard to maintain. For correctness it is enough to ensure that deletes are shipped after the puts, not sure that's easy to do, though. Then again in cases where we want to ship the deletes, there's better an appropriate setup on the receiving to keep delete markers around correct, otherwise it makes not sense to ship them... not maybe not an issue at all? > [0.98] CopyTable is extremely slow when moving delete markers > - > > Key: HBASE-14791 > URL: https://issues.apache.org/jira/browse/HBASE-14791 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.16 >Reporter: Lars Hofhansl >Assignee: Alex Araujo > > We found that some of our copy table job run for many hours, even when there > isn't that much data to copy. > [~vik.karma] did his magic and found that the issue is with copying delete > markers (we use raw mode to also move deletes across). > Looking at the code in 0.98 it's immediately obvious that deletes (unlike > puts) are not batched and hence sent to the other side one by one, causing a > network RTT for each delete marker. > Looks like in trunk it's doing the right thing (using BufferedMutators for > all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, > 1.2?) issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
