[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13632707#comment-13632707 ] Hudson commented on HBASE-7801: --- Integrated in HBase-0.94-security #137 (See [https://builds.apache.org/job/HBase-0.94-security/137/]) HBASE-7801 Allow a deferred sync option per Mutation. (Revision 1468179) Result = SUCCESS larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Durability.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Mutation.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.94-v7.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, > 7801-0.96-v7.txt, 7801-0.96-v8.txt, 7801-0.96-v9.txt, > hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13632515#comment-13632515 ] Hudson commented on HBASE-7801: --- Integrated in HBase-0.94 #961 (See [https://builds.apache.org/job/HBase-0.94/961/]) HBASE-7801 Allow a deferred sync option per Mutation. (Revision 1468179) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Durability.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Mutation.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.94-v7.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, > 7801-0.96-v7.txt, 7801-0.96-v8.txt, 7801-0.96-v9.txt, > hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631982#comment-13631982 ] Ted Yu commented on HBASE-7801: --- Looks good to me. Thanks for the perseverance, Lars. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.94-v7.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, > 7801-0.96-v7.txt, 7801-0.96-v8.txt, 7801-0.96-v9.txt, > hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631892#comment-13631892 ] Lars H commented on HBASE-7801: --- Alright will have a new patch soon. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631813#comment-13631813 ] Ted Yu commented on HBASE-7801: --- bq. fall back to checking the value of writeToWAL. Before the ideal solution is found, I prefer the above. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631805#comment-13631805 ] Lars H commented on HBASE-7801: --- Neither is ideal. Which would you prefer? > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631721#comment-13631721 ] Ted Yu commented on HBASE-7801: --- We can either throw IOException at the server or, fall back to checking the value of writeToWAL. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631497#comment-13631497 ] Lars Hofhansl commented on HBASE-7801: -- Hmm... So translate to a DoNotRetryIOException at the server? > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631482#comment-13631482 ] Ted Yu commented on HBASE-7801: --- bq. I'd rather know in the client code if there's something wrong the enum. doMiniBatchMutation() is called inside HRegion which is in region server. I think we should be a bit protective. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631476#comment-13631476 ] Lars Hofhansl commented on HBASE-7801: -- We have a silent agreement that attributes starting with _ are for internal use only (same for the cluster id), I agree that that should have been documented somewhere. As for catching the exception... Not sure what we should do in this case, other than throwing an Exception. Or you mean you want to catch and ignore it and then check the value of writeToWAL? Could do that too, but it seems a bit overkill. I'd rather know in the client code if there's something wrong the enum. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631429#comment-13631429
]
Ted Yu commented on HBASE-7801:
---
{code}
+ public Durability getDurability() {
+byte[] attr = getAttribute(DURABILITY_ID_ATTR);
+if (attr == null) {
+ return writeToWAL ? Durability.USE_DEFAULT : Durability.SKIP_WAL;
+}
+return Durability.valueOf(Bytes.toInt(attr));
{code}
Is it possible that "_d.id_" has been used by certain user code ? Should
exception from Durability.valueOf() be caught so that we can check the value of
writeToWAL ?
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.94.6, 0.95.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.7, 0.95.1
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.96-full-v2.txt,
> 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt,
> 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt,
> 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631424#comment-13631424 ] Lars Hofhansl commented on HBASE-7801: -- Going to commit to 0.94 today or tomorrow, unless I hear objections. Since we had discussions about adding feaures to 0.94, please be vocal. At Salesforce we need this to do be able to seed a table with data from another system (and still allow that this is eventually replicated to a replication slave) and then allow further incremental updates with synchronous WAL sync. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631108#comment-13631108 ] ramkrishna.s.vasudevan commented on HBASE-7801: --- Patch looks good. The changes that you have not done for 0.94 makes sense. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631042#comment-13631042 ] Ted Yu commented on HBASE-7801: --- @Lars: Can you update release notes for this JIRA ? Thanks > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630942#comment-13630942 ] Hudson commented on HBASE-7801: --- Integrated in HBase-TRUNK #4060 (See [https://builds.apache.org/job/HBase-TRUNK/4060/]) HBASE-7801 Addendum - fix Mutation size (Revision 1467522) Result = SUCCESS larsh : Files : * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630940#comment-13630940 ] Lars Hofhansl commented on HBASE-7801: -- Here's a list of changes I deliberately avoided in the 0.94 patch to keep it small and low risk: * Changing the coprocessor signature. The pre/post hooks still get only a boolean indicating whether the entry written to the WAL or not * Make HRegion.interalPut honor deferred sync. Currently internalPut does not honor this at all (not even when set on the Table); it is use for checkAndPut. This patch does not fix that. * cleanup the Increment/Appand method signatures. These receive an Append (or Increment) and also a writeToWAL boolean, even though the Append/Increment have that boolean already. If we wanted to change any of these, it should be a separate jira. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.94-v6.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630930#comment-13630930 ] Lars Hofhansl commented on HBASE-7801: -- bq. I think we should add to javadoc for ASYNC_WAL that table descriptor setting is not considered. I think you misunderstand the setting. That's the case that Anoop needs where the deferred log flusher is disabled completely. In the case we also do not honor the table's settings, but flush everything synchronously, since otherwise it will never get flushed. In that case the RegionServer will log a warning on startup. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630910#comment-13630910 ] Hudson commented on HBASE-7801: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #496 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/496/]) HBASE-7801 Addendum - fix Mutation size (Revision 1467522) Result = FAILURE larsh : Files : * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630885#comment-13630885 ] Hudson commented on HBASE-7801: --- Integrated in hbase-0.95-on-hadoop2 #67 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/67/]) HBASE-7801 Addendum - fix Mutation size (Revision 1467521) HBASE-7801 Allow a deferred sync option per Mutation. (Revision 1467356) Result = FAILURE larsh : Files : * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java larsh : Files : * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Durability.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java * /hbase/branches/0.95/hbase-protocol/src/main/protobuf/Client.proto * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableOutputFormat.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/BaseRowProcessor.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiRowMutationProcessor.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowProcessor.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftUtilities.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Merge.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaMigrationConvertingToPB.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestCloneSnapshotFromClient.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultipleTimestamps.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTimestampsFilter.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestAggregateProtocol.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestBigDecimalColumnInterpreter.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverBypass.java * /hbase/branches/0.95/hbase-se
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630790#comment-13630790 ] Lars Hofhansl commented on HBASE-7801: -- Committed addendum. Thanks again Jeffrey. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630787#comment-13630787 ] Lars Hofhansl commented on HBASE-7801: -- Oops. Thanks Jeffrey, will commit addendum ASAP. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630738#comment-13630738 ] Ted Yu commented on HBASE-7801: --- bq. What Exception would be best here? How about IllegalArgumentException ? > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt, hbase-7801-addendum.patch > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630732#comment-13630732 ] Lars Hofhansl commented on HBASE-7801: -- [~te...@apache.org] Thanks for looking. bq. Should exception be raised above ? Otherwise we would see NPE somewhere else. Yep... I'll change that. What Exception would be best here? bq. So the durability of SKIP_WAL is higher than that of USE_DEFAULT ? It also says that USE_DEFAULT needs to be the first one, I think the wording is fine, unless you think it should be clarified. bq. Should the m.getWriteToWAL() call be replaced with m.getDurability() ? Could use tmpDur here. My intend was to keep the patch as small as possible. I'll change that. bq. Minor: it would be easier to maintain if the above two conditions are separated, with same this.log.sync(txid) call. Personally, I find it more readable this way. It documents clearly in the code that these two are aliases. Will post a new patch later today. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630648#comment-13630648
]
Ted Yu commented on HBASE-7801:
---
I ran 0.94 test suite with patch and all clear.
{code}
+ case ASYNC_WAL:
+// defer the sync, unless we globally can't
+if (this.deferredLogSyncDisabled) {
{code}
I think we should add to javadoc for ASYNC_WAL that table descriptor setting is
not considered.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.94.6, 0.95.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.7, 0.95.1
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.96-full-v2.txt,
> 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt,
> 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt,
> 7801-0.96-v8.txt, 7801-0.96-v9.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630645#comment-13630645 ] Hudson commented on HBASE-7801: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #495 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/495/]) HBASE-7801 Allow a deferred sync option per Mutation. (Revision 1467357) Result = FAILURE larsh : Files : * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Durability.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java * /hbase/trunk/hbase-protocol/src/main/protobuf/Client.proto * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableOutputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/BaseRowProcessor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiRowMutationProcessor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowProcessor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftUtilities.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Merge.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaMigrationConvertingToPB.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestCloneSnapshotFromClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultipleTimestamps.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTimestampsFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestAggregateProtocol.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestBigDecimalColumnInterpreter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverBypass.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java * /hbase/trunk/hbase-server/src/tes
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630621#comment-13630621
]
Ted Yu commented on HBASE-7801:
---
In
{code}
+ public static Durability valueOf(int ordinal) {
{code}
{code}
+ default: return null;
{code}
Should exception be raised above ? Otherwise we would see NPE somewhere else.
{code}
+ * Note that the items must be sorted in order of increasing durability
{code}
So the durability of SKIP_WAL is higher than that of USE_DEFAULT ?
{code}
+ Durability tmpDur = m.getDurability();
+ if (tmpDur.ordinal() > durability.ordinal()) {
+durability = tmpDur;
+ }
if (m.getWriteToWAL()) {
{code}
Should the m.getWriteToWAL() call be replaced with m.getDurability() ?
{code}
+ case SYNC_WAL:
+ case FSYNC_WAL:
+// sync the WAL edit (SYNC and FSYNC treated the same for now)
+this.log.sync(txid);
+break;
{code}
Minor: it would be easier to maintain if the above two conditions are
separated, with same this.log.sync(txid) call.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.94.6, 0.95.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.7, 0.95.1
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.96-full-v2.txt,
> 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt,
> 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt,
> 7801-0.96-v8.txt, 7801-0.96-v9.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630528#comment-13630528 ] Hudson commented on HBASE-7801: --- Integrated in HBase-TRUNK #4058 (See [https://builds.apache.org/job/HBase-TRUNK/4058/]) HBASE-7801 Allow a deferred sync option per Mutation. (Revision 1467357) Result = SUCCESS larsh : Files : * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Durability.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java * /hbase/trunk/hbase-protocol/src/main/protobuf/Client.proto * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableOutputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/BaseRowProcessor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiRowMutationProcessor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowProcessor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftUtilities.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Merge.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaMigrationConvertingToPB.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestCloneSnapshotFromClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultipleTimestamps.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTimestampsFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestAggregateProtocol.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestBigDecimalColumnInterpreter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverBypass.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630507#comment-13630507 ] Lars Hofhansl commented on HBASE-7801: -- Please have a look at the 0.94 patch ([~stack], [~te...@apache.org], [~enis], [~anoop.hbase]). I would like to commit this to 0.94.7... If you can't get to it or have concerns that's fine too, I'll move it to 0.94.8, or remove from 0.94. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.94-v4.txt, 7801-0.94-v5.txt, 7801-0.96-full-v2.txt, > 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, 7801-0.96-full-v5.txt, > 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13629586#comment-13629586 ] Lars Hofhansl commented on HBASE-7801: -- Enis and Anoop +1'd already (and I have only clarified the code and added a test since then). If there are no objections I will commit this tomorrow. I would also like to have this client side API in 0.94 (but without the rest of the intrusive changes). Thinking about how to do that is backward and binary compatible way now. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, > 7801-0.96-v7.txt, 7801-0.96-v8.txt, 7801-0.96-v9.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13629508#comment-13629508
]
Hadoop QA commented on HBASE-7801:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12578283/7801-0.96-v10.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 159
new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestFullLogReconstruction
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5275//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5275//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5275//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5275//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5275//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5275//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5275//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5275//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5275//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5275//console
This message is automatically generated.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.94.6, 0.95.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.7, 0.95.1
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt,
> 7801-0.96-full-v5.txt, 7801-0.96-v10.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt,
> 7801-0.96-v7.txt, 7801-0.96-v8.txt, 7801-0.96-v9.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628633#comment-13628633 ] Lars Hofhansl commented on HBASE-7801: -- I think I ran into the PB 2.5 vs 2.4.1 issue. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628631#comment-13628631 ] Lars Hofhansl commented on HBASE-7801: -- Need to rebase the patch again. Grrr. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628508#comment-13628508 ] Enis Soztutar commented on HBASE-7801: -- bq. I'm thinking about how I could test this. Would need to something like holding the async flush, and doing that on request in a test. If you plug in a mock hlog implementation, and do no-op in sync(), or do a latch there you might be able to do that. However, I suspect it can still get messy. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628494#comment-13628494 ] Lars Hofhansl commented on HBASE-7801: -- Expect a (hopefully) final patch soon. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627509#comment-13627509 ] Anoop Sam John commented on HBASE-7801: --- #3 - Oh yes.. Sorry I missed that.. Ya readability wise it may be better to track this way. Up to you Lars to change or not :) +1 on the patch > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627502#comment-13627502 ] Lars Hofhansl commented on HBASE-7801: -- Thanks Anoop. #1 yes #2 Oops, you're right. It's per table. Will fix. #3 That works, because later in doMiniBatchMutation we only call syncOrDefer if the WALEdit is not empty. But you are right, this can be made safer/easier-to-read if we track the durability even in that case. I'm thinking about how I could test this. Would need to something like holding the async flush, and doing that on request in a test. That way one write a bunch, check that it's not in the log, then run the async flush, check again. Maybe for later. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt, 7801-0.96-v9.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627477#comment-13627477
]
Anoop Sam John commented on HBASE-7801:
---
1.{code}
- byte [] row = delete.getRow();
+ delete.getRow();
{code}
We can avoid this line fully right? Good that u make some cleanup also. :)
2.{code}
// do what CF defaults to
+ if (!isDeferredLogSyncEnabled()) {
{code}
We can say do what table defaults to?
Even here also
{code}
+public enum Durability {
+ /**
+ * Use the column family's default setting to determine durability.
+ * This must remain the first option.
+ */
+ USE_DEFAULT,
{code}
3.As per code in syncOrDefer() when all the Mutations in the Mini batch says
for SKIP_WAL, we should not call the sync.
{code}
+ case SKIP_WAL:
+// nothing do to
+break;
{code}
But see the code in doMiniBatchMutation()
{code}
+ Durability durability = Durability.USE_DEFAULT;
...
+Durability tmpDur = m.getDurability();
+if (tmpDur == Durability.SKIP_WAL) {
if (m instanceof Put) {
recordPutWithoutWal(m.getFamilyMap());
}
continue;
+} else if (tmpDur.ordinal() > durability.ordinal()) {
+ durability = tmpDur;
}
{code}
This is correct for not considering one Mutation's SKIP_WAL to affect with the
USE_DEFAULT of others. But when all the Mutation say SKIP_WAL, still finally
the durability = Durability.USE_DEFAULT and will cause a log sync?
Excellent work Lars.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.94.6, 0.95.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.7, 0.95.1
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt,
> 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt,
> 7801-0.96-v8.txt, 7801-0.96-v9.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627292#comment-13627292
]
Enis Soztutar commented on HBASE-7801:
--
+1 from me, the API looks very good now.
Just a note, below is technically incompatible, but we do not care I guess at
this point. Actually using 6 is cleaner.
{code}
- optional bool writeToWAL = 6 [default = true];
+ optional Durability durability = 6 [default = USE_DEFAULT];
{code}
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.94.6, 0.95.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.7, 0.95.1
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt,
> 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt,
> 7801-0.96-v8.txt, 7801-0.96-v9.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627169#comment-13627169
]
Hadoop QA commented on HBASE-7801:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577889/7801-0.96-v9.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 156
new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5226//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5226//console
This message is automatically generated.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.94.6, 0.95.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.7, 0.95.1
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt,
> 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt,
> 7801-0.96-v8.txt, 7801-0.96-v9.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627046#comment-13627046 ] Lars Hofhansl commented on HBASE-7801: -- I did add the InterfaceAudience... I must have forgotten to save the file before I took the diff, will add again, sorry. (I hope there weren't more changes I didn't save). Re: hflush and hsync, can't point to Hadoop Javadoc I think as this changed over the releases (Syncable only has hflush and hsync since hadoop 2.0) > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13626882#comment-13626882 ] Enis Soztutar commented on HBASE-7801: -- Thanks Lars, this looks better. Could you please address these two as well: - Durability.java need InterfaceAudience annotations. - Should we add some pointers in SYNC_WAL, and FSYNC_WAL javadocs to hflush, hsync semantics? It might be confusing to first time comers. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt, > 7801-0.96-v8.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13626053#comment-13626053 ] Lars Hofhansl commented on HBASE-7801: -- Thanks [~enis]. Expect a new patch today. bq. Does it make sense to allow more room in between? If we use ordinals we should not need to reserve space. The PB enum items are never compared to each other, and for the client side enum only the relationship between the enum items matters, not their absolute positioning. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13625953#comment-13625953 ] Enis Soztutar commented on HBASE-7801: -- API looks good. A couple of comments: - There are still some places referring writeGuarantee, useWriteGuarantee, or wg. Could you please change those to be consistent. - We rely on Durability enum ordinals being in ascending sorted order in terms of their semantic guarantees. We should note this in the enum definitions. Does it make sense to allow more room in between? Just thinking out loud, fsync ack from majority of replicas, etc? - Durability.java need InterfaceAudience annotations. - Should we add some pointers in SYNC_WAL, and FSYNC_WAL javadocs to hflush, hsync semantics? It might be confusing to first time comers. Can do as a follow up. - Should BaseRowProcessor#useWriteGuarantee() return USE_DEFAULT? It is not used, but for documentation, and further reference. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13625134#comment-13625134 ] Anoop Sam John commented on HBASE-7801: --- [~lhofhansl] I will see the patch today. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13625133#comment-13625133 ] Lars Hofhansl commented on HBASE-7801: -- Any comments on the latest patch? It should give us all the functionality with maximum future flexibility. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.7, 0.95.1 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13624701#comment-13624701
]
Hadoop QA commented on HBASE-7801:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577426/7801-0.96-v7.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 156
new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5166//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5166//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5166//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5166//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5166//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5166//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5166//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5166//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5166//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5166//console
This message is automatically generated.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.95.1, 0.98.0, 0.94.7
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt,
> 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt, 7801-0.96-v7.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13624505#comment-13624505
]
Hadoop QA commented on HBASE-7801:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577359/7801-0.96-v6.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 156
new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.protobuf.TestProtobufUtil
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5163//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5163//console
This message is automatically generated.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.95.1, 0.98.0, 0.94.7
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt,
> 7801-0.96-full-v5.txt, 7801-0.96-v1.txt, 7801-0.96-v6.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13624315#comment-13624315 ] Lars Hofhansl commented on HBASE-7801: -- [~enis] Anoop had the same question above. I do think the column family setting should take precedence, unless we want rethink this completely. For example an option in the WriteGuarantee enum could be USE_DEFAULT, which would do whatever was setup for the CF. The CF should probably also have an option to avoid writing the WAL entries... But that's for another jira. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.1, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13608302#comment-13608302
]
Enis Soztutar commented on HBASE-7801:
--
Two more comments:
Should we rename WriteGuarantee -> Durability, as in ACID? WriteGuarantee is
good enough, but Durability is well understood. Alternatively
DurabilityGuarantee?
In,
{code}
+ private void syncOrDefer(long txid, boolean syncRequested) throws
IOException {
+if (this.getRegionInfo().isMetaRegion()
+|| (!this.htableDescriptor.isDeferredLogFlush() && syncRequested)
+|| this.deferredLogSyncDisabled) {
this.log.sync(txid);
}
{code}
The logic seems to be that deferred flush in HDT overrides WriteGuarantee in
Mutation. Is this intended? I think, semantically, it would be most user
friendly, if
- Durability setting in Mutation always takes precedence
- If no Durability setting in Mutation, the one in HTD takes into affect.
This allows us to do sync mutations to otherwise deferred column families, and
vice versa. This can be achieved, by defining a Durability setting in HTD
(getting rid of current skipwal and deferred stuff), and defaulting it to
SYNC_WAL. Mutation will have an optional Durability which defaults to null.
Wdyt?
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt,
> 7801-0.96-full-v5.txt, 7801-0.96-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13607558#comment-13607558 ] Anoop Sam John commented on HBASE-7801: --- bq.If DefferedLogFlush is enabled in HTD and my mutation says ASYNC_WAL it will go with defferedLogFlush right? Just for clarification. Yes When table is not having DefferedLogFlush set, if all the Mutations in a mini batch says for async flush, the flush will be deferred. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13607385#comment-13607385 ] ramkrishna.s.vasudevan commented on HBASE-7801: --- Patch looks fine to me. Did not check the testcases though. If DefferedLogFlush is enabled in HTD and my mutation says ASYNC_WAL it will go with defferedLogFlush right? Just for clarification. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13606615#comment-13606615 ] Enis Soztutar commented on HBASE-7801: -- bq. Not sure we want to leak the Hadoop nomenclature up here. We have to explain the semantics of hflush vs hsync somehow to the user. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13606602#comment-13606602 ] Lars Hofhansl commented on HBASE-7801: -- Yeah I was wondering about this too. Not sure we want to leak the Hadoop nomenclature up here. I'm not good at naming either. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13606581#comment-13606581
]
Enis Soztutar commented on HBASE-7801:
--
I think this is a good time to think about how are we going to add proper hsync
support. Are you thinking smt like:
{code}
public enum WriteGuarantee {
SKIP_WAL,
ASYNC_WAL,
SYNC_WAL,
FSYNC_WAL,
}
{code}
How about SKIP_WAL, DEFERRED_SYNC_WAL, HFLUSH_WAL, HSYNC_WAL. Just my 2 cents,
I am not so good at naming.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt,
> 7801-0.96-full-v5.txt, 7801-0.96-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13606569#comment-13606569 ] Enis Soztutar commented on HBASE-7801: -- bq. Are generated filed (like ClientProtos.java) excluded from the long-line check? bq. I logged HBASE-8129 for making long line detection smarter No, it is just a simple bash script. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13606120#comment-13606120 ] Anoop Sam John commented on HBASE-7801: --- Gone through the latest Trunk Patch. Seems good to me. +1 It is not cleanly applying (Conflict with Hregion.java etc..) > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13604499#comment-13604499 ] Ted Yu commented on HBASE-7801: --- I logged HBASE-8129 for making long line detection smarter > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13604498#comment-13604498 ] Lars Hofhansl commented on HBASE-7801: -- Might be a good thing to add anyway. The long lines left that I noticed were in ClientProtos.java. I'll double check the patch. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13604496#comment-13604496
]
Ted Yu commented on HBASE-7801:
---
I don't think so:
{code}
ll=`cat $PATCH_DIR/patch | grep "^+" | grep -v "^@@" | grep -v "^+++" | grep
-v "import" | wc -L`
MAX_LINE_LENGTH_PATCH=`expr $MAX_LINE_LENGTH + 1`
if [[ "$ll" -gt "$MAX_LINE_LENGTH_PATCH" ]]; then
{code}
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt,
> 7801-0.96-full-v5.txt, 7801-0.96-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13604493#comment-13604493 ] Lars Hofhansl commented on HBASE-7801: -- Are generated filed (like ClientProtos.java) excluded from the long-line check? > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-full-v5.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13604476#comment-13604476
]
Hadoop QA commented on HBASE-7801:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574030/7801-0.96-full-v5.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 153
new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4850//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4850//console
This message is automatically generated.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt,
> 7801-0.96-full-v5.txt, 7801-0.96-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13604297#comment-13604297 ] Lars Hofhansl commented on HBASE-7801: -- Of course. Will also look at the long line complaint. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13604159#comment-13604159 ] Ted Yu commented on HBASE-7801: --- Can you take a look at the java of warnings ? Thanks, Lars. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt, > 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13604139#comment-13604139
]
Hadoop QA commented on HBASE-7801:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12573996/7801-0.96-full-v4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 153
new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 6
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4847//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4847//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4847//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4847//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4847//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4847//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4847//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4847//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4847//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4847//console
This message is automatically generated.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-full-v4.txt,
> 7801-0.96-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13600378#comment-13600378
]
Hadoop QA commented on HBASE-7801:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12573393/7801-0.96-full-v3.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 153
new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 6
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.protobuf.TestProtobufUtil
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4781//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4781//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4781//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4781//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4781//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4781//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4781//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4781//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4781//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4781//console
This message is automatically generated.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.96-full-v2.txt, 7801-0.96-full-v3.txt, 7801-0.96-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13600347#comment-13600347
]
Lars Hofhansl commented on HBASE-7801:
--
I think {{p.setWriteToWAL(true)}} in TestFilter is a typo.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.96-full-v2.txt, 7801-0.96-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13600338#comment-13600338 ] Lars Hofhansl commented on HBASE-7801: -- Thanks Anoop. New patch coming soon. Please note that HRegion.delete as well as HRegion.put completely ignored the 2nd parameter, which is why I just removed it. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599979#comment-13599979 ] Anoop Sam John commented on HBASE-7801: --- If you are busy and need any help let me know Lars. I can help you :) > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599973#comment-13599973 ] Anoop Sam John commented on HBASE-7801: --- Change of writeToWAL -> writeTwriteGuarantee not handled in all the places? eg: Delete.java - Delete(final Delete d) > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599976#comment-13599976
]
Anoop Sam John commented on HBASE-7801:
---
1. HBaseTestCase
{code}
- this.region.delete(delete, writeToWAL);
+ this.region.delete(delete);
{code}
Do we need to set SKIP_WAL/SYNC_WAL based on writeToWAL into Delete object?
2.TestFilter
{code}
Delete d = new Delete(ROW);
d.deleteColumns(FAMILIES[0], QUALIFIERS_ONE[1]);
d.deleteColumns(FAMILIES[1], QUALIFIERS_ONE[1]);
- this.region.delete(d, false);
+ this.region.delete(d);
{code}
Do we need to set SKIP_WAL into Delete object?
{code}
@@ -217,7 +218,7 @@
// create new rows and column family to show how reseek works..
for (byte[] ROW : ROWS_THREE) {
Put p = new Put(ROW);
- p.setWriteToWAL(true);
+ p.setWriteGuarantee(WriteGuarantee.SKIP_WAL);
{code}
Not SKIP_WAL but SYNC_WAL.
3.TestProtobufUtil
{code}
-assertEquals(true, proto.getWriteToWAL());
+assertEquals(true, proto.getWriteGuarantee());
{code}
Assertion is wrong now. Test will fail. Other places also where assertEquals
related changes are there.
4.TestCompaction
{code}
- r.delete(new Delete(results.get(0).getRow()), false);
+ r.delete(new Delete(results.get(0).getRow()));
{code}
Do we need to set SKIP_WAL into Delete object?
5.TestGetClosestAtOrBefore
{code}
-mr.delete(new Delete(keys.get(0).getRow()), false);
+mr.delete(new Delete(keys.get(0).getRow()));
{code}
Do we need to set SKIP_WAL into Delete object? And other similar places in this
file
6.TestHRegion
{code}
-region.delete(delete, false);
+region.delete(delete);
{code}
Do we need to set SKIP_WAL into Delete object?
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.96-full-v2.txt, 7801-0.96-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599933#comment-13599933
]
Anoop Sam John commented on HBASE-7801:
---
Gone through the changes in src code
1. {code}
/**
+ * Write the Mutation of the WAL synchronously.
{code}
you mean Mutation to the WAL? In other places too
2. HRegion#increment(Increment increment)
{code}
-syncOrDefer(txid); // sync the transaction log outside the rowlock
+syncOrDefer(txid, true); // sync the transaction log outside the
rowlock
{code}
Hard coded true. Can check whether defered sync or not? Increment is a
Mutation.
Will see test code part now. Patch not cleanly applying on
TestRestoreFlushSnapshotFromClient.java
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt,
> 7801-0.96-full-v2.txt, 7801-0.96-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599864#comment-13599864 ] ramkrishna.s.vasudevan commented on HBASE-7801: --- Will check this later today Lars. By the time if Anoop has gone over then fine. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599768#comment-13599768 ] Anoop Sam John commented on HBASE-7801: --- [~lhofhansl] I am going through the patch. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599740#comment-13599740 ] Lars Hofhansl commented on HBASE-7801: -- Any comments? > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-full-v2.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13591967#comment-13591967 ] Lars Hofhansl commented on HBASE-7801: -- Sigh... I will try to have trunk patch tomorrow. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13588071#comment-13588071 ] Lars Hofhansl commented on HBASE-7801: -- new trunk patch with PB coming tomorrow. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.95.0, 0.94.6 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.95.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.94-v3.txt, > 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583896#comment-13583896 ] Lars Hofhansl commented on HBASE-7801: -- Yep that was an artifact of of my first attempt where I just or'd shouldSyncWal in each iterarion before I added the else if. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583711#comment-13583711
]
Sergey Shelukhin commented on HBASE-7801:
-
{code}
+} else if (!shouldSyncWal) {
+ shouldSyncWal |= !m.getDeferWALFlush();
{code}
|= is not necessary.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.6
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.96-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583498#comment-13583498
]
Enis Soztutar commented on HBASE-7801:
--
bq. As far the Java API in concerned the fact that we use PB is an
implementation detail and should not leak out.
My argument above is definitely longer term, when we decide to officially
support multi-language clients.
bq. Are you also suggesting changing the Mutation API? I.e. remove
{set|get}WriteToWAL and replace it with setWriteGuarantee(WriteGuarantee wg)?
Yes, we can do both. Add the enums in the java client API, and the PB
serialization. We can deprecate {set|get}WriteToWAL() in 0.94, and remove in
0.96.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.6
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.96-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13582925#comment-13582925
]
Lars Hofhansl commented on HBASE-7801:
--
As far the Java API in concerned the fact that we use PB is an implementation
detail and should not leak out.
Are you also suggesting changing the Mutation API? I.e. remove
{set|get}WriteToWAL and replace it with setWriteGuarantee(WriteGuarantee wg)?
Could do that in 0.94 as well (but still use Attributes to transmit the
information - and obviously we have to keep {set|get}WriteToWAL in 0.94)
Something like setWriteGuarantee(WriteGuarantee wg) is certainly nice, because
it let's us add more options later (like a true sync to disk).
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.6
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.96-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13582854#comment-13582854 ] Enis Soztutar commented on HBASE-7801: -- Makes sense. I think it is not a horrible idea to expose the PB's as client API's. If we want to have non-java clients (especially third party ones), then the java api is not our only client-facing API. Our RPC, and the PB's are client facing as well. Also, we might get rid of the client objects -> PB -> wire -> PB -> server objects flow, and go with client PB -> wire -> server PB -> wal/memstore. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13582811#comment-13582811 ] Lars Hofhansl commented on HBASE-7801: -- Oh another problem with PB is that we need to introduce two enums, because we do not want to leak the PB definition into the public API. So a public enum that is part of the API that is then translated into the PB enum. (Did I mention I do not like PB? :) ) > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13582810#comment-13582810 ] Lars Hofhansl commented on HBASE-7801: -- Was trying to avoid a PB change, but there's really no reason to. Maybe while I'm at it I might remove the cluster id attribute as well in favor of an attribute on the PB message. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13582788#comment-13582788
]
Enis Soztutar commented on HBASE-7801:
--
For 0.94, it looks good, but for trunk I think we can merge writeToWAL and
syncWAL into one enum. Smt like:
{code}
message Mutate {
required bytes row = 1;
enum WriteGuarantee {
SYNC_WAL = 0,
ASYNC_WAL = 1,
SKIP_WAL = 2
}
optional WriteGuarantee writeGuarantee = 6 [default = SYNC_WAL];
{code}
I am sure, we can think of a better name than WriteGuarantee though.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.6
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.96-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580575#comment-13580575 ] Anoop Sam John commented on HBASE-7801: --- +1 We can add in release notes about how to achieve the different use cases. bq. Increment is not supported as it does not extend Mutation (but it could be supported if we make an PB change). For 0.96 we have a JIRA issue for making Increment also a Mutation right? I forgot the issue id. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580390#comment-13580390
]
Ted Yu commented on HBASE-7801:
---
I looked at trunk patch. It looks good.
{code}
+ * @return true of the resulting edit should sync'ed to the WAL, false if
not.
{code}
Please correct the syntax of above javadoc.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.6
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.96-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580120#comment-13580120 ] Lars Hofhansl commented on HBASE-7801: -- Will fix the line length. As for this patch, any comments? Have been thinking about unittest, but this would tricky to verify (we do not have any test for deferred log flush, probably for the same reason). bq. In our case we didn't want the LogSyncer to be there at all Looking at the code, we could just not start the LogSyncer if hbase.regionserver.optionallogflushinterval is set to 0. I'd fear there might be other side effects, but it could just work. But that's for a different jira anyway. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580080#comment-13580080
]
Hadoop QA commented on HBASE-7801:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12569695/7801-0.96-v1.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4450//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4450//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4450//console
This message is automatically generated.
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.6
>
> Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.96-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13579746#comment-13579746 ] ramkrishna.s.vasudevan commented on HBASE-7801: --- @Lars We need this. As far as me and Anoop are concerned, have felt a need for such things. +1 on patch. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13579705#comment-13579705 ] Lars Hofhansl commented on HBASE-7801: -- Both #1 and #2 will be use cases, and both are possible with this. Generally, though, I do not see much interest in this from other folks. Is this not something people come across periodically? > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578926#comment-13578926 ] Anoop Sam John commented on HBASE-7801: --- [~lhofhansl] Thanks for sharing the use case. So we can say 1. When for a table most of the Mutations need WAL write flush immediately and very few can come with defer flush, we will not set the table defer flush but those Mutations to be defer flush 2. When for a table most of the Mutations can be defer flushed and few must be immediately flushed, we will not set table defer. On majority of the Mutations we will set defer flush attribute. Those which need immediate flush, we wont set the defer attribute. 3. When all the Mutations on a table can go with defer flush, set the table defer flush attribute and no other change is needed. Just saying as a conclusion. If most of the use case will be like #1, then I am fine Lars. :) BTW looked at the patch. Looks fine with me otherwise. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578472#comment-13578472 ] Lars Hofhansl commented on HBASE-7801: -- The use case that I see a lot at Saleforce is similar to the writeToWAL use case. By default the Mutation is written to the WAL unless the Mutation says not to. The defer case is the same IMHO. Doing the reverse without making it confusing would require a "forceFlush" (or something like this) option in the Mutation. So we'd have writeToWal, deferFlush, and forceFlush. Even then IMHO that would be more confusing the helpful. The key here is that it should possible to decide per Mutation whether to defer or not. Without this change that is not possible. With this change it is possible by leaving the table's default (flush) and then set the defer bit on a Mutation. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578216#comment-13578216 ] Anoop Sam John commented on HBASE-7801: --- bq.Not running the LogSyncer thread at all seems like a recipe for unexpected behavior. Or are you saying you to temporarily disable the syncer thread, while a bigger "transaction" is in progress? In our case we didn't want the LogSyncer to be there at all as all the Mutation to all tables need to be sync immediately. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578215#comment-13578215 ] Anoop Sam John commented on HBASE-7801: --- [~lhofhansl] What all use cases you see? I have one table for which I want a sync immediately for almost all Mutations. For certain others I can have defered flush. Can there be reverse case also? In the current implementation how we can achieve that. Sorry I dont have any clear use case with me for this. What do you say? > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578205#comment-13578205 ] ramkrishna.s.vasudevan commented on HBASE-7801: --- Yes right Lars. If the Log syncer is on, suppose thro the WALObserver we have done a WAL edit and in the main flow we do another WAlEdit but both needs to be synced together as atomic, then we need to the logsyncer to be disabled. Or else the logsyncer may sync up the first Edit that happened thro WALObserver and then when the main sync happens there could be a failure leading to inconsistency. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578196#comment-13578196 ] Lars Hofhansl commented on HBASE-7801: -- [~ram_krish] Not running the LogSyncer thread at all seems like a recipe for unexpected behavior. Or are you saying you to temporarily disable the syncer thread, while a bigger "transaction" is in progress? > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578193#comment-13578193 ] Lars Hofhansl commented on HBASE-7801: -- [~anoopsamjohn] I was thinking that if the table is set up with deferred flush then we'd honor that like we do now. Otherwise that option would be always ignored unless all Mutations are marked with deferFlush (something that an old client cannot even do). Note that a Mutation does not explicitly say it wants to sync, it only explicitly state when it doesn't. Another option to make this a bit nicer is to pass the shouldSyncWal flag to syncOrDefer and hence all the decision logic about whether to flush or not in that method. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578154#comment-13578154 ] ramkrishna.s.vasudevan commented on HBASE-7801: --- For Anoop's point on the LogSyncerThread, what we were thinking is to have a config in the RS level itself to say if to allow the LogSyncer thread to start or not. But not sure if this can fit in the kernel code. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[
https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578148#comment-13578148
]
Anoop Sam John commented on HBASE-7801:
---
{code}
+ if (walEdit.size() > 0 && shouldSyncWal) {
syncOrDefer(txid);
}
private void syncOrDefer(long txid) throws IOException {
if (this.regionInfo.isMetaRegion() ||
!this.htableDescriptor.isDeferredLogFlush()) {
this.log.sync(txid);
}
}
{code}
When Mutation specifically say to sync WAL immediately, we need to do so with
out considering what is set for Table? If so the syncOrDefer will again defer
it right?
> Allow a deferred sync option per Mutation.
> --
>
> Key: HBASE-7801
> URL: https://issues.apache.org/jira/browse/HBASE-7801
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.6
>
> Attachments: 7801-0.94-v1.txt
>
>
> Won't have time for parent. But a deferred sync option on a per operation
> basis comes up quite frequently.
> In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a
> special mutation attribute.
> For batch operation we'd take the safest sync option of any of the mutations.
> I.e. if there is at least one that wants to be flushed we'd sync the batch,
> if there's none of those but at least one that wants deferred flush we defer
> flush the batch, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578077#comment-13578077 ] Lars Hofhansl commented on HBASE-7801: -- I testing the patch locally. Works fine. Any comments? I could do the same for 0.96, or make a proper protobuf change there. (btw. for setting the cluster id in trunk we're still using attributes as opposed to encoding this in the protobuf message). It is actually an interesting question whether we should maintain optional Mutation attributes via attributes or via a protobuf change. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
