Re: Extensions for patches accepted by QA bot

2013-12-03 Thread Anoop John
+1 for .patch, .txt, .diff -Anoop- On Tue, Dec 3, 2013 at 6:16 AM, Nick Dimiduk ndimi...@gmail.com wrote: Sounds reasonable to me. Maybe add .diff to that list? On Mon, Dec 2, 2013 at 4:34 PM, Ted Yu yuzhih...@gmail.com wrote: During review of HBASE-10044 test-patch.sh should filter

Re: Extensions for patches accepted by QA bot

2013-12-03 Thread Ted Yu
Over in HBASE-10017, the following attachment triggered a QA warning: http://issues.apache.org/jira/secure/attachment/12616816/patchSiteOutput.txt Should the acceptance criteria include the notion that there should be JIRA number in the filename ? Thanks On Tue, Dec 3, 2013 at 12:39 AM, Anoop

[jira] [Resolved] (HBASE-9832) Add MR support for Visibility labels

2013-12-03 Thread ramkrishna.s.vasudevan (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-9832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan resolved HBASE-9832. --- Resolution: Fixed Hadoop Flags: Reviewed Thanks for the review Anoop.

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Enis Söztutar
Thanks Jon for bringing this to dev@. On Mon, Dec 2, 2013 at 10:01 PM, Jonathan Hsieh j...@cloudera.com wrote: Fundamentally, I'd prefer focusing on making HBase HBasier instead of tackling a feature that other systems architecturally can do better (inconsistent reads). I consider

Re: [Shadow Regions / Read Replicas ] Wal per region?

2013-12-03 Thread Devaraj Das
On Mon, Dec 2, 2013 at 10:20 PM, Jonathan Hsieh j...@cloudera.com wrote: Deveraj: Jonathan Hsieh, WAL per region (WALpr) would give you the locality (and hence HDFS short circuit) of reads if you were to couple it with the favored nodes. The cost is of course more WAL files... In the

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Devaraj Das
On Tue, Dec 3, 2013 at 11:07 AM, Enis Söztutar e...@apache.org wrote: Thanks Jon for bringing this to dev@. On Mon, Dec 2, 2013 at 10:01 PM, Jonathan Hsieh j...@cloudera.com wrote: Fundamentally, I'd prefer focusing on making HBase HBasier instead of tackling a feature that other systems

Re: [Shadow Regions / Read Replicas ] Block Affinity

2013-12-03 Thread Enis Söztutar
Responses inlined. On Mon, Dec 2, 2013 at 10:00 PM, Jonathan Hsieh j...@cloudera.com wrote: Enis: I was trying to refer to not having co-location constraints for secondary replicas whose primaries are hosted by the same RS. For example, if R1(replica=0), and R2(replica=0) are hosted on

Re: [Shadow Regions / Read Replicas ] Wal per region?

2013-12-03 Thread Enis Söztutar
On Mon, Dec 2, 2013 at 10:20 PM, Jonathan Hsieh j...@cloudera.com wrote: Deveraj: Jonathan Hsieh, WAL per region (WALpr) would give you the locality (and hence HDFS short circuit) of reads if you were to couple it with the favored nodes. The cost is of course more WAL files... In the

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Jonathan Hsieh
To keep the discussion focused on the design goals, I'm going start referring to enis and deveraj's eventually consistent read replicas as the *read replica* design, and consistent fast read recovery mechanism based on shadowing/tailing the wals as *shadow regions* or *shadow memstores*. Can we

[jira] [Created] (HBASE-10073) [Hadoop1]: hbase zkcli broken due to slf4j incompatibility

2013-12-03 Thread Aleksandr Shulman (JIRA)
Aleksandr Shulman created HBASE-10073: - Summary: [Hadoop1]: hbase zkcli broken due to slf4j incompatibility Key: HBASE-10073 URL: https://issues.apache.org/jira/browse/HBASE-10073 Project: HBase

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Vladimir Rodionov
The downside: - Double/Triple memstore usage - Increased block cache usage (effectively, block cache will have 50% capacity may be less) These downsides are pretty serious ones. This will result: 1. in decreased overall performance due to decreased efficient block cache size 2. In more

Helix and Hbase

2013-12-03 Thread kishore g
Hello, I have been wanting to write a proposal on using Apache Helix for cluster management in HBase but wanted to hear some thoughts/feedback on whether such an exercise would be useful. I understand that Hbase has built its own cluster management solution but integrating with Helix can provide

[jira] [Resolved] (HBASE-9840) Large scans and BlockCache evictions problems

2013-12-03 Thread Lars Hofhansl (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-9840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-9840. -- Resolution: Later Large scans and BlockCache evictions problems

[jira] [Resolved] (HBASE-9746) RegionServer can't start when replication tries to replicate to an unknown host

2013-12-03 Thread Lars Hofhansl (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-9746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-9746. -- Resolution: Invalid RegionServer can't start when replication tries to replicate to an

Re: [Shadow Regions / Read Replicas ] Wal per region?

2013-12-03 Thread Jonathan Hsieh
On Tue, Dec 3, 2013 at 11:21 AM, Devaraj Das d...@hortonworks.com wrote: On Mon, Dec 2, 2013 at 10:20 PM, Jonathan Hsieh j...@cloudera.com wrote: With this in mind, I actually I making the case that we would group the all the regions from RS-A onto the same set of preferred regions

Re: [Shadow Regions / Read Replicas ] Wal per region?

2013-12-03 Thread Jonathan Hsieh
On Tue, Dec 3, 2013 at 11:42 AM, Enis Söztutar enis@gmail.com wrote: On Mon, Dec 2, 2013 at 10:20 PM, Jonathan Hsieh j...@cloudera.com wrote: Deveraj: Jonathan Hsieh, WAL per region (WALpr) would give you the locality (and hence HDFS short circuit) of reads if you were to couple

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Enis Söztutar
On Tue, Dec 3, 2013 at 11:51 AM, Jonathan Hsieh j...@cloudera.com wrote: To keep the discussion focused on the design goals, I'm going start referring to enis and deveraj's eventually consistent read replicas as the *read replica* design, and consistent fast read recovery mechanism based on

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Devaraj Das
On Tue, Dec 3, 2013 at 12:31 PM, Vladimir Rodionov vladrodio...@gmail.comwrote: The downside: - Double/Triple memstore usage - Increased block cache usage (effectively, block cache will have 50% capacity may be less) These downsides are pretty serious ones. This will result: 1. in

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Enis Söztutar
On Tue, Dec 3, 2013 at 12:31 PM, Vladimir Rodionov vladrodio...@gmail.comwrote: The downside: - Double/Triple memstore usage - Increased block cache usage (effectively, block cache will have 50% capacity may be less) These are covered at the tradeoff section at the design doc. These

[jira] [Created] (HBASE-10074) consolidate and improve capacity/sizing documentation

2013-12-03 Thread Sergey Shelukhin (JIRA)
Sergey Shelukhin created HBASE-10074: Summary: consolidate and improve capacity/sizing documentation Key: HBASE-10074 URL: https://issues.apache.org/jira/browse/HBASE-10074 Project: HBase

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Vladimir Rodionov
MTTR and this work is ortagonal. In a distributed system, you cannot differentiate between a process not responding because it is down or it is busy or network is down, or whatnot. Having a couple of seconds detection time is unrealistic. You will end up in a very unstable state where you will be

Re: [Shadow Regions / Read Replicas ] Wal per region?

2013-12-03 Thread Enis Söztutar
On Tue, Dec 3, 2013 at 2:03 PM, Jonathan Hsieh j...@cloudera.com wrote: On Tue, Dec 3, 2013 at 11:42 AM, Enis Söztutar enis@gmail.com wrote: On Mon, Dec 2, 2013 at 10:20 PM, Jonathan Hsieh j...@cloudera.com wrote: Deveraj: Jonathan Hsieh, WAL per region (WALpr) would give you

Re: Branching 0.98

2013-12-03 Thread Andrew Purtell
As you've probably noticed I've been holding as some last minute details come in. However I am going to make the branch today. I will just do the svn copy some time in the afternoon here, so that will be overnight in the Pacific timezone. It would be tremendously helpful if someone with Jenkins

Re: [Shadow Regions / Read Replicas ] Block Affinity

2013-12-03 Thread Jonathan Hsieh
On Tue, Dec 3, 2013 at 11:37 AM, Enis Söztutar enis@gmail.com wrote: Responses inlined. On Mon, Dec 2, 2013 at 10:00 PM, Jonathan Hsieh j...@cloudera.com wrote: For the most efficient consistent read-recovery (shadow regions/memstores), it would make sense to have them assigned to

Re: Extensions for patches accepted by QA bot

2013-12-03 Thread Nick Dimiduk
I don't think that's a necessary filter; it'll still get false-positives like HBASE-1234_design_v1.txt. I'd prefer the automation to be kind and understanding in accepting and evaluating community contributions :) Running failing to apply a non-patch doesn't really hurt anything. On Tue, Dec 3,

Re: Extensions for patches accepted by QA bot

2013-12-03 Thread Sergey Shelukhin
Hive QA has a more strict filter, which can also be used to test patches against specific branch: Upload a patch to a JIRA in the following format HIVE-.patch, HIVE-.XX.patch, HIVE-.D.XX.patch (phabricator), or HIVE--branch.patch (allows targeting a branch).

[jira] [Created] (HBASE-10075) add a locality-aware balancer

2013-12-03 Thread Chengxiang Li (JIRA)
Chengxiang Li created HBASE-10075: - Summary: add a locality-aware balancer Key: HBASE-10075 URL: https://issues.apache.org/jira/browse/HBASE-10075 Project: HBase Issue Type: New Feature

Re: [Shadow Regions / Read Replicas ] Wal per region?

2013-12-03 Thread Jonathan Hsieh
On Tue, Dec 3, 2013 at 3:07 PM, Enis Söztutar enis@gmail.com wrote: On Tue, Dec 3, 2013 at 2:03 PM, Jonathan Hsieh j...@cloudera.com wrote: On Tue, Dec 3, 2013 at 11:42 AM, Enis Söztutar enis@gmail.com wrote: On Mon, Dec 2, 2013 at 10:20 PM, Jonathan Hsieh j...@cloudera.com

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Jonathan Hsieh
On Tue, Dec 3, 2013 at 2:48 PM, Vladimir Rodionov vladrodio...@gmail.comwrote: MTTR and this work is ortagonal. In a distributed system, you cannot differentiate between a process not responding because it is down or it is busy or network is down, or whatnot. Having a couple of seconds

Re: Extensions for patches accepted by QA bot

2013-12-03 Thread Jonathan Hsieh
I like the idea of being able to target patches at a branch. +1 if someone does it. :) On Tue, Dec 3, 2013 at 5:10 PM, Sergey Shelukhin ser...@hortonworks.comwrote: Hive QA has a more strict filter, which can also be used to test patches against specific branch: Upload a patch to a JIRA in

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Jonathan Hsieh
On Tue, Dec 3, 2013 at 2:04 PM, Enis Söztutar enis@gmail.com wrote: On Tue, Dec 3, 2013 at 11:51 AM, Jonathan Hsieh j...@cloudera.com wrote: On Tue, Dec 3, 2013 at 11:07 AM, Enis Söztutar e...@apache.org wrote: Thanks Jon for bringing this to dev@. On Mon, Dec 2, 2013 at

Re: [Shadow Regions / Read Replicas ] External replication disqualified?

2013-12-03 Thread Jonathan Hsieh
The read replicas doc mentions something a little more intrusive in the 3 options section but doesn't seem to disqualify it. Relatedly just as another strawman, for the mostly read only use case and bulk load only usecases, why not use normal replication against two clusters in the same HDFS /

Re: Branching 0.98

2013-12-03 Thread Stack
On Wed, Dec 4, 2013 at 8:08 AM, Andrew Purtell apurt...@apache.org wrote: ...It would be tremendously helpful if someone with Jenkins karma could help set up jobs for the new branch once that's done. I can do this Andrew (you should have same Karma as I but I've done this before...) St.Ack

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Jimmy Xiang
A separate branch similar to that for snapshot is great. +1. For wal tailing, we can just skip those edits not for the shadow regions, right? To tail the wal, we need to wait till the wal block is available. There seems to be a hard latency. Is it better to have a pool of daemon threads to ship

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Jimmy Xiang
We don't have to ship the edits one by one. We can use a configurable batch to control the impact on network. On Tue, Dec 3, 2013 at 7:59 PM, Jimmy Xiang jxi...@cloudera.com wrote: A separate branch similar to that for snapshot is great. +1. For wal tailing, we can just skip those edits not

Re: Branching 0.98

2013-12-03 Thread lars hofhansl
You beat me to it, Stack. :) From: Stack st...@duboce.net To: HBase Dev List dev@hbase.apache.org Sent: Tuesday, December 3, 2013 7:12 PM Subject: Re: Branching 0.98 On Wed, Dec 4, 2013 at 8:08 AM, Andrew Purtell apurt...@apache.org wrote: ...It would

[jira] [Resolved] (HBASE-9986) Incorporate HTTPS support for HBase (0.94 port)

2013-12-03 Thread Lars Hofhansl (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-9986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-9986. -- Resolution: Fixed Committed to 0.94. Thanks for the patch Aditya. Incorporate HTTPS support

[jira] [Resolved] (HBASE-9680) 0.94 backport: HBASE-7404 Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE

2013-12-03 Thread Lars Hofhansl (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-9680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-9680. -- Resolution: Later Fix Version/s: (was: 0.94.15) I don't see this going into 0.94.

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Devaraj Das
On Tue, Dec 3, 2013 at 6:47 PM, Jonathan Hsieh j...@cloudera.com wrote: On Tue, Dec 3, 2013 at 2:04 PM, Enis Söztutar enis@gmail.com wrote: On Tue, Dec 3, 2013 at 11:51 AM, Jonathan Hsieh j...@cloudera.com wrote: On Tue, Dec 3, 2013 at 11:07 AM, Enis Söztutar e...@apache.org

Re: [Shadow Regions / Read Replicas ] External replication disqualified?

2013-12-03 Thread Devaraj Das
On Tue, Dec 3, 2013 at 6:49 PM, Jonathan Hsieh j...@cloudera.com wrote: The read replicas doc mentions something a little more intrusive in the 3 options section but doesn't seem to disqualify it. I don't quite see what you are referring to actually... Can you please copy-paste a relevant