Re: hbase corruption - missing region files in HDFS

2012-12-09 Thread Kevin O'dell
Chris, Thank you for the very descriptive update. On Sun, Dec 9, 2012 at 6:29 PM, Chris Waterson wrote: > Well, I upgraded to 0.92.2, since the version I was running on (0.92.1) > didn't have those options for "hbck". > > That helped. > > It took me a while to realize that I had to make the roo

Re: hbase corruption - missing region files in HDFS

2012-12-09 Thread Chris Waterson
Well, I upgraded to 0.92.2, since the version I was running on (0.92.1) didn't have those options for "hbck". That helped. It took me a while to realize that I had to make the root filesystem writable so that "hbck -repair" could create itself a directory. So, once that was done, it at least

Re: hbase corruption - missing region files in HDFS

2012-12-09 Thread Kevin O'dell
can you run hbase hbck -fixMeta -fixAssignments This should assign those region servers and fix the hole. On Sat, Dec 8, 2012 at 11:30 PM, Chris Waterson wrote: > Hello! I've gotten myself into trouble where I'm missing files on HDFS > that HBase thinks ought to be there. In particular, runni

Re: Using HBaseAdmin.getAlterStatus

2012-12-09 Thread Harsh J
Yes, checked in-between or monitored in parallel. On Mon, Dec 10, 2012 at 2:32 AM, Mesika, Asaf wrote: > Yeah, I forgot to mention that important bit: > the pair returned is always 0 on the firstKey and never changes. > > So if I understand you correctly, I need to use get AlterStatus between the

Re: HBase Integration with Active Directory

2012-12-09 Thread Harsh J
Ah alright. To rephrase my answer: Authentication in HBase via AD is supported, but direct Authorization of tables via AD is not. You'd need to either come up with your own co-processors or enhance the AccessController to feed its ACL data off of LDAP instead of a system table (a pluggable design

Re: Checking if a coprocessor was loaded successfully from client

2012-12-09 Thread Ted Yu
On region server web UI, you should see the list of coprocessors loaded. But I guess you're looking for a programmatic way of detecting coprocessor deployment. Cheers On Sun, Dec 9, 2012 at 12:29 PM, Mesika, Asaf wrote: > 0.94.0 > On Dec 9, 2012, at 3:03 PM, yuzhih...@gmail.com wrote: > > > Wh

hbase corruption - missing region files in HDFS

2012-12-09 Thread Chris Waterson
Hello! I've gotten myself into trouble where I'm missing files on HDFS that HBase thinks ought to be there. In particular, running "hbase hbck" yields the below message: two regions are "not deployed on any region server" (because there is no file in HDFS for the region), and "there is a hole

Re: HBase Integration with Active Directory

2012-12-09 Thread anil gupta
Hi Harsh, HBase has a concept of ACL. But, these ACL's are maintained as another system table "*_acl_*"(similar to Meta and Root) in HBase. See: hbase.apache.org/book/hbase.accesscontrol.configuration.html. Instead of HBase maintaining these ACL's as a system table we want HBase to understand the

EHR storage system in HBase

2012-12-09 Thread Marcos Ortiz
Hi to everyone here. I'm working in a new product focused on Electronic Healthcare Records in HBase and it would be nice to discuss some design ideas with you. Have any of you done something seemed before? If you have some good schema design examples, it would be nice to use that knowledge. T

Re: Using HBaseAdmin.getAlterStatus

2012-12-09 Thread Mesika, Asaf
Yeah, I forgot to mention that important bit: the pair returned is always 0 on the firstKey and never changes. So if I understand you correctly, I need to use get AlterStatus between the modifyTable and enableTable ? On Dec 9, 2012, at 10:53 PM, Harsh J wrote: > Can you clarify on what "doesn't

Re: HBase Integration with Active Directory

2012-12-09 Thread Harsh J
Hi, Correct me if I'm wrong, but HBase presently has no reliance on the concept of groups, just users. For authenticating users, it relies on Hadoop Common's security libraries, which is the same as is used by HDFS for authentication. The Hadoop Common security libraries provided auth_to_local for

Re: Using HBaseAdmin.getAlterStatus

2012-12-09 Thread Harsh J
Can you clarify on what "doesn't work" a bit more? Do you get an exception or are you not receiving some data that you are expecting? AFAICT, this API is helpful for a parallel monitoring of an issued schema update. In your code bits, which I will assume is serial, you already disable-update-enab

Re: HBase Integration with Active Directory

2012-12-09 Thread anil gupta
Hi Harsh, We are in process of installing a HBase cluster with a secure HDFS and HBase. We already have a secure HDFS integrated with AD but we are still trying to figure out a way to integrate HBase with AD(directly or indirectly throgh KDC). I think my colleague has already implemented the stuff

Using HBaseAdmin.getAlterStatus

2012-12-09 Thread Mesika, Asaf
Hi, I've tried using HBaseAdmin.getAlterStatus to check on a HBaseAdmin.modifyTable command I've issued, but it doesn't work. Sample Code: Modifying Table tableDescriptor = admin.getTableDescriptor(tableNameBytes); if (tableDescriptor.hasCoprocessor(observerClass

Re: Checking if a coprocessor was loaded successfully from client

2012-12-09 Thread Mesika, Asaf
0.94.0 On Dec 9, 2012, at 3:03 PM, yuzhih...@gmail.com wrote: > Which hbase version are you targeting ? > > Thanks > > > > On Dec 9, 2012, at 2:55 AM, "Mesika, Asaf" wrote: > >> Hi, >> >> I wrote a custom Region Observer. >> I'm currently writing an Installer class for it. >> In this instal

Re: Checking if a coprocessor was loaded successfully from client

2012-12-09 Thread yuzhihong
Which hbase version are you targeting ? Thanks On Dec 9, 2012, at 2:55 AM, "Mesika, Asaf" wrote: > Hi, > > I wrote a custom Region Observer. > I'm currently writing an Installer class for it. > In this installer I'm adding the region observer by adding a coprocessor to > the HTableDescripto

Checking if a coprocessor was loaded successfully from client

2012-12-09 Thread Mesika, Asaf
Hi, I wrote a custom Region Observer. I'm currently writing an Installer class for it. In this installer I'm adding the region observer by adding a coprocessor to the HTableDescriptor, and then calling modifyTable by HBaseAdmin. My question is: How can I check whether region observer was loaded

RE: Please review my GC options, and give me some advice.

2012-12-09 Thread Jain Rahul
1) From these params young gc size is very low. Set the -Xmn with a higher memory let's say 1g. if your data is short lived than long-lived then you can try it making 2g or less. 2) Since -XX:SurvivorRatio is 8 here so both Survivors getting 1/8 of young gc (i.e allocated to -Xmn). Have them

Re: Meaure server time of Get/Scan - through RPC logging?

2012-12-09 Thread Asaf Mesika
The scan are done in parallel in many region servers and are specific to your query, so I don't think any jmx counter can help you. Maybe you measure it using your own Region Observer on Pre/Post Scan, and writing it to a shared log file on hdfs. Sent from my iPhone On 9 בדצמ 2012, at 04:41, Wei

Re: Heterogeneous cluster

2012-12-09 Thread Asaf Mesika
So just to get this right: the class you have built is a custom Load Balancer which replaces the default hbase load balancer implementation? Sent from my iPhone On 8 בדצמ 2012, at 05:33, Jean-Marc Spaggiari wrote: Hi, Here is the situation. I have an heterogeneous cluster with 2 cores CPUs, 4

Re: Heterogeneous cluster

2012-12-09 Thread Michael Segel
Ok... From a production/commercial grade answer... With respect to HBase, you will have 1 live copy and 2 replications. (Assuming you didn't change this.) So when you run against HBase, data locality becomes less of an issue. And again, you have to temper that with that it depends on the num