Re: consistency, availability and partition pattern of HBase
Correct. You are limited to the throughput of a single region server while interacting with a particular region. This throughput limitation is typically handled by designing your keys such that your data is distributed well across the cluster. Having multiple region servers serve a single region gets you into the land of maintaining consistency across copies, which is challenging. It might be doable but that's not the design choice Bigtable (and hence HBase) made initially. On Thu, Aug 9, 2012 at 11:04 AM, Lin Ma wrote: > Thanks > > "only a single RegionServer ever hosts a region at once" -- I know HDFS > have multiple copies for the same file. Is region server works in > active-passive way, i.e. even if there are multiple copies, only one region > server could serve? If so, will it be bottleneck, supposing the traffic to > that region is too high? > > regards, > Lin > > On Thu, Aug 9, 2012 at 11:09 AM, Bryan Beaudreault < > bbeaudrea...@hubspot.com > > wrote: > > > Actual data backing hbase is replicated, but that is handled by HDFS. > Yes, > > if you lose an hdfs datanode, clients (in this case the client is hbase) > > move to the next node in the pipeline. > > > > However, only a single RegionServer ever hosts a region at once. If the > > RegionServer dies, there is a period where the master must notice the > > regions are unhosted and move them to other regionservers. During that > > period, data is inaccessible or modifiable. > > > > On Wed, Aug 8, 2012 at 10:32 PM, Lin Ma wrote: > > > > > Thank you Lars. > > > > > > Is the same data store duplicated copy across region server? If so, if > > one > > > primary server for the region dies, client just need to read from the > > > secondary server for the same region. Why there is data is unavailable > > > time? > > > > > > BTW: please feel free to correct me for any wrong knowledge about > HBase. > > > > > > regards, > > > Lin > > > > > > On Thu, Aug 9, 2012 at 9:31 AM, lars hofhansl > > wrote: > > > > > > > After a write completes the next read (regardless of the location it > is > > > > issued from) will see the latest value. > > > > This is because at any given time exactly RegionServer is responsible > > for > > > > a specific Key > > > > (through assignment of key ranges to regions and regions to > > > RegionServers). > > > > > > > > > > > > As Mohit said, the trade off is that data is unavailable if a > > > RegionServer > > > > dies until another RegionServer picks up the regions (and by > extension > > > the > > > > key range) > > > > > > > > -- Lars > > > > > > > > > > > > - Original Message - > > > > From: Lin Ma > > > > To: user@hbase.apache.org > > > > Cc: > > > > Sent: Wednesday, August 8, 2012 8:47 AM > > > > Subject: Re: consistency, availability and partition pattern of HBase > > > > > > > > And consistency is not sacrificed? i.e. all distributed clients' > update > > > > will results in sequential / real time update? Once update is done by > > one > > > > client, all other client could see results immediately? > > > > > > > > regards, > > > > Lin > > > > > > > > On Wed, Aug 8, 2012 at 11:17 PM, Mohit Anchlia < > mohitanch...@gmail.com > > > > >wrote: > > > > > > > > > I think availability is sacrificed in the sense that if region > server > > > > > fails clients will have data inaccessible for the time region comes > > up > > > on > > > > > some other server, not to confuse with data loss. > > > > > > > > > > Sent from my iPad > > > > > > > > > > On Aug 7, 2012, at 11:56 PM, Lin Ma wrote: > > > > > > > > > > > Thank you Wei! > > > > > > > > > > > > Two more comments, > > > > > > > > > > > > 1. How about Hadoop's CAP characters do you think about? > > > > > > 2. For your comments, if HBase implements "per key sequential > > > > > consistency", > > > > > > what are the missing characters for consistency? Cross-key update > > > > > > sequences? Could you show me an example about what you think are > > > > missed? > > > > > > thanks. > > > > > > > > > > > > regards, > > > > > > Lin > > > > > > > > > > > > On Wed, Aug 8, 2012 at 12:18 PM, Wei Tan > wrote: > > > > > > > > > > > >> Hi Lin, > > > > > >> > > > > > >> In the CAP theorem > > > > > >> Consistency stands for atomic consistency, i.e., each CRUD > > operation > > > > > >> occurs sequentially in a global, real-time clock > > > > > >> Availability means each server if not partitioned can accept > > > requests > > > > > >> > > > > > >> Partition means network partition > > > > > >> > > > > > >> As far as I understand (although I do not see any official > > > > > documentation), > > > > > >> HBase achieved "per key sequential consistency", i.e., for a > > > specific > > > > > key, > > > > > >> there is an agreed sequence, for all operations on it. This is > > > weaker > > > > > than > > > > > >> strong or sequential consistency, but stronger than "eventual > > > > > >> consistency". > > > > > >> > > > > > >> BTW: CAP was proposed by Prof. Eric Brewer... > > > > > >> http://
Re: Encryption in HBase
Hello Michael, I got the point. Thank you so much. Regards, Mohammad Tariq On Thu, Aug 9, 2012 at 5:12 AM, Michael Segel wrote: > Hi, > > First you have to ask yourself what's the purpose of the encryption. > Usually encryption is part of a bigger security implementation, so I don't > think you'll gain much by encrypting at this level of granularity. > > Looking at trying to encrypt at the HFile level > > Unless I'm missing something, your HFIle will span multiple HDFS blocks. Each > HDFS block is actually a file, right? > So what happens when you fetch a row from a file in a simple get()? > > You can see that you have a lot of work to do. > > Compare that to encrypting the column data itself. > Storing and encrypted byte array vs an un-encrypted byte array isn't much > different. > > > Does that make sense? > > > > > On Aug 8, 2012, at 2:47 PM, Mohammad Tariq wrote: > >> Hello Michael, >> >> Thank you for the valuable suggestion. Could you please provide >> me the the drawbacks of it?(Just out of curiosity). >> Regards, >>Mohammad Tariq >> >> > > >> On Wed, Aug 8, 2012 at 8:01 PM, Michael Segel >> wrote: >>> You don't want to do that. >>> I mean you really don't want to do that. ;-) >>> >>> You would be better off doing a strong encryption at the cell level. You >>> can use co-processors to do that if you'd like. >>> >>> YMMV >>> >>> On Aug 8, 2012, at 9:17 AM, Mohammad Tariq wrote: >>> Hello Stack, Would it be feasible to encrypt the HFile itself?? Regards, Mohammad Tariq On Wed, Aug 8, 2012 at 6:53 PM, Jean-Marc Spaggiari wrote: > Will it be possible to use the compression methods to call an > encryption method instead? > > 2012/8/8, Farrokh Shahriari : >> Thanks >> How can I encrypt a cell ? Is there a special way for encryption ? >> >> On Wed, Aug 8, 2012 at 3:24 PM, Stack wrote: >> >>> On Tue, Aug 7, 2012 at 12:06 PM, Farrokh Shahriari >>> wrote: Hi How can I enable encryption on hbase ? if I want to use encryption in a cluster with hbase running on top of hadoop,what can I do? Can't hadoop be configured to automatically encrypt each file which is going to be written on it? >>> >>> HBase does not provide this facility. You'd have to encrypt cell >>> content yourself and then put it into HBase. >>> >>> St.Ack >>> >> >>> >> >
Re: consistency, availability and partition pattern of HBase
HDFS also chooses to degrade availability in the face of partitions. On Thu, Aug 9, 2012 at 11:08 AM, Lin Ma wrote: > Amandeep, thanks for your comments, and I will definitely read the paper > you suggested. > > For Hadoop itself, what do you think its CAP features? Which one of the > CAP is sacrificed? > > regards, > Lin > > On Thu, Aug 9, 2012 at 1:34 PM, Amandeep Khurana wrote: > >> Firstly, I recommend you read the GFS and Bigtable papers. That'll give >> you >> a good understanding of the architecture. Adhoc question on the mailing >> list won't. >> >> I'll try to answer some of your questions briefly. Think of HBase as a >> database layer over an underlying filesystem (the same way MySQL is over >> ext2/3/4 etc). The filesystem for HBase in this case is HDFS. HDFS >> replicates data for redundancy and fault tolerance. HBase has region >> servers that serve the regions. Regions form tables. Region servers >> persist >> their data on HDFS. Now, every region is served by one and only one region >> server. So, HBase is not replicating anything. Replication is handled at >> the storage layer. If a region server goes down, all its regions now need >> to be served by some other region server. During this period of region >> assignment, the clients experience degraded availability if they try to >> interact with any of those regions. >> >> Coming back to CAP. HBase chooses to degrade availability in the face of >> partitions. "Partition" is a very general term here and does not >> necessarily mean network partitions. Any node falling off the HBase >> cluster >> can be considered to be a partition. So, when failures happen, HBase >> degrades availability but does not give up consistency. Consistency in >> this >> context is sort of the equivalent of atomicity in ACID. In the context of >> HBase, any copy of data that is written to HBase will be visible to all >> clients. There is no concept of multiple different versions that the >> clients need to reconcile between. When you read, you always get the same >> version of the row you are reading. In other words, HBase is strongly >> consistent. >> >> Hope that clears things up a bit. >> >> On Thu, Aug 9, 2012 at 8:02 AM, Lin Ma wrote: >> >> > Thank you Lars. >> > >> > Is the same data store duplicated copy across region server? If so, if >> one >> > primary server for the region dies, client just need to read from the >> > secondary server for the same region. Why there is data is unavailable >> > time? >> > >> > BTW: please feel free to correct me for any wrong knowledge about HBase. >> > >> > regards, >> > Lin >> > >> > On Thu, Aug 9, 2012 at 9:31 AM, lars hofhansl >> wrote: >> > >> > > After a write completes the next read (regardless of the location it >> is >> > > issued from) will see the latest value. >> > > This is because at any given time exactly RegionServer is responsible >> for >> > > a specific Key >> > > (through assignment of key ranges to regions and regions to >> > RegionServers). >> > > >> > > >> > > As Mohit said, the trade off is that data is unavailable if a >> > RegionServer >> > > dies until another RegionServer picks up the regions (and by extension >> > the >> > > key range) >> > > >> > > -- Lars >> > > >> > > >> > > - Original Message - >> > > From: Lin Ma >> > > To: user@hbase.apache.org >> > > Cc: >> > > Sent: Wednesday, August 8, 2012 8:47 AM >> > > Subject: Re: consistency, availability and partition pattern of HBase >> > > >> > > And consistency is not sacrificed? i.e. all distributed clients' >> update >> > > will results in sequential / real time update? Once update is done by >> one >> > > client, all other client could see results immediately? >> > > >> > > regards, >> > > Lin >> > > >> > > On Wed, Aug 8, 2012 at 11:17 PM, Mohit Anchlia < >> mohitanch...@gmail.com >> > > >wrote: >> > > >> > > > I think availability is sacrificed in the sense that if region >> server >> > > > fails clients will have data inaccessible for the time region comes >> up >> > on >> > > > some other server, not to confuse with data loss. >> > > > >> > > > Sent from my iPad >> > > > >> > > > On Aug 7, 2012, at 11:56 PM, Lin Ma wrote: >> > > > >> > > > > Thank you Wei! >> > > > > >> > > > > Two more comments, >> > > > > >> > > > > 1. How about Hadoop's CAP characters do you think about? >> > > > > 2. For your comments, if HBase implements "per key sequential >> > > > consistency", >> > > > > what are the missing characters for consistency? Cross-key update >> > > > > sequences? Could you show me an example about what you think are >> > > missed? >> > > > > thanks. >> > > > > >> > > > > regards, >> > > > > Lin >> > > > > >> > > > > On Wed, Aug 8, 2012 at 12:18 PM, Wei Tan wrote: >> > > > > >> > > > >> Hi Lin, >> > > > >> >> > > > >> In the CAP theorem >> > > > >> Consistency stands for atomic consistency, i.e., each CRUD >> operation >> > > > >> occurs sequentially in a global, real-time clock >> > > > >> Availability means each server i
Re: consistency, availability and partition pattern of HBase
Amandeep, thanks for your comments, and I will definitely read the paper you suggested. For Hadoop itself, what do you think its CAP features? Which one of the CAP is sacrificed? regards, Lin On Thu, Aug 9, 2012 at 1:34 PM, Amandeep Khurana wrote: > Firstly, I recommend you read the GFS and Bigtable papers. That'll give you > a good understanding of the architecture. Adhoc question on the mailing > list won't. > > I'll try to answer some of your questions briefly. Think of HBase as a > database layer over an underlying filesystem (the same way MySQL is over > ext2/3/4 etc). The filesystem for HBase in this case is HDFS. HDFS > replicates data for redundancy and fault tolerance. HBase has region > servers that serve the regions. Regions form tables. Region servers persist > their data on HDFS. Now, every region is served by one and only one region > server. So, HBase is not replicating anything. Replication is handled at > the storage layer. If a region server goes down, all its regions now need > to be served by some other region server. During this period of region > assignment, the clients experience degraded availability if they try to > interact with any of those regions. > > Coming back to CAP. HBase chooses to degrade availability in the face of > partitions. "Partition" is a very general term here and does not > necessarily mean network partitions. Any node falling off the HBase cluster > can be considered to be a partition. So, when failures happen, HBase > degrades availability but does not give up consistency. Consistency in this > context is sort of the equivalent of atomicity in ACID. In the context of > HBase, any copy of data that is written to HBase will be visible to all > clients. There is no concept of multiple different versions that the > clients need to reconcile between. When you read, you always get the same > version of the row you are reading. In other words, HBase is strongly > consistent. > > Hope that clears things up a bit. > > On Thu, Aug 9, 2012 at 8:02 AM, Lin Ma wrote: > > > Thank you Lars. > > > > Is the same data store duplicated copy across region server? If so, if > one > > primary server for the region dies, client just need to read from the > > secondary server for the same region. Why there is data is unavailable > > time? > > > > BTW: please feel free to correct me for any wrong knowledge about HBase. > > > > regards, > > Lin > > > > On Thu, Aug 9, 2012 at 9:31 AM, lars hofhansl > wrote: > > > > > After a write completes the next read (regardless of the location it is > > > issued from) will see the latest value. > > > This is because at any given time exactly RegionServer is responsible > for > > > a specific Key > > > (through assignment of key ranges to regions and regions to > > RegionServers). > > > > > > > > > As Mohit said, the trade off is that data is unavailable if a > > RegionServer > > > dies until another RegionServer picks up the regions (and by extension > > the > > > key range) > > > > > > -- Lars > > > > > > > > > - Original Message - > > > From: Lin Ma > > > To: user@hbase.apache.org > > > Cc: > > > Sent: Wednesday, August 8, 2012 8:47 AM > > > Subject: Re: consistency, availability and partition pattern of HBase > > > > > > And consistency is not sacrificed? i.e. all distributed clients' update > > > will results in sequential / real time update? Once update is done by > one > > > client, all other client could see results immediately? > > > > > > regards, > > > Lin > > > > > > On Wed, Aug 8, 2012 at 11:17 PM, Mohit Anchlia > > >wrote: > > > > > > > I think availability is sacrificed in the sense that if region server > > > > fails clients will have data inaccessible for the time region comes > up > > on > > > > some other server, not to confuse with data loss. > > > > > > > > Sent from my iPad > > > > > > > > On Aug 7, 2012, at 11:56 PM, Lin Ma wrote: > > > > > > > > > Thank you Wei! > > > > > > > > > > Two more comments, > > > > > > > > > > 1. How about Hadoop's CAP characters do you think about? > > > > > 2. For your comments, if HBase implements "per key sequential > > > > consistency", > > > > > what are the missing characters for consistency? Cross-key update > > > > > sequences? Could you show me an example about what you think are > > > missed? > > > > > thanks. > > > > > > > > > > regards, > > > > > Lin > > > > > > > > > > On Wed, Aug 8, 2012 at 12:18 PM, Wei Tan wrote: > > > > > > > > > >> Hi Lin, > > > > >> > > > > >> In the CAP theorem > > > > >> Consistency stands for atomic consistency, i.e., each CRUD > operation > > > > >> occurs sequentially in a global, real-time clock > > > > >> Availability means each server if not partitioned can accept > > requests > > > > >> > > > > >> Partition means network partition > > > > >> > > > > >> As far as I understand (although I do not see any official > > > > documentation), > > > > >> HBase achieved "per key sequential consistency", i.e., for a > > specific > > > > key, > >
Re: consistency, availability and partition pattern of HBase
Firstly, I recommend you read the GFS and Bigtable papers. That'll give you a good understanding of the architecture. Adhoc question on the mailing list won't. I'll try to answer some of your questions briefly. Think of HBase as a database layer over an underlying filesystem (the same way MySQL is over ext2/3/4 etc). The filesystem for HBase in this case is HDFS. HDFS replicates data for redundancy and fault tolerance. HBase has region servers that serve the regions. Regions form tables. Region servers persist their data on HDFS. Now, every region is served by one and only one region server. So, HBase is not replicating anything. Replication is handled at the storage layer. If a region server goes down, all its regions now need to be served by some other region server. During this period of region assignment, the clients experience degraded availability if they try to interact with any of those regions. Coming back to CAP. HBase chooses to degrade availability in the face of partitions. "Partition" is a very general term here and does not necessarily mean network partitions. Any node falling off the HBase cluster can be considered to be a partition. So, when failures happen, HBase degrades availability but does not give up consistency. Consistency in this context is sort of the equivalent of atomicity in ACID. In the context of HBase, any copy of data that is written to HBase will be visible to all clients. There is no concept of multiple different versions that the clients need to reconcile between. When you read, you always get the same version of the row you are reading. In other words, HBase is strongly consistent. Hope that clears things up a bit. On Thu, Aug 9, 2012 at 8:02 AM, Lin Ma wrote: > Thank you Lars. > > Is the same data store duplicated copy across region server? If so, if one > primary server for the region dies, client just need to read from the > secondary server for the same region. Why there is data is unavailable > time? > > BTW: please feel free to correct me for any wrong knowledge about HBase. > > regards, > Lin > > On Thu, Aug 9, 2012 at 9:31 AM, lars hofhansl wrote: > > > After a write completes the next read (regardless of the location it is > > issued from) will see the latest value. > > This is because at any given time exactly RegionServer is responsible for > > a specific Key > > (through assignment of key ranges to regions and regions to > RegionServers). > > > > > > As Mohit said, the trade off is that data is unavailable if a > RegionServer > > dies until another RegionServer picks up the regions (and by extension > the > > key range) > > > > -- Lars > > > > > > - Original Message - > > From: Lin Ma > > To: user@hbase.apache.org > > Cc: > > Sent: Wednesday, August 8, 2012 8:47 AM > > Subject: Re: consistency, availability and partition pattern of HBase > > > > And consistency is not sacrificed? i.e. all distributed clients' update > > will results in sequential / real time update? Once update is done by one > > client, all other client could see results immediately? > > > > regards, > > Lin > > > > On Wed, Aug 8, 2012 at 11:17 PM, Mohit Anchlia > >wrote: > > > > > I think availability is sacrificed in the sense that if region server > > > fails clients will have data inaccessible for the time region comes up > on > > > some other server, not to confuse with data loss. > > > > > > Sent from my iPad > > > > > > On Aug 7, 2012, at 11:56 PM, Lin Ma wrote: > > > > > > > Thank you Wei! > > > > > > > > Two more comments, > > > > > > > > 1. How about Hadoop's CAP characters do you think about? > > > > 2. For your comments, if HBase implements "per key sequential > > > consistency", > > > > what are the missing characters for consistency? Cross-key update > > > > sequences? Could you show me an example about what you think are > > missed? > > > > thanks. > > > > > > > > regards, > > > > Lin > > > > > > > > On Wed, Aug 8, 2012 at 12:18 PM, Wei Tan wrote: > > > > > > > >> Hi Lin, > > > >> > > > >> In the CAP theorem > > > >> Consistency stands for atomic consistency, i.e., each CRUD operation > > > >> occurs sequentially in a global, real-time clock > > > >> Availability means each server if not partitioned can accept > requests > > > >> > > > >> Partition means network partition > > > >> > > > >> As far as I understand (although I do not see any official > > > documentation), > > > >> HBase achieved "per key sequential consistency", i.e., for a > specific > > > key, > > > >> there is an agreed sequence, for all operations on it. This is > weaker > > > than > > > >> strong or sequential consistency, but stronger than "eventual > > > >> consistency". > > > >> > > > >> BTW: CAP was proposed by Prof. Eric Brewer... > > > >> http://en.wikipedia.org/wiki/Eric_Brewer_%28scientist%29 > > > >> > > > >> Best Regards, > > > >> Wei > > > >> > > > >> Wei Tan > > > >> Research Staff Member > > > >> IBM T. J. Watson Research Center > > > >> 19 Skyline Dr, Hawthorne, NY 10532 >
Re: consistency, availability and partition pattern of HBase
Thanks "only a single RegionServer ever hosts a region at once" -- I know HDFS have multiple copies for the same file. Is region server works in active-passive way, i.e. even if there are multiple copies, only one region server could serve? If so, will it be bottleneck, supposing the traffic to that region is too high? regards, Lin On Thu, Aug 9, 2012 at 11:09 AM, Bryan Beaudreault wrote: > Actual data backing hbase is replicated, but that is handled by HDFS. Yes, > if you lose an hdfs datanode, clients (in this case the client is hbase) > move to the next node in the pipeline. > > However, only a single RegionServer ever hosts a region at once. If the > RegionServer dies, there is a period where the master must notice the > regions are unhosted and move them to other regionservers. During that > period, data is inaccessible or modifiable. > > On Wed, Aug 8, 2012 at 10:32 PM, Lin Ma wrote: > > > Thank you Lars. > > > > Is the same data store duplicated copy across region server? If so, if > one > > primary server for the region dies, client just need to read from the > > secondary server for the same region. Why there is data is unavailable > > time? > > > > BTW: please feel free to correct me for any wrong knowledge about HBase. > > > > regards, > > Lin > > > > On Thu, Aug 9, 2012 at 9:31 AM, lars hofhansl > wrote: > > > > > After a write completes the next read (regardless of the location it is > > > issued from) will see the latest value. > > > This is because at any given time exactly RegionServer is responsible > for > > > a specific Key > > > (through assignment of key ranges to regions and regions to > > RegionServers). > > > > > > > > > As Mohit said, the trade off is that data is unavailable if a > > RegionServer > > > dies until another RegionServer picks up the regions (and by extension > > the > > > key range) > > > > > > -- Lars > > > > > > > > > - Original Message - > > > From: Lin Ma > > > To: user@hbase.apache.org > > > Cc: > > > Sent: Wednesday, August 8, 2012 8:47 AM > > > Subject: Re: consistency, availability and partition pattern of HBase > > > > > > And consistency is not sacrificed? i.e. all distributed clients' update > > > will results in sequential / real time update? Once update is done by > one > > > client, all other client could see results immediately? > > > > > > regards, > > > Lin > > > > > > On Wed, Aug 8, 2012 at 11:17 PM, Mohit Anchlia > > >wrote: > > > > > > > I think availability is sacrificed in the sense that if region server > > > > fails clients will have data inaccessible for the time region comes > up > > on > > > > some other server, not to confuse with data loss. > > > > > > > > Sent from my iPad > > > > > > > > On Aug 7, 2012, at 11:56 PM, Lin Ma wrote: > > > > > > > > > Thank you Wei! > > > > > > > > > > Two more comments, > > > > > > > > > > 1. How about Hadoop's CAP characters do you think about? > > > > > 2. For your comments, if HBase implements "per key sequential > > > > consistency", > > > > > what are the missing characters for consistency? Cross-key update > > > > > sequences? Could you show me an example about what you think are > > > missed? > > > > > thanks. > > > > > > > > > > regards, > > > > > Lin > > > > > > > > > > On Wed, Aug 8, 2012 at 12:18 PM, Wei Tan wrote: > > > > > > > > > >> Hi Lin, > > > > >> > > > > >> In the CAP theorem > > > > >> Consistency stands for atomic consistency, i.e., each CRUD > operation > > > > >> occurs sequentially in a global, real-time clock > > > > >> Availability means each server if not partitioned can accept > > requests > > > > >> > > > > >> Partition means network partition > > > > >> > > > > >> As far as I understand (although I do not see any official > > > > documentation), > > > > >> HBase achieved "per key sequential consistency", i.e., for a > > specific > > > > key, > > > > >> there is an agreed sequence, for all operations on it. This is > > weaker > > > > than > > > > >> strong or sequential consistency, but stronger than "eventual > > > > >> consistency". > > > > >> > > > > >> BTW: CAP was proposed by Prof. Eric Brewer... > > > > >> http://en.wikipedia.org/wiki/Eric_Brewer_%28scientist%29 > > > > >> > > > > >> Best Regards, > > > > >> Wei > > > > >> > > > > >> Wei Tan > > > > >> Research Staff Member > > > > >> IBM T. J. Watson Research Center > > > > >> 19 Skyline Dr, Hawthorne, NY 10532 > > > > >> w...@us.ibm.com; 914-784-6752 > > > > >> > > > > >> > > > > >> > > > > >> From: Lin Ma > > > > >> To:user@hbase.apache.org, > > > > >> Date: 08/07/2012 09:30 PM > > > > >> Subject:consistency, availability and partition pattern of > > > HBase > > > > >> > > > > >> > > > > >> > > > > >> Hello guys, > > > > >> > > > > >> According to the notes by Werner*, "*He presented the CAP theorem, > > > which > > > > >> states that of three properties of shared-data systems—data > > > consistency, > > > > >> system availability, and tolerance to network partition—only two > ca
Re: consistency, availability and partition pattern of HBase
On Wed, Aug 8, 2012 at 7:32 PM, Lin Ma wrote: > Thank you Lars. > > Is the same data store duplicated copy across region server? If so, if one > primary server for the region dies, client just need to read from the > secondary server for the same region. Why there is data is unavailable > time? > > To get better understanding of this I suggest looking at how the WAL logs are stored. WAL stores multiple regions in one log. Before region is alive on other region server master needs to split the logs so that it can replayed by the region server. This process causes downtime with respect to the region which is being replayed using edit logs. > BTW: please feel free to correct me for any wrong knowledge about HBase. > > regards, > Lin > > On Thu, Aug 9, 2012 at 9:31 AM, lars hofhansl wrote: > > > After a write completes the next read (regardless of the location it is > > issued from) will see the latest value. > > This is because at any given time exactly RegionServer is responsible for > > a specific Key > > (through assignment of key ranges to regions and regions to > RegionServers). > > > > > > As Mohit said, the trade off is that data is unavailable if a > RegionServer > > dies until another RegionServer picks up the regions (and by extension > the > > key range) > > > > -- Lars > > > > > > - Original Message - > > From: Lin Ma > > To: user@hbase.apache.org > > Cc: > > Sent: Wednesday, August 8, 2012 8:47 AM > > Subject: Re: consistency, availability and partition pattern of HBase > > > > And consistency is not sacrificed? i.e. all distributed clients' update > > will results in sequential / real time update? Once update is done by one > > client, all other client could see results immediately? > > > > regards, > > Lin > > > > On Wed, Aug 8, 2012 at 11:17 PM, Mohit Anchlia > >wrote: > > > > > I think availability is sacrificed in the sense that if region server > > > fails clients will have data inaccessible for the time region comes up > on > > > some other server, not to confuse with data loss. > > > > > > Sent from my iPad > > > > > > On Aug 7, 2012, at 11:56 PM, Lin Ma wrote: > > > > > > > Thank you Wei! > > > > > > > > Two more comments, > > > > > > > > 1. How about Hadoop's CAP characters do you think about? > > > > 2. For your comments, if HBase implements "per key sequential > > > consistency", > > > > what are the missing characters for consistency? Cross-key update > > > > sequences? Could you show me an example about what you think are > > missed? > > > > thanks. > > > > > > > > regards, > > > > Lin > > > > > > > > On Wed, Aug 8, 2012 at 12:18 PM, Wei Tan wrote: > > > > > > > >> Hi Lin, > > > >> > > > >> In the CAP theorem > > > >> Consistency stands for atomic consistency, i.e., each CRUD operation > > > >> occurs sequentially in a global, real-time clock > > > >> Availability means each server if not partitioned can accept > requests > > > >> > > > >> Partition means network partition > > > >> > > > >> As far as I understand (although I do not see any official > > > documentation), > > > >> HBase achieved "per key sequential consistency", i.e., for a > specific > > > key, > > > >> there is an agreed sequence, for all operations on it. This is > weaker > > > than > > > >> strong or sequential consistency, but stronger than "eventual > > > >> consistency". > > > >> > > > >> BTW: CAP was proposed by Prof. Eric Brewer... > > > >> http://en.wikipedia.org/wiki/Eric_Brewer_%28scientist%29 > > > >> > > > >> Best Regards, > > > >> Wei > > > >> > > > >> Wei Tan > > > >> Research Staff Member > > > >> IBM T. J. Watson Research Center > > > >> 19 Skyline Dr, Hawthorne, NY 10532 > > > >> w...@us.ibm.com; 914-784-6752 > > > >> > > > >> > > > >> > > > >> From: Lin Ma > > > >> To:user@hbase.apache.org, > > > >> Date: 08/07/2012 09:30 PM > > > >> Subject:consistency, availability and partition pattern of > > HBase > > > >> > > > >> > > > >> > > > >> Hello guys, > > > >> > > > >> According to the notes by Werner*, "*He presented the CAP theorem, > > which > > > >> states that of three properties of shared-data systems—data > > consistency, > > > >> system availability, and tolerance to network partition—only two can > > be > > > >> achieved at any given time." => > > > >> > > http://www.allthingsdistributed.com/2008/12/eventually_consistent.html > > > >> > > > >> But it seems HBase could achieve all of the 3 features at the same > > time. > > > >> Does it mean HBase breaks the rule by Werner. :-) > > > >> > > > >> If not, which one is sacrificed -- consistency (by using HDFS), > > > >> availability (by using Zookeeper) or partition (by using region / > > column > > > >> family) ? And why? > > > >> > > > >> regards, > > > >> Lin > > > >> > > > >> > > > >> > > > > > > > >
Hbase on EC2- what hardware and architecture works best?
Hi, we are running a 9 node Hadoop/Hbase cluster on EC2. We expect the number of nodes to grow multi fold in a couple of months. Of these 9 instances, three are m1.medium and they act as Hadoop Namenode, Hbase master and the secondary namenode. The remaining 6 nodes are all mi.large instances where we run datanodes,tasktrackers and the regionservers. The large instances have 7.5 GB of total memory. There are couple of issues that we are facing, and we feel that this architecture is not going to scale for us. 1) The uptime. We have been struggling to keep the cluster up and running. Sometimes the region-servers fail due to overload. Sometimes the GC pause forces the zookeeper to declare them dead. Sometime the regionserver get killed due to OutOfMemory. The map reduce always brings atleast 2-3 region-servers down. 2) We have 4G memory allocated for Region servers and we run data nodes, task trackers on them as well. We have observed that our region servers are vulnerable when we run MR jobs in our cluster. Is this some sign of competing resources (Memory) on the region servers or is this(having RS and data nodes/task trackers) generally not advisable? 3) So far our data size is close to 3TB(including replication) split over 6 region servers. An example of regionserver stat- numberOfOnlineRegions=231, numberOfStores=2055, numberOfStorefiles=1180, storefileIndexSizeMB=15, rootIndexSizeKB=16256. We would like to understand what is an ideal load for a region server - both in terms of rows and the actual data that it can serve? This is because, I would like understand if we are over pounding our system which it is not expected to handle. 4). We have always observed that, our Region Servers go down usually just after a long GC pause(DURATION), because, it prevented the RSs from acking to ZK for the session maintenance, or, it is usually a OOM. 5)The cost. The large instances and the 3 factor replication are expensive. We are not sure if horizontally scaling with large instances is the way to go. It'd be very useful if we can understand some already known limitations of the system, so that, we wont end up spending time in the wrong direction. Though we have been able to fix a lot of issues when they happen, we are looking for a more stable architecture. We feel that the cluster setup we have is wrong and we need confirmation and suggestions from the community. we will really appreciate your suggestions/pointers and any ideas would be very helpful. Also, we are hoping you can just tell us about your architecture which is not giving you nightmares. regards, Prem
Re: consistency, availability and partition pattern of HBase
What Bryan said :) - Original Message - From: Bryan Beaudreault To: user@hbase.apache.org Cc: lars hofhansl Sent: Wednesday, August 8, 2012 8:09 PM Subject: Re: consistency, availability and partition pattern of HBase Actual data backing hbase is replicated, but that is handled by HDFS. Yes, if you lose an hdfs datanode, clients (in this case the client is hbase) move to the next node in the pipeline. However, only a single RegionServer ever hosts a region at once. If the RegionServer dies, there is a period where the master must notice the regions are unhosted and move them to other regionservers. During that period, data is inaccessible or modifiable. On Wed, Aug 8, 2012 at 10:32 PM, Lin Ma wrote: > Thank you Lars. > > Is the same data store duplicated copy across region server? If so, if one > primary server for the region dies, client just need to read from the > secondary server for the same region. Why there is data is unavailable > time? > > BTW: please feel free to correct me for any wrong knowledge about HBase. > > regards, > Lin > > On Thu, Aug 9, 2012 at 9:31 AM, lars hofhansl wrote: > > > After a write completes the next read (regardless of the location it is > > issued from) will see the latest value. > > This is because at any given time exactly RegionServer is responsible for > > a specific Key > > (through assignment of key ranges to regions and regions to > RegionServers). > > > > > > As Mohit said, the trade off is that data is unavailable if a > RegionServer > > dies until another RegionServer picks up the regions (and by extension > the > > key range) > > > > -- Lars > > > > > > - Original Message - > > From: Lin Ma > > To: user@hbase.apache.org > > Cc: > > Sent: Wednesday, August 8, 2012 8:47 AM > > Subject: Re: consistency, availability and partition pattern of HBase > > > > And consistency is not sacrificed? i.e. all distributed clients' update > > will results in sequential / real time update? Once update is done by one > > client, all other client could see results immediately? > > > > regards, > > Lin > > > > On Wed, Aug 8, 2012 at 11:17 PM, Mohit Anchlia > >wrote: > > > > > I think availability is sacrificed in the sense that if region server > > > fails clients will have data inaccessible for the time region comes up > on > > > some other server, not to confuse with data loss. > > > > > > Sent from my iPad > > > > > > On Aug 7, 2012, at 11:56 PM, Lin Ma wrote: > > > > > > > Thank you Wei! > > > > > > > > Two more comments, > > > > > > > > 1. How about Hadoop's CAP characters do you think about? > > > > 2. For your comments, if HBase implements "per key sequential > > > consistency", > > > > what are the missing characters for consistency? Cross-key update > > > > sequences? Could you show me an example about what you think are > > missed? > > > > thanks. > > > > > > > > regards, > > > > Lin > > > > > > > > On Wed, Aug 8, 2012 at 12:18 PM, Wei Tan wrote: > > > > > > > >> Hi Lin, > > > >> > > > >> In the CAP theorem > > > >> Consistency stands for atomic consistency, i.e., each CRUD operation > > > >> occurs sequentially in a global, real-time clock > > > >> Availability means each server if not partitioned can accept > requests > > > >> > > > >> Partition means network partition > > > >> > > > >> As far as I understand (although I do not see any official > > > documentation), > > > >> HBase achieved "per key sequential consistency", i.e., for a > specific > > > key, > > > >> there is an agreed sequence, for all operations on it. This is > weaker > > > than > > > >> strong or sequential consistency, but stronger than "eventual > > > >> consistency". > > > >> > > > >> BTW: CAP was proposed by Prof. Eric Brewer... > > > >> http://en.wikipedia.org/wiki/Eric_Brewer_%28scientist%29 > > > >> > > > >> Best Regards, > > > >> Wei > > > >> > > > >> Wei Tan > > > >> Research Staff Member > > > >> IBM T. J. Watson Research Center > > > >> 19 Skyline Dr, Hawthorne, NY 10532 > > > >> w...@us.ibm.com; 914-784-6752 > > > >> > > > >> > > > >> > > > >> From: Lin Ma > > > >> To: user@hbase.apache.org, > > > >> Date: 08/07/2012 09:30 PM > > > >> Subject: consistency, availability and partition pattern of > > HBase > > > >> > > > >> > > > >> > > > >> Hello guys, > > > >> > > > >> According to the notes by Werner*, "*He presented the CAP theorem, > > which > > > >> states that of three properties of shared-data systems—data > > consistency, > > > >> system availability, and tolerance to network partition—only two can > > be > > > >> achieved at any given time." => > > > >> > > http://www.allthingsdistributed.com/2008/12/eventually_consistent.html > > > >> > > > >> But it seems HBase could achieve all of the 3 features at the same > > time. > > > >> Does it mean HBase breaks the rule by Werner. :-) > > > >> > > > >> If not, which one is sacrificed -- consistency (by using HDFS), > > > >> availability (by using Zookeeper) or partition (by using region / >
Re: consistency, availability and partition pattern of HBase
Actual data backing hbase is replicated, but that is handled by HDFS. Yes, if you lose an hdfs datanode, clients (in this case the client is hbase) move to the next node in the pipeline. However, only a single RegionServer ever hosts a region at once. If the RegionServer dies, there is a period where the master must notice the regions are unhosted and move them to other regionservers. During that period, data is inaccessible or modifiable. On Wed, Aug 8, 2012 at 10:32 PM, Lin Ma wrote: > Thank you Lars. > > Is the same data store duplicated copy across region server? If so, if one > primary server for the region dies, client just need to read from the > secondary server for the same region. Why there is data is unavailable > time? > > BTW: please feel free to correct me for any wrong knowledge about HBase. > > regards, > Lin > > On Thu, Aug 9, 2012 at 9:31 AM, lars hofhansl wrote: > > > After a write completes the next read (regardless of the location it is > > issued from) will see the latest value. > > This is because at any given time exactly RegionServer is responsible for > > a specific Key > > (through assignment of key ranges to regions and regions to > RegionServers). > > > > > > As Mohit said, the trade off is that data is unavailable if a > RegionServer > > dies until another RegionServer picks up the regions (and by extension > the > > key range) > > > > -- Lars > > > > > > - Original Message - > > From: Lin Ma > > To: user@hbase.apache.org > > Cc: > > Sent: Wednesday, August 8, 2012 8:47 AM > > Subject: Re: consistency, availability and partition pattern of HBase > > > > And consistency is not sacrificed? i.e. all distributed clients' update > > will results in sequential / real time update? Once update is done by one > > client, all other client could see results immediately? > > > > regards, > > Lin > > > > On Wed, Aug 8, 2012 at 11:17 PM, Mohit Anchlia > >wrote: > > > > > I think availability is sacrificed in the sense that if region server > > > fails clients will have data inaccessible for the time region comes up > on > > > some other server, not to confuse with data loss. > > > > > > Sent from my iPad > > > > > > On Aug 7, 2012, at 11:56 PM, Lin Ma wrote: > > > > > > > Thank you Wei! > > > > > > > > Two more comments, > > > > > > > > 1. How about Hadoop's CAP characters do you think about? > > > > 2. For your comments, if HBase implements "per key sequential > > > consistency", > > > > what are the missing characters for consistency? Cross-key update > > > > sequences? Could you show me an example about what you think are > > missed? > > > > thanks. > > > > > > > > regards, > > > > Lin > > > > > > > > On Wed, Aug 8, 2012 at 12:18 PM, Wei Tan wrote: > > > > > > > >> Hi Lin, > > > >> > > > >> In the CAP theorem > > > >> Consistency stands for atomic consistency, i.e., each CRUD operation > > > >> occurs sequentially in a global, real-time clock > > > >> Availability means each server if not partitioned can accept > requests > > > >> > > > >> Partition means network partition > > > >> > > > >> As far as I understand (although I do not see any official > > > documentation), > > > >> HBase achieved "per key sequential consistency", i.e., for a > specific > > > key, > > > >> there is an agreed sequence, for all operations on it. This is > weaker > > > than > > > >> strong or sequential consistency, but stronger than "eventual > > > >> consistency". > > > >> > > > >> BTW: CAP was proposed by Prof. Eric Brewer... > > > >> http://en.wikipedia.org/wiki/Eric_Brewer_%28scientist%29 > > > >> > > > >> Best Regards, > > > >> Wei > > > >> > > > >> Wei Tan > > > >> Research Staff Member > > > >> IBM T. J. Watson Research Center > > > >> 19 Skyline Dr, Hawthorne, NY 10532 > > > >> w...@us.ibm.com; 914-784-6752 > > > >> > > > >> > > > >> > > > >> From: Lin Ma > > > >> To:user@hbase.apache.org, > > > >> Date: 08/07/2012 09:30 PM > > > >> Subject:consistency, availability and partition pattern of > > HBase > > > >> > > > >> > > > >> > > > >> Hello guys, > > > >> > > > >> According to the notes by Werner*, "*He presented the CAP theorem, > > which > > > >> states that of three properties of shared-data systems—data > > consistency, > > > >> system availability, and tolerance to network partition—only two can > > be > > > >> achieved at any given time." => > > > >> > > http://www.allthingsdistributed.com/2008/12/eventually_consistent.html > > > >> > > > >> But it seems HBase could achieve all of the 3 features at the same > > time. > > > >> Does it mean HBase breaks the rule by Werner. :-) > > > >> > > > >> If not, which one is sacrificed -- consistency (by using HDFS), > > > >> availability (by using Zookeeper) or partition (by using region / > > column > > > >> family) ? And why? > > > >> > > > >> regards, > > > >> Lin > > > >> > > > >> > > > >> > > > > > > > >
Re: consistency, availability and partition pattern of HBase
Thank you Lars. Is the same data store duplicated copy across region server? If so, if one primary server for the region dies, client just need to read from the secondary server for the same region. Why there is data is unavailable time? BTW: please feel free to correct me for any wrong knowledge about HBase. regards, Lin On Thu, Aug 9, 2012 at 9:31 AM, lars hofhansl wrote: > After a write completes the next read (regardless of the location it is > issued from) will see the latest value. > This is because at any given time exactly RegionServer is responsible for > a specific Key > (through assignment of key ranges to regions and regions to RegionServers). > > > As Mohit said, the trade off is that data is unavailable if a RegionServer > dies until another RegionServer picks up the regions (and by extension the > key range) > > -- Lars > > > - Original Message - > From: Lin Ma > To: user@hbase.apache.org > Cc: > Sent: Wednesday, August 8, 2012 8:47 AM > Subject: Re: consistency, availability and partition pattern of HBase > > And consistency is not sacrificed? i.e. all distributed clients' update > will results in sequential / real time update? Once update is done by one > client, all other client could see results immediately? > > regards, > Lin > > On Wed, Aug 8, 2012 at 11:17 PM, Mohit Anchlia >wrote: > > > I think availability is sacrificed in the sense that if region server > > fails clients will have data inaccessible for the time region comes up on > > some other server, not to confuse with data loss. > > > > Sent from my iPad > > > > On Aug 7, 2012, at 11:56 PM, Lin Ma wrote: > > > > > Thank you Wei! > > > > > > Two more comments, > > > > > > 1. How about Hadoop's CAP characters do you think about? > > > 2. For your comments, if HBase implements "per key sequential > > consistency", > > > what are the missing characters for consistency? Cross-key update > > > sequences? Could you show me an example about what you think are > missed? > > > thanks. > > > > > > regards, > > > Lin > > > > > > On Wed, Aug 8, 2012 at 12:18 PM, Wei Tan wrote: > > > > > >> Hi Lin, > > >> > > >> In the CAP theorem > > >> Consistency stands for atomic consistency, i.e., each CRUD operation > > >> occurs sequentially in a global, real-time clock > > >> Availability means each server if not partitioned can accept requests > > >> > > >> Partition means network partition > > >> > > >> As far as I understand (although I do not see any official > > documentation), > > >> HBase achieved "per key sequential consistency", i.e., for a specific > > key, > > >> there is an agreed sequence, for all operations on it. This is weaker > > than > > >> strong or sequential consistency, but stronger than "eventual > > >> consistency". > > >> > > >> BTW: CAP was proposed by Prof. Eric Brewer... > > >> http://en.wikipedia.org/wiki/Eric_Brewer_%28scientist%29 > > >> > > >> Best Regards, > > >> Wei > > >> > > >> Wei Tan > > >> Research Staff Member > > >> IBM T. J. Watson Research Center > > >> 19 Skyline Dr, Hawthorne, NY 10532 > > >> w...@us.ibm.com; 914-784-6752 > > >> > > >> > > >> > > >> From: Lin Ma > > >> To:user@hbase.apache.org, > > >> Date: 08/07/2012 09:30 PM > > >> Subject:consistency, availability and partition pattern of > HBase > > >> > > >> > > >> > > >> Hello guys, > > >> > > >> According to the notes by Werner*, "*He presented the CAP theorem, > which > > >> states that of three properties of shared-data systems—data > consistency, > > >> system availability, and tolerance to network partition—only two can > be > > >> achieved at any given time." => > > >> > http://www.allthingsdistributed.com/2008/12/eventually_consistent.html > > >> > > >> But it seems HBase could achieve all of the 3 features at the same > time. > > >> Does it mean HBase breaks the rule by Werner. :-) > > >> > > >> If not, which one is sacrificed -- consistency (by using HDFS), > > >> availability (by using Zookeeper) or partition (by using region / > column > > >> family) ? And why? > > >> > > >> regards, > > >> Lin > > >> > > >> > > >> > > > >
Re: consistency, availability and partition pattern of HBase
After a write completes the next read (regardless of the location it is issued from) will see the latest value. This is because at any given time exactly RegionServer is responsible for a specific Key (through assignment of key ranges to regions and regions to RegionServers). As Mohit said, the trade off is that data is unavailable if a RegionServer dies until another RegionServer picks up the regions (and by extension the key range) -- Lars - Original Message - From: Lin Ma To: user@hbase.apache.org Cc: Sent: Wednesday, August 8, 2012 8:47 AM Subject: Re: consistency, availability and partition pattern of HBase And consistency is not sacrificed? i.e. all distributed clients' update will results in sequential / real time update? Once update is done by one client, all other client could see results immediately? regards, Lin On Wed, Aug 8, 2012 at 11:17 PM, Mohit Anchlia wrote: > I think availability is sacrificed in the sense that if region server > fails clients will have data inaccessible for the time region comes up on > some other server, not to confuse with data loss. > > Sent from my iPad > > On Aug 7, 2012, at 11:56 PM, Lin Ma wrote: > > > Thank you Wei! > > > > Two more comments, > > > > 1. How about Hadoop's CAP characters do you think about? > > 2. For your comments, if HBase implements "per key sequential > consistency", > > what are the missing characters for consistency? Cross-key update > > sequences? Could you show me an example about what you think are missed? > > thanks. > > > > regards, > > Lin > > > > On Wed, Aug 8, 2012 at 12:18 PM, Wei Tan wrote: > > > >> Hi Lin, > >> > >> In the CAP theorem > >> Consistency stands for atomic consistency, i.e., each CRUD operation > >> occurs sequentially in a global, real-time clock > >> Availability means each server if not partitioned can accept requests > >> > >> Partition means network partition > >> > >> As far as I understand (although I do not see any official > documentation), > >> HBase achieved "per key sequential consistency", i.e., for a specific > key, > >> there is an agreed sequence, for all operations on it. This is weaker > than > >> strong or sequential consistency, but stronger than "eventual > >> consistency". > >> > >> BTW: CAP was proposed by Prof. Eric Brewer... > >> http://en.wikipedia.org/wiki/Eric_Brewer_%28scientist%29 > >> > >> Best Regards, > >> Wei > >> > >> Wei Tan > >> Research Staff Member > >> IBM T. J. Watson Research Center > >> 19 Skyline Dr, Hawthorne, NY 10532 > >> w...@us.ibm.com; 914-784-6752 > >> > >> > >> > >> From: Lin Ma > >> To: user@hbase.apache.org, > >> Date: 08/07/2012 09:30 PM > >> Subject: consistency, availability and partition pattern of HBase > >> > >> > >> > >> Hello guys, > >> > >> According to the notes by Werner*, "*He presented the CAP theorem, which > >> states that of three properties of shared-data systems—data consistency, > >> system availability, and tolerance to network partition—only two can be > >> achieved at any given time." => > >> http://www.allthingsdistributed.com/2008/12/eventually_consistent.html > >> > >> But it seems HBase could achieve all of the 3 features at the same time. > >> Does it mean HBase breaks the rule by Werner. :-) > >> > >> If not, which one is sacrificed -- consistency (by using HDFS), > >> availability (by using Zookeeper) or partition (by using region / column > >> family) ? And why? > >> > >> regards, > >> Lin > >> > >> > >> >
Multiple scan input split for MR job
Hi everyone, I've been searching for a way to specify an MR job on an HBase table using multiple key ranges (instead of just one), and as far as I can tell, the best way is still to create a custom InputFormat like MultiSegmentTableInputFormat and override getSplits to return splits based on multiple scan objects. Is this still the best way to do this or is there any official support yet? If it is still the best way to do it, does anyone have an implementation of this that they'd be willing to share? I'm new to HBase and I'm not so sure I'd be able to do that well myself. Thank you for your time!
Re: Ideal row size
That's the exactly sort of target I was looking for -- thanks for the help! I'll probably shoot for something close to 48KB so I don't exceed that block size. On Tue, Aug 7, 2012 at 2:26 PM, Jean-Daniel Cryans wrote: > Hi Eric, > > An ideal cell size would probably be the size of a block, so 64KB > including the keys. Having bigger cells would inflate the size of your > blocks but then you'd be outside of the normal HBase settings. > > That, and do some experiments. > > J-D > > On Tue, Aug 7, 2012 at 6:35 AM, Eric Czech wrote: >> Hello everyone, >> >> I'm trying to store many small values in indexes created via MR jobs, >> and I was hoping to get some advice on how to structure my rows. >> Essentially, I have complete control over how large the rows should be >> as the values are small, consistent in size, and can be grouped >> together in any way I'd like. My question then is, what's the ideal >> size for a row in Hbase, in bytes? I'm trying to determine how to >> group my values together into larger values, and I think having a >> target size to hit would make that a lot easier. >> >> I know fewer rows is generally better to avoid the repetitive storage >> of keys, cfs, and qualifiers provided that those rows still suit a >> given application, but I'm not sure at what point the scale will tip >> in the other direction and I'll start to see undue memory pressure or >> compaction issues with rows that are too large. >> >> Thanks in advance!
setAutoFlush(false) in Thrift
Hi,
Hbase thrift doesn't expose setAutoFlush interface. I ran thrift out of the
Hbase cluster. I want to have a client side write buffering on the Thrift
server.
So I change the ThriftServer.java, and add a setAutoFlush(false) as follows:
if (!tables.containsKey(table)) {
boolean autoFlush = conf.getBoolean("hbase.client.autoflush", true);
HTable htable = new HTable(conf, tableName);
htable.setAutoFlush(autoFlush);
LOG.debug("Thrift setAutoFlush:"+autoFlush);
tables.put(table, htable);
}
The write buffer actually works when I call mutateRow(in PHP) with small data
(less than 500 bytes), write latency is only 1 ms. (php and thrift are in the
same machine)
But when I use larger payload (2KB and more), the mutateRow takes constant 40
ms to finish no matter how large is the write buffer size and no matter how
large is the payload.
Any thoughts? Or is there a another way for client size write buffering
through Thrift.
Thanks!
Gen
Re: Encryption in HBase
Hi, First you have to ask yourself what's the purpose of the encryption. Usually encryption is part of a bigger security implementation, so I don't think you'll gain much by encrypting at this level of granularity. Looking at trying to encrypt at the HFile level Unless I'm missing something, your HFIle will span multiple HDFS blocks. Each HDFS block is actually a file, right? So what happens when you fetch a row from a file in a simple get()? You can see that you have a lot of work to do. Compare that to encrypting the column data itself. Storing and encrypted byte array vs an un-encrypted byte array isn't much different. Does that make sense? On Aug 8, 2012, at 2:47 PM, Mohammad Tariq wrote: > Hello Michael, > > Thank you for the valuable suggestion. Could you please provide > me the the drawbacks of it?(Just out of curiosity). > Regards, >Mohammad Tariq > > > On Wed, Aug 8, 2012 at 8:01 PM, Michael Segel > wrote: >> You don't want to do that. >> I mean you really don't want to do that. ;-) >> >> You would be better off doing a strong encryption at the cell level. You can >> use co-processors to do that if you'd like. >> >> YMMV >> >> On Aug 8, 2012, at 9:17 AM, Mohammad Tariq wrote: >> >>> Hello Stack, >>> >>> Would it be feasible to encrypt the HFile itself?? >>> >>> Regards, >>> Mohammad Tariq >>> >>> >>> On Wed, Aug 8, 2012 at 6:53 PM, Jean-Marc Spaggiari >>> wrote: Will it be possible to use the compression methods to call an encryption method instead? 2012/8/8, Farrokh Shahriari : > Thanks > How can I encrypt a cell ? Is there a special way for encryption ? > > On Wed, Aug 8, 2012 at 3:24 PM, Stack wrote: > >> On Tue, Aug 7, 2012 at 12:06 PM, Farrokh Shahriari >> wrote: >>> Hi >>> How can I enable encryption on hbase ? >>> if I want to use encryption in a cluster with hbase running on top of >>> hadoop,what can I do? Can't hadoop be configured to automatically >>> encrypt >>> each file which is going to be written on it? >>> >> >> HBase does not provide this facility. You'd have to encrypt cell >> content yourself and then put it into HBase. >> >> St.Ack >> > >>> >> >
Re: Encryption in HBase
Thanks syed..makes sense to me. Regards, Mohammad Tariq On Thu, Aug 9, 2012 at 1:41 AM, syed kather wrote: > Tariq , >As wkt hbase is in Memory database when we need to scan or get from meta > file it will hit specified hfile . See if we encrypting the hfile then > internally we are trying to encrypting set of Key Length + Value Length + > Row Length + CF Length + Timestamp + Key Value also. We are interested in > encrypting key value . If we encrypt everything then decrypt will be pain > right. > > If we are encrypting only key values it is more then enough . Other than > why we need to do at co processor level . Wkt co processor will play with > minimum level data that like MR basis so really it will as add advantage to > our performance. And that too we can do decrypts for the data which matches > our query .. > Please correct me if I am wrong . > > Thanks and regards > Syed Abdul kather > send from Samsung S3 > On Aug 9, 2012 1:19 AM, "Mohammad Tariq" wrote: > >> Hello Michael, >> >>Thank you for the valuable suggestion. Could you please provide >> me the the drawbacks of it?(Just out of curiosity). >> Regards, >> Mohammad Tariq >> >> >> On Wed, Aug 8, 2012 at 8:01 PM, Michael Segel >> wrote: >> > You don't want to do that. >> > I mean you really don't want to do that. ;-) >> > >> > You would be better off doing a strong encryption at the cell level. You >> can use co-processors to do that if you'd like. >> > >> > YMMV >> > >> > On Aug 8, 2012, at 9:17 AM, Mohammad Tariq wrote: >> > >> >> Hello Stack, >> >> >> >>Would it be feasible to encrypt the HFile itself?? >> >> >> >> Regards, >> >>Mohammad Tariq >> >> >> >> >> >> On Wed, Aug 8, 2012 at 6:53 PM, Jean-Marc Spaggiari >> >> wrote: >> >>> Will it be possible to use the compression methods to call an >> >>> encryption method instead? >> >>> >> >>> 2012/8/8, Farrokh Shahriari : >> Thanks >> How can I encrypt a cell ? Is there a special way for encryption ? >> >> On Wed, Aug 8, 2012 at 3:24 PM, Stack wrote: >> >> > On Tue, Aug 7, 2012 at 12:06 PM, Farrokh Shahriari >> > wrote: >> >> Hi >> >> How can I enable encryption on hbase ? >> >> if I want to use encryption in a cluster with hbase running on top >> of >> >> hadoop,what can I do? Can't hadoop be configured to automatically >> >> encrypt >> >> each file which is going to be written on it? >> >> >> > >> > HBase does not provide this facility. You'd have to encrypt cell >> > content yourself and then put it into HBase. >> > >> > St.Ack >> > >> >> >> >> > >>
Re: Encryption in HBase
Tariq , As wkt hbase is in Memory database when we need to scan or get from meta file it will hit specified hfile . See if we encrypting the hfile then internally we are trying to encrypting set of Key Length + Value Length + Row Length + CF Length + Timestamp + Key Value also. We are interested in encrypting key value . If we encrypt everything then decrypt will be pain right. If we are encrypting only key values it is more then enough . Other than why we need to do at co processor level . Wkt co processor will play with minimum level data that like MR basis so really it will as add advantage to our performance. And that too we can do decrypts for the data which matches our query .. Please correct me if I am wrong . Thanks and regards Syed Abdul kather send from Samsung S3 On Aug 9, 2012 1:19 AM, "Mohammad Tariq" wrote: > Hello Michael, > >Thank you for the valuable suggestion. Could you please provide > me the the drawbacks of it?(Just out of curiosity). > Regards, > Mohammad Tariq > > > On Wed, Aug 8, 2012 at 8:01 PM, Michael Segel > wrote: > > You don't want to do that. > > I mean you really don't want to do that. ;-) > > > > You would be better off doing a strong encryption at the cell level. You > can use co-processors to do that if you'd like. > > > > YMMV > > > > On Aug 8, 2012, at 9:17 AM, Mohammad Tariq wrote: > > > >> Hello Stack, > >> > >>Would it be feasible to encrypt the HFile itself?? > >> > >> Regards, > >>Mohammad Tariq > >> > >> > >> On Wed, Aug 8, 2012 at 6:53 PM, Jean-Marc Spaggiari > >> wrote: > >>> Will it be possible to use the compression methods to call an > >>> encryption method instead? > >>> > >>> 2012/8/8, Farrokh Shahriari : > Thanks > How can I encrypt a cell ? Is there a special way for encryption ? > > On Wed, Aug 8, 2012 at 3:24 PM, Stack wrote: > > > On Tue, Aug 7, 2012 at 12:06 PM, Farrokh Shahriari > > wrote: > >> Hi > >> How can I enable encryption on hbase ? > >> if I want to use encryption in a cluster with hbase running on top > of > >> hadoop,what can I do? Can't hadoop be configured to automatically > >> encrypt > >> each file which is going to be written on it? > >> > > > > HBase does not provide this facility. You'd have to encrypt cell > > content yourself and then put it into HBase. > > > > St.Ack > > > > >> > > >
ANN: HBase 0.94.1 is available for download
The HBase Team is pleased to announce the release of HBase 0.94.1. Download it from your favorite Apache mirror [1]. HBase 0.94.1 is a bug fix release and has 156 issues resolved against it. It should be considered the current stable release of HBase at this point. For a complete list of changes, see release notes [2]. Yours, The HBase Team P.S. Thanks goes out to all who contributed to this release! 1. http://www.apache.org/dyn/closer.cgi/hbase/ 2. https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12320257
Re: Encryption in HBase
Hello Michael, Thank you for the valuable suggestion. Could you please provide me the the drawbacks of it?(Just out of curiosity). Regards, Mohammad Tariq On Wed, Aug 8, 2012 at 8:01 PM, Michael Segel wrote: > You don't want to do that. > I mean you really don't want to do that. ;-) > > You would be better off doing a strong encryption at the cell level. You can > use co-processors to do that if you'd like. > > YMMV > > On Aug 8, 2012, at 9:17 AM, Mohammad Tariq wrote: > >> Hello Stack, >> >>Would it be feasible to encrypt the HFile itself?? >> >> Regards, >>Mohammad Tariq >> >> >> On Wed, Aug 8, 2012 at 6:53 PM, Jean-Marc Spaggiari >> wrote: >>> Will it be possible to use the compression methods to call an >>> encryption method instead? >>> >>> 2012/8/8, Farrokh Shahriari : Thanks How can I encrypt a cell ? Is there a special way for encryption ? On Wed, Aug 8, 2012 at 3:24 PM, Stack wrote: > On Tue, Aug 7, 2012 at 12:06 PM, Farrokh Shahriari > wrote: >> Hi >> How can I enable encryption on hbase ? >> if I want to use encryption in a cluster with hbase running on top of >> hadoop,what can I do? Can't hadoop be configured to automatically >> encrypt >> each file which is going to be written on it? >> > > HBase does not provide this facility. You'd have to encrypt cell > content yourself and then put it into HBase. > > St.Ack > >> >
Re: HBase won't run on OSX 10.8
Did this fix end up working? I'm hesitant to upgrade to 10.8 if I'm going to run into this issue. I'm running the CDH3 jars locally to mirror my production cluster. On Tue, Jul 31, 2012 at 9:41 PM, Ryan Rawson wrote: > I shall try that. I submitted a patch too that quashes the extra % > where it is causing problems. > > > > On Tue, Jul 31, 2012 at 6:28 PM, Andrew Purtell > wrote: > > -Djava.net.preferIPv4Stack=true ? > > > > Does that still work? > > > > On Tue, Jul 31, 2012 at 6:24 PM, Ryan Rawson wrote: > >> Hi all, > >> > >> Something has changed in how OSX and java handles IPv6, and now you > >> will get a log like: > >> > >> 2012-07-31 18:21:39,824 INFO org.apache.hadoop.hbase.master.HMaster: > >> Server active/primary master; 0:0:0:0:0:0:0:0%0, > >> 59736,1343784093521, sessionid=0x138dfc60416, cluster-up flag > was=false > >> 2012-07-31 18:21:39,832 INFO > >> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: > >> globalMemStoreLimit=397.6m, global > >> MemStoreLimitLowMark=347.9m, maxHeap=994.1m > >> 2012-07-31 18:21:39,847 FATAL org.apache.hadoop.hbase.master.HMaster: > >> Master server abort: loaded coprocessors are: [ > >> ] > >> 2012-07-31 18:21:39,847 INFO > >> org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 16mins, > >> 40sec > >> 2012-07-31 18:21:39,848 FATAL org.apache.hadoop.hbase.master.HMaster: > >> Unhandled exception. Starting shutdown. > >> java.util.UnknownFormatConversionException: Conversion = '0' > >> at java.util.Formatter.checkText(Formatter.java:2503) > >> at java.util.Formatter.parse(Formatter.java:2467) > >> at java.util.Formatter.format(Formatter.java:2414) > >> at java.util.Formatter.format(Formatter.java:2367) > >> at java.lang.String.format(String.java:2769) > >> at > com.google.common.util.concurrent.ThreadFactoryBuilder.setNameFormat(ThreadFactoryBuilder.java:68) > >> at > org.apache.hadoop.hbase.executor.ExecutorService$Executor.(ExecutorService.java:299) > >> at > org.apache.hadoop.hbase.executor.ExecutorService.startExecutorService(ExecutorService.java:185) > >> at > org.apache.hadoop.hbase.executor.ExecutorService.startExecutorService(ExecutorService.java:227) > >> at > org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:821) > >> at > org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:507) > >> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:344) > >> at > org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.run(HMasterCommandLine.java:220) > >> at java.lang.Thread.run(Thread.java:680) > >> 2012-07-31 18:21:39,908 INFO org.apache.hadoop.hbase.master.HMaster: > Aborting > >> > >> As you can see, the hostname is now "0:0:0:0:0:0:0:0%0". The code > >> uses this as the prefix to a string, then that string is passed into a > >> formatter. Hence it is confused by %0, and aborts in a weird place > >> causing the entire thing to crash n burn. > >> > >> As I see it there are several solutions: > >> - s/%/_/ for the hostname at some point > >> - attempt not to use ipv6 > >> > >> the latter seems hacky though, surely someone would want to support > ipv6? > > > > > > > > -- > > Best regards, > > > >- Andy > > > > Problems worthy of attack prove their worth by hitting back. - Piet > > Hein (via Tom White) >
Re: Update table schema via REST
Addressing for table creation or schema update (PUT or POST), schema
query (GET), or delete (DELETE)
path := '/' / 'schema'
So that should be /TransactionREST/schema
However, I'd recommend that you not take administrative actions via
the REST API. Use the HBase shell instead. Consider them DBA-type
operations.
- Andy
On Wed, Aug 8, 2012 at 11:15 AM, Corbett Martin wrote:
> According to the Stargate documentation
> (http://wiki.apache.org/hadoop/Hbase/Stargate) I should be able to perform a
> POST operation to perform a table schema update. The docs say to supply a
> "well formed schema fragment" for a table schema update. My issue is the
> test code below always returns:
>
> HTTP ERROR 405
> Problem accessing /TransactionREST/. Reason:
> Method Not Allowed
>
> What am I missing? I cannot find any errors in the log files.
>
> It's worth mentioning that I've been able to successfully create a table via
> the REST api but now I want to use the api to make a table schema change,
> like in this example I want to add a new column-family to a table.
>
> class AlterTableExampleREST {
>
> private static final String ALTER_TABLE =
>
> "{\"name\":\"TransactionREST\",\"ColumnSchema\":[{\"name\":\"foo\"}]}";
>
> public void modifyTable() throws IOException {
> // Setup a Cluster list adding all known REST server hosts
> Cluster cluster = new Cluster();
> cluster.add("127.0.0.1", 8080);
> // Create Client to handle HTTP communication
> Client client = new Client(cluster);
> Response response = client.post("/TransactionREST/",
> "application/json",
> Bytes.toBytes(ALTER_TABLE));
> if(response != null) {
> System.out.println("response code: " + response.getCode());
> if(response.hasBody()) {
> System.out.println("reponse body: " +
> Bytes.toString(response.getBody()));
> }
> }
> }
> }
>
> Thanks in advance!
>
>
> This message and its contents (to include attachments) are the property of
> National Health Systems, Inc. and may contain confidential and proprietary
> information. This email and any files transmitted with it are intended solely
> for the use of the individual or entity to whom they are addressed. You are
> hereby notified that any unauthorized disclosure, copying, or distribution of
> this message, or the taking of any unauthorized action based on information
> contained herein is strictly prohibited. Unauthorized use of information
> contained herein may subject you to civil and criminal prosecution and
> penalties. If you are not the intended recipient, you should delete this
> message immediately and notify the sender immediately by telephone or by
> replying to this transmission.
Update table schema via REST
According to the Stargate documentation
(http://wiki.apache.org/hadoop/Hbase/Stargate) I should be able to perform a
POST operation to perform a table schema update. The docs say to supply a
"well formed schema fragment" for a table schema update. My issue is the test
code below always returns:
HTTP ERROR 405
Problem accessing /TransactionREST/. Reason:
Method Not Allowed
What am I missing? I cannot find any errors in the log files.
It's worth mentioning that I've been able to successfully create a table via
the REST api but now I want to use the api to make a table schema change, like
in this example I want to add a new column-family to a table.
class AlterTableExampleREST {
private static final String ALTER_TABLE =
"{\"name\":\"TransactionREST\",\"ColumnSchema\":[{\"name\":\"foo\"}]}";
public void modifyTable() throws IOException {
// Setup a Cluster list adding all known REST server hosts
Cluster cluster = new Cluster();
cluster.add("127.0.0.1", 8080);
// Create Client to handle HTTP communication
Client client = new Client(cluster);
Response response = client.post("/TransactionREST/",
"application/json",
Bytes.toBytes(ALTER_TABLE));
if(response != null) {
System.out.println("response code: " + response.getCode());
if(response.hasBody()) {
System.out.println("reponse body: " +
Bytes.toString(response.getBody()));
}
}
}
}
Thanks in advance!
This message and its contents (to include attachments) are the property of
National Health Systems, Inc. and may contain confidential and proprietary
information. This email and any files transmitted with it are intended solely
for the use of the individual or entity to whom they are addressed. You are
hereby notified that any unauthorized disclosure, copying, or distribution of
this message, or the taking of any unauthorized action based on information
contained herein is strictly prohibited. Unauthorized use of information
contained herein may subject you to civil and criminal prosecution and
penalties. If you are not the intended recipient, you should delete this
message immediately and notify the sender immediately by telephone or by
replying to this transmission.
Re: consistency, availability and partition pattern of HBase
And consistency is not sacrificed? i.e. all distributed clients' update will results in sequential / real time update? Once update is done by one client, all other client could see results immediately? regards, Lin On Wed, Aug 8, 2012 at 11:17 PM, Mohit Anchlia wrote: > I think availability is sacrificed in the sense that if region server > fails clients will have data inaccessible for the time region comes up on > some other server, not to confuse with data loss. > > Sent from my iPad > > On Aug 7, 2012, at 11:56 PM, Lin Ma wrote: > > > Thank you Wei! > > > > Two more comments, > > > > 1. How about Hadoop's CAP characters do you think about? > > 2. For your comments, if HBase implements "per key sequential > consistency", > > what are the missing characters for consistency? Cross-key update > > sequences? Could you show me an example about what you think are missed? > > thanks. > > > > regards, > > Lin > > > > On Wed, Aug 8, 2012 at 12:18 PM, Wei Tan wrote: > > > >> Hi Lin, > >> > >> In the CAP theorem > >> Consistency stands for atomic consistency, i.e., each CRUD operation > >> occurs sequentially in a global, real-time clock > >> Availability means each server if not partitioned can accept requests > >> > >> Partition means network partition > >> > >> As far as I understand (although I do not see any official > documentation), > >> HBase achieved "per key sequential consistency", i.e., for a specific > key, > >> there is an agreed sequence, for all operations on it. This is weaker > than > >> strong or sequential consistency, but stronger than "eventual > >> consistency". > >> > >> BTW: CAP was proposed by Prof. Eric Brewer... > >> http://en.wikipedia.org/wiki/Eric_Brewer_%28scientist%29 > >> > >> Best Regards, > >> Wei > >> > >> Wei Tan > >> Research Staff Member > >> IBM T. J. Watson Research Center > >> 19 Skyline Dr, Hawthorne, NY 10532 > >> w...@us.ibm.com; 914-784-6752 > >> > >> > >> > >> From: Lin Ma > >> To: user@hbase.apache.org, > >> Date: 08/07/2012 09:30 PM > >> Subject:consistency, availability and partition pattern of HBase > >> > >> > >> > >> Hello guys, > >> > >> According to the notes by Werner*, "*He presented the CAP theorem, which > >> states that of three properties of shared-data systems—data consistency, > >> system availability, and tolerance to network partition—only two can be > >> achieved at any given time." => > >> http://www.allthingsdistributed.com/2008/12/eventually_consistent.html > >> > >> But it seems HBase could achieve all of the 3 features at the same time. > >> Does it mean HBase breaks the rule by Werner. :-) > >> > >> If not, which one is sacrificed -- consistency (by using HDFS), > >> availability (by using Zookeeper) or partition (by using region / column > >> family) ? And why? > >> > >> regards, > >> Lin > >> > >> > >> >
Re: consistency, availability and partition pattern of HBase
I think availability is sacrificed in the sense that if region server fails clients will have data inaccessible for the time region comes up on some other server, not to confuse with data loss. Sent from my iPad On Aug 7, 2012, at 11:56 PM, Lin Ma wrote: > Thank you Wei! > > Two more comments, > > 1. How about Hadoop's CAP characters do you think about? > 2. For your comments, if HBase implements "per key sequential consistency", > what are the missing characters for consistency? Cross-key update > sequences? Could you show me an example about what you think are missed? > thanks. > > regards, > Lin > > On Wed, Aug 8, 2012 at 12:18 PM, Wei Tan wrote: > >> Hi Lin, >> >> In the CAP theorem >> Consistency stands for atomic consistency, i.e., each CRUD operation >> occurs sequentially in a global, real-time clock >> Availability means each server if not partitioned can accept requests >> >> Partition means network partition >> >> As far as I understand (although I do not see any official documentation), >> HBase achieved "per key sequential consistency", i.e., for a specific key, >> there is an agreed sequence, for all operations on it. This is weaker than >> strong or sequential consistency, but stronger than "eventual >> consistency". >> >> BTW: CAP was proposed by Prof. Eric Brewer... >> http://en.wikipedia.org/wiki/Eric_Brewer_%28scientist%29 >> >> Best Regards, >> Wei >> >> Wei Tan >> Research Staff Member >> IBM T. J. Watson Research Center >> 19 Skyline Dr, Hawthorne, NY 10532 >> w...@us.ibm.com; 914-784-6752 >> >> >> >> From: Lin Ma >> To: user@hbase.apache.org, >> Date: 08/07/2012 09:30 PM >> Subject:consistency, availability and partition pattern of HBase >> >> >> >> Hello guys, >> >> According to the notes by Werner*, "*He presented the CAP theorem, which >> states that of three properties of shared-data systems—data consistency, >> system availability, and tolerance to network partition—only two can be >> achieved at any given time." => >> http://www.allthingsdistributed.com/2008/12/eventually_consistent.html >> >> But it seems HBase could achieve all of the 3 features at the same time. >> Does it mean HBase breaks the rule by Werner. :-) >> >> If not, which one is sacrificed -- consistency (by using HDFS), >> availability (by using Zookeeper) or partition (by using region / column >> family) ? And why? >> >> regards, >> Lin >> >> >>
Re: Encryption in HBase
You don't want to do that. I mean you really don't want to do that. ;-) You would be better off doing a strong encryption at the cell level. You can use co-processors to do that if you'd like. YMMV On Aug 8, 2012, at 9:17 AM, Mohammad Tariq wrote: > Hello Stack, > >Would it be feasible to encrypt the HFile itself?? > > Regards, >Mohammad Tariq > > > On Wed, Aug 8, 2012 at 6:53 PM, Jean-Marc Spaggiari > wrote: >> Will it be possible to use the compression methods to call an >> encryption method instead? >> >> 2012/8/8, Farrokh Shahriari : >>> Thanks >>> How can I encrypt a cell ? Is there a special way for encryption ? >>> >>> On Wed, Aug 8, 2012 at 3:24 PM, Stack wrote: >>> On Tue, Aug 7, 2012 at 12:06 PM, Farrokh Shahriari wrote: > Hi > How can I enable encryption on hbase ? > if I want to use encryption in a cluster with hbase running on top of > hadoop,what can I do? Can't hadoop be configured to automatically > encrypt > each file which is going to be written on it? > HBase does not provide this facility. You'd have to encrypt cell content yourself and then put it into HBase. St.Ack >>> >
Re: Encryption in HBase
Hello Stack, Would it be feasible to encrypt the HFile itself?? Regards, Mohammad Tariq On Wed, Aug 8, 2012 at 6:53 PM, Jean-Marc Spaggiari wrote: > Will it be possible to use the compression methods to call an > encryption method instead? > > 2012/8/8, Farrokh Shahriari : >> Thanks >> How can I encrypt a cell ? Is there a special way for encryption ? >> >> On Wed, Aug 8, 2012 at 3:24 PM, Stack wrote: >> >>> On Tue, Aug 7, 2012 at 12:06 PM, Farrokh Shahriari >>> wrote: >>> > Hi >>> > How can I enable encryption on hbase ? >>> > if I want to use encryption in a cluster with hbase running on top of >>> > hadoop,what can I do? Can't hadoop be configured to automatically >>> > encrypt >>> > each file which is going to be written on it? >>> > >>> >>> HBase does not provide this facility. You'd have to encrypt cell >>> content yourself and then put it into HBase. >>> >>> St.Ack >>> >>
Re: Encryption in HBase
Will it be possible to use the compression methods to call an encryption method instead? 2012/8/8, Farrokh Shahriari : > Thanks > How can I encrypt a cell ? Is there a special way for encryption ? > > On Wed, Aug 8, 2012 at 3:24 PM, Stack wrote: > >> On Tue, Aug 7, 2012 at 12:06 PM, Farrokh Shahriari >> wrote: >> > Hi >> > How can I enable encryption on hbase ? >> > if I want to use encryption in a cluster with hbase running on top of >> > hadoop,what can I do? Can't hadoop be configured to automatically >> > encrypt >> > each file which is going to be written on it? >> > >> >> HBase does not provide this facility. You'd have to encrypt cell >> content yourself and then put it into HBase. >> >> St.Ack >> >
Re: Encryption in HBase
Thanks How can I encrypt a cell ? Is there a special way for encryption ? On Wed, Aug 8, 2012 at 3:24 PM, Stack wrote: > On Tue, Aug 7, 2012 at 12:06 PM, Farrokh Shahriari > wrote: > > Hi > > How can I enable encryption on hbase ? > > if I want to use encryption in a cluster with hbase running on top of > > hadoop,what can I do? Can't hadoop be configured to automatically encrypt > > each file which is going to be written on it? > > > > HBase does not provide this facility. You'd have to encrypt cell > content yourself and then put it into HBase. > > St.Ack >
RE: more tables or more rows
Hi, A slightly related question: We have time series data continuously flowing into the system and has to be stored in HBase. We have retention policy to retain data for 90 days, so data older than 90 days have to be deleted from HBase every midnight. There are two (that we know) ways of doing this: 1) Since bulk deletes could be costly and dropping an entire table is easier, we could have day wise tables and drop entire table 2) This post http://permalink.gmane.org/gmane.comp.java.hadoop.hbase.user/9603 suggests that we can have a single table and use the TTL feature for ageing out data. May I request someone to briefly list out the pros and cons of either options? PS: We expect around 200 million records per day and each record would be approx.. 500 bytes. Thanks & Regards MK -Original Message- From: Mohammad Tariq [mailto:donta...@gmail.com] Sent: 08 August 2012 03:19 To: user@hbase.apache.org Subject: Re: more tables or more rows Hello sir, It is absolutely fine to have as many tables as we like. My point was that if we have a large no of tables then it might add some overhead in locating the user region, as there will be a huge amount of mapping from "user tables" to "region servers". Also, client will have to cache more information blocking the additional memory. So, I suggested to have small no of large tables rather than large no of small tables, if the data is similar. Regards, Mohammad Tariq On Tue, Aug 7, 2012 at 5:30 PM, Eric Czech wrote: > Thanks Mohammad, > > By saying the major purpose is to host very large tables (implying a > smaller number of them), are you referring to anything other than the > memstores per column family taking up sizable portions of physical memory? > Are there other components or design aspects that make using large > numbers of tables inadvisable? > > On Sun, Aug 5, 2012 at 5:55 PM, Mohammad Tariq wrote: >> Hello sir, >> >> Going for a single table with 30+ rows would be a better >> choice, if the data from all the sources is not very different. >> Since, you are considering Hbase as your data store, it wouldn't be >> wise to have several small rows. The major purpose of Hbase is to >> host very large tables that may go beyond billions of rows and millions of >> columns. >> >> Regards, >> Mohammad Tariq >> >> >> On Mon, Aug 6, 2012 at 3:18 AM, Eric Czech wrote: >>> I need to support data that comes from 30+ sources and the structure >>> of that data is consistent across all the sources, but what I'm not >>> clear on is whether or not I should use 30+ tables with roughly the >>> same format or 1 table where the row key reflects the source. >>> >>> Anybody have a strong argument one way or the other? >>> >>> Thanks!
Re: Hbase- Hadoop DFS size not decreased even after deleting a column family
Hi Stack,
we have got a lot of these in the logs after we deleted the column family.
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column
family ol does not exist in region
at
org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:4759)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkFamilies(HRegion.java:2639)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2065)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1962)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3428)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1386)
2012-08-05 00:11:37,697 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
rollbackMemstore rolled back 0 keyvalues from start:0 to end:1
On Tue, Aug 7, 2012 at 10:27 PM, Ted Yu wrote:
> Looking at TableDeleteFamilyHandler, I see:
>
> protected void handleTableOperation(List hris) throws
> IOException {
> // Update table descriptor in HDFS
> HTableDescriptor htd =
> this.masterServices.getMasterFileSystem().deleteColumn(tableName,
> familyName);
> // Update in-memory descriptor cache
> this.masterServices.getTableDescriptors().add(htd);
>
> MasterFileSystem.deleteColumn() calls:
>
> HTableDescriptor htd =
> this.services.getTableDescriptors().get(tableName);
> htd.removeFamily(familyName);
>
> I will dig some more.
>
> But looks like there is a bug.
>
> On Tue, Aug 7, 2012 at 9:21 AM, Stack wrote:
>
> > On Tue, Aug 7, 2012 at 9:59 AM, prem yadav wrote:
> > > Hi Stack,
> > > yes . I can still see the column family.
> > >
> > > we ran the following command too:
> > >
> > > fs -dus /hbase/// and it
> > > returned a large value which means that the space has not been
> reclaimed.
> > > The hbase version we are running is 0.94.1.
> > >
> > > is there a manual way to force reclaim the space?
> > >
> >
> > You could delete the column family from hdfs under each region.
> > Anything in the logs around the remove of the column family? Maybe we
> > just don't do the delete cleanup in hdfs when we drop a cf (I've not
> > checked)?
> >
> > St.Ack
> >
>
Re: Encryption in HBase
On Tue, Aug 7, 2012 at 12:06 PM, Farrokh Shahriari wrote: > Hi > How can I enable encryption on hbase ? > if I want to use encryption in a cluster with hbase running on top of > hadoop,what can I do? Can't hadoop be configured to automatically encrypt > each file which is going to be written on it? > HBase does not provide this facility. You'd have to encrypt cell content yourself and then put it into HBase. St.Ack
Re: hfile v2 in cloudera cdh3
Hi Weishung, No, CDH3's HBase, being based on 0.90.x, does not carry this new format. On Wed, Aug 8, 2012 at 3:23 PM, Weishung Chung wrote: > Hi HBase users, > > Does any of the cloudera cdh3 versions contain the new hfile v2 ? I know it > will be in cdh4. > > Thank you, > Wei Shung -- Harsh J
Re: client keeps writting hbase put info to stdout.log
yes i found i wrote some stupid system out log, thank you for your kind reply! 在 2012-8-7，下午3:33， Stack 写道： > On Tue, Aug 7, 2012 at 3:11 AM, deanforwever2010 > wrote: >> hi Stack >> the version is 0.94 >> it is the client side output >> > > I took a look at our System.outs and I don't see what could be > responsible for the above (I could be missing it). Any chance your > client code is doing the System.out on a Put? > > St.Ack
Reg:Insertion of boolean values from the hbase shell
Hi, Can anyone tell me how to insert the boolean values from the hbase shell. Regards Swathi Kristam *** This e-mail and attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient's) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
