Re: [ANNOUNCE] HBase 0.94.9 is available for download

Thanks Lars. It looks like HBASE-8656 , though part of the tarball, is not included in 0.94.9 tag. Regards, Aditya... On Thu, Jul 4, 2013 at 10:58 AM, lars hofhansl wrote: > The HBase Team is pleased to announce the immediate release of HBase

Re: Which hbase version has Coprocessor function?

You might found usefull information there: http://bit.ly/1aLhUh1 Looks like 0.92 is the right answer. 2013/7/5 yuhe : > I use cdh3u4 I do not know if it's hbase has coprocessor function,thanks all > > -- > 使用语盒发送 @2013-07-06 01:40 > http://www.yuchs.com

Which hbase version has Coprocessor function?

I use cdh3u4 I do not know if it's hbase has coprocessor function,thanks all -- 使用语盒发送 @2013-07-06 01:40 http://www.yuchs.com

Re: HBase masters file

Hi Sudarshan, Are you using 0.94.6? Or 0.94.6.1? The former is not recommend. regarding the master file, no, it's not required anymore, so you can stop looking for it ;) Regarding the ZK nodes, sorry, I don't know how it was in 0.92 so I can't compare. JM 2013/7/5 Sudarshan Kadambi (BLOOMBERG/

Re: help with filter combination

Thanks Ted, The deptId and empid are analogies to the case at hand. The SingleColumnValueFilter expects (colfam, columnQualifier, CompareOp, value). In my case colfam = pt, columnQualifier = empid. But the empid is not a fixed string. Instead it varies based on my input data. So 100 employees would

Re: help with filter combination

Consider SingleColumnValueFilter You can refer to src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueFilter.java BTW why is deptId stored in the pt:employeeId column ? Cheers On Fri, Jul 5, 2013 at 4:43 PM, Rohit Kelkar wrote: > Each row in my hbase table contains the following

help with filter combination

Each row in my hbase table contains the following data: rowkey column=pt:np, value="abcd" column=pt:vb, value="efgh" column=pt:employeeId, value=deptId Using a combination of filters is it possible to get all rows and all qualifiers within the pt column family where de

Re: When to expand vertically vs. horizontally in Hbase

Sure. Maybe it's useful to talk about the functional aspect of relationships in models. In an RDBMS, explicit relationship play a couple roles: - foreign key constraints: don't allow a tuple in relation A to point to a row in relation B that doesn't exist - join optimization - knowledge of how t

HBase masters file

HBase 0.92 used to have a /etc/hbase/conf/masters file, which contained the location of the HBase master. Has this file been removed in HBase 0.94.6? I see the regionservers file, but not the masters file in our deployment. I also used the zoookeeper-client to browse the /hbase subtree. The /hb

Re: When to expand vertically vs. horizontally in Hbase

An entity is an entity. When you couple them you are saying that there's a relationship to them in the model. What I am saying is that you can have an HBase model which is not a single table, however when you look at your use case, you are querying data from a single table at a time. Going

Re: MapReduce job with mixed data sources: HBase table and HDFS files

Actually you can, albeit it will be slower than you would think. You'd have to do a single threaded scan to pull the data from the remote cluster to the local cluster then once its local you can parallelize the HDFS m/r portion of the job. Note: Can do some thing versus can't do something does

Re: Bulk loading HFiles via LoadIncrementalHFiles fails at a region that is being compacted, a bug?

Hey Stan: 60seconds is a long time. If you try upping it the wait, does it still fail still? (looking at how long to wait is calculated, it is a little hairy figuring what to change). Any chance of thread dump while it is hung up? Might tell us something? Good on you Stan, St.Ack On Fri, Ju

Re: HBase Upgrade

On Fri, Jul 5, 2013 at 12:40 PM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > I am using 0.94.6.1 so i will be able to rolling upgrade to 0.94.8. > > As you said that clients and servers calls between HBase minor releases are > supposed to be compatible. So there should not be any iss

Re: HBase Upgrade

Please read this: http://hbase.apache.org/book.html#upgrade0.96 Cheers On Fri, Jul 5, 2013 at 12:40 PM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > I am using 0.94.6.1 so i will be able to rolling upgrade to 0.94.8. > > As you said that clients and servers calls between HBase minor

Re: HBase Upgrade

I am using 0.94.6.1 so i will be able to rolling upgrade to 0.94.8. As you said that clients and servers calls between HBase minor releases are supposed to be compatible. So there should not be any issue for rolling upgrades between HBase minor releases. Will it be possible to rolling upgrade fr

Re: When to expand vertically vs. horizontally in Hbase

Mike, what do you mean by "you can have entities, except that they are not coupled"? You mean, they have no relationship to each other? Or the relationship is defined elsewhere (e.g. application code)? The concept of "coupling" seems a little overloaded and not as concise here as "relationship".

Re: When to expand vertically vs. horizontally in Hbase

LOL... Ian wrote: "But, something just occurred to me: just because your physical implementation (HBase) doesn't support normalized entities and relationships doesn't mean your *problem* doesn't have entities and relationships. :) An Author is one entity, a Title is another, and a Genre is a th

Re: When to expand vertically vs. horizontally in Hbase

Mike and I get into good discussions about ERD modeling and HBase a lot ... :) Mike's right that you should avoid a design that relies heavily on relationships when modeling data in HBase, because relationships are tricky (they're the first thing that gets throw out the window in a database that

Re: When to expand vertically vs. horizontally in Hbase

Sorry, but you missed the point. (Note: This is why I keep trying to put a talk at Strata and the other conferences on Schema design yet for some reason... it just doesn't seem important enough or sexy enough... maybe if I worked for Cloudera/Intel/etc ... ;-) Look, The issue is what is a

Bulk loading HFiles via LoadIncrementalHFiles fails at a region that is being compacted, a bug?

We are running HBase version 0.94.6-cdh4.3.0 and I have noticed that quite often while bulk loading HFiles created with a MR job fails with following Exception: 13/06/26 18:28:15 ERROR mapreduce.LoadIncrementalHFiles: Encountered unrecoverable error from region server org.apache.hadoop.hbase.clien

Re: When to expand vertically vs. horizontally in Hbase

I understand that there shouldn't be unlimited number of column families. I am using this example on purpose to see how it comes into play. On Fri, Jul 5, 2013 at 12:07 PM, Michael Segel wrote: > Why do you have so many column families (CF) ? > > Its not a question on the physical limitations, b

Re: When to expand vertically vs. horizontally in Hbase

Why do you have so many column families (CF) ? Its not a question on the physical limitations, but more on the issue of data design. There aren't that many really good examples of where you would have multiple column families that would require more than a handful of CFs. When I teach or le

Re: When to expand vertically vs. horizontally in Hbase

Asaf, I am using the Genre/Author stuff as an example but yes at the moment I only have 5 column families. However, over time I may have more (no upper limit decided that this point). See below for more responses On Wed, Jul 3, 2013 at 3:42 PM, Asaf Mesika wrote: > Do you have only 5 static a

Re: Calculating size in HBase.

hadoop dfs -dus /hbase *Thanks & Regards* ∞ Shashwat Shriparv On Fri, Jul 5, 2013 at 4:54 PM, Amit Mor wrote: > From hadooo shell man > > "DU > Usage: hadoop fs -du URI [URI …] > > Displays aggregate length of files contained in the directory or the length > of a file in case its just a

Re: Calculating size in HBase.

>From hadooo shell man "DU Usage: hadoop fs -du URI [URI …] Displays aggregate length of files contained in the directory or the length of a file in case its just a file. Example: hadoop fs -du /user/hadoop/dir1 /user/hadoop/file1 hdfs://nn.example.com/user/hadoop/dir1 Exit Code: Returns 0 on suc

Re: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.WritableRpcEngine not found

This problem was due to older version of the hadoop core jar. It is now solved. -- View this message in context: http://apache-hbase.679495.n3.nabble.com/java-lang-RuntimeException-java-lang-ClassNotFoundException-org-apache-hadoop-hbase-ipc-WritableRpcEd-tp4047442p4047452.html Sent from the HB

Re: question about clienttrace logs in hdfs and shortcircuit read

Sweet! enabled debug logging for org.apache.hadoop.hdfs.DFSClient and found the "New BlockReaderLocal" log line. Got some verification that SCR is ON and working fine. Regarding no clienttrace lines in DN, I verified that too. Last time I saw a few lines since I forgot to remove HDFS_WRITE lines.

Re: question about clienttrace logs in hdfs and shortcircuit read

Thanks Viral.. Confusion came to me as the heading of the mails say abt SCR metric and finally speaks abt hbase handled checksum metric which are 2 features.. Agree that hbase checksum works only with SCR as of now, this might change later.. That is why I was just making it clear .. NP :) As I

Re: question about clienttrace logs in hdfs and shortcircuit read

No worries, Anoop. Here is some clarification for this chain. It started initially to figure out how to check whether SCR is effective at the RS or not. I could not find the metric anywhere in ganglia/JMX and didn't find any RegionServer level metric either and so started looking at my DN logs. I

Re: question about clienttrace logs in hdfs and shortcircuit read

Viral DFS client uses org.apache.hadoop.hdfs.BlockReaderLocal for SCR.. I can see some debug level logs in this * LOG*.debug("New BlockReaderLocal for file " + blkfile + " of size " + blkfile.length() + " startOffset " + startOffset + " length " + length + " short circuit checksu

Re: question about clienttrace logs in hdfs and shortcircuit read

checksumFailures will get updated when the HBase handled checksum feature is in use and checksum check done at RS side failed.. If it happens we will try to read from DN with DN checksum check enabled. Agree that right now the HBase handled checksum will work only with SCR. But it might work wit

Re: question about clienttrace logs in hdfs and shortcircuit read

Yes I was checking 0.94 code. And sorry for the brain fart, I just spotted the metric in ganglia. There are just too many metrics in ganglia and skipped this one! It was under the group "hbase.regionserver", while I was expected it to be "hbase.regionserver.RegionServerStatistics". The chart show

Re: question about clienttrace logs in hdfs and shortcircuit read

I was checking with the trunk code and i saw that /** * Number of checksum verification failures. It also * clears the counter. */ public static final long getChecksumFailuresCount() { return checksumFailures.getAndSet(0); } Is not getting used in the trunk code but it is getting

Re: question about clienttrace logs in hdfs and shortcircuit read

I saw the same code and also saw the following in RegionServerMetrics.java /** * Number of times checksum verification failed. */ public final MetricsLongValue checksumFailuresCount = new MetricsLongValue("checksumFailuresCount", registry); The registry is then registered in JMX via: // expo