On Thu, Jun 6, 2013 at 8:15 AM, Stack wrote:
>
>>
>> bq. increase tickTime in zoo.cfg?
>>
>> For shared zookeeper quorum, the above should be done.
>>
>>
> What?
>
>
What I meant to say is, how does this answer the question "Is this even
relevant anymore? hbase.zookeeper.property.tickTime ?"?
S
On Thu, Jun 6, 2013 at 4:57 AM, Ted Yu wrote:
> bq. I just dont find this "hbase.zookeeper.property.tickTime" anywhere in
> the code base.
>
> Neither do I. Mind filing a JIRA to correct this in troubleshooting.xml ?
>
It intentionally does not exist in the hbase code base. Read the first
par
Hi Azuryy ,
Currently i am not able to reproduce the problem.
Also i had checked namenode log ,i did not find any issue in it.
On Thu, Jun 6, 2013 at 10:53 AM, Azuryy Yu wrote:
> And, please check your namenode log.
>
>
> On Thu, Jun 6, 2013 at 1:20 PM, Azuryy Yu wrote:
>
> > Can you reproduce
> I feel that warming up the block and
index cache could be a useful feature for many workflows. Would it be a
good idea to have a JIRA for that?
This will be against the concept of multi level index block structure in
HFile V2 where we dont want the whole index data to be loaded at once
initially
Hbase depends on hdfs. So NameNode should be started first, followed by data
nodes.
Then you can start hbase cluster.
When stopping, stop hbase cluster followed by stopping hdfs.
Cheers
On Jun 5, 2013, at 10:28 PM, Vimal Jain wrote:
> Hi,
> What should be the order of starting and stopping
Hi,
What should be the order of starting and stopping Hbase and HDFS related
process ?
--
Thanks and Regards,
Vimal Jain
And, please check your namenode log.
On Thu, Jun 6, 2013 at 1:20 PM, Azuryy Yu wrote:
> Can you reproduce the problem? if yes,
>
> add the following in your hbase-env.sh
>
> export HBASE_MASTER_OPTS="-verbose:gc -XX:+PrintGCDateStamps
> -XX:+PrintGCDetails -Xloggc:$HBASE_LOG_DIR/hmaster_gc.log
Can you reproduce the problem? if yes,
add the following in your hbase-env.sh
export HBASE_MASTER_OPTS="-verbose:gc -XX:+PrintGCDateStamps
-XX:+PrintGCDetails -Xloggc:$HBASE_LOG_DIR/hmaster_gc.log
$HBASE_MASTER_OPTS"
export HBASE_REGIONSERVER_OPTS="-verbose:gc -XX:+PrintGCDateStamps
-XX:+PrintGC
HBASE-7404 Bucket Cache has done some work in this regard.
Please refer to the latest patch on that JIRA
Thanks
On Jun 5, 2013, at 9:48 PM, Shixiaolong wrote:
> Hi,
>Anyone could give some suggestion about block cache extension. We want to
> Replace the existing LRU block cache mechanism
Hi,
Anyone could give some suggestion about block cache extension. We want
to
Replace the existing LRU block cache mechanism to another one to fasten reading
, but I don't know how to do it.
Thanks
Shixiaolong
How many total RS in the cluster? You mean u can not do any operation on
other regions in the live clusters? It should not happen.. Is it so
happening that the client ops are targetted at the regions which were in
the dead RS( and in transition now)? Can u have a closer look and see?
If not pl
> I feel that warming up the block and
index cache could be a useful feature for many workflows. Would it be a
good idea to have a JIRA for that?
I would suggest yes. You can log the details that you observed and can
discuss over there.
In your case you had multi level index blocks itself and henc
bq. I just dont find this "hbase.zookeeper.property.tickTime" anywhere in
the code base.
Neither do I. Mind filing a JIRA to correct this in troubleshooting.xml ?
bq. increase tickTime in zoo.cfg?
For shared zookeeper quorum, the above should be done.
On Wed, Jun 5, 2013 at 5:45 PM, Ameya Kan
Hi Azuryy/Ted,
Can you please help here...
On Jun 5, 2013 7:23 PM, "Kevin O'dell" wrote:
> No!
>
> Just kidding, you can unsubscribe by going to the Apache site:
>
> http://hbase.apache.org/mail-lists.html
>
>
> On Wed, Jun 5, 2013 at 9:34 AM, Joseph Coleman <
> joe.cole...@infinitecampus.com> wr
I'm not sure what caused so many index block misses. At the time I ran the
experiment had over 12 GB of RAM assigned to block cache. My understanding
is that since I had restarted HBase before running this experiment it was
basically loading index blocks as and when needed and thus index misses
wer
Take a look at release notes for HBASE-5416.
Below is from SingleColumnValueFilter:
public boolean isFamilyEssential(byte[] name) {
return !this.filterIfMissing || Bytes.equals(name, this.columnFamily);
}
Also refer to TestJoinedScanners
Cheers
On Wed, Jun 5, 2013 at 6:58 PM, Siyuan he
Hello,
Can some one tell how to define a column family as a "essential column
family"?
Thanks,
siyuan
On Wed, Jun 5, 2013 at 4:21 PM, Ted Yu wrote:
> For thrift, there is already such support.
>
> Take a look at (0.94 codebase):
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java
>
> * HRegionThriftServer - this class starts up a Thrift server in the same
> * JVM where
That's exactly the kind of pre-fetching I was investigating a bit ago (made a
patch, but ran out of time).
This pre-fetching is strictly client only, where the client keeps the server
busy while it is processing the previous batch, but filling up a 2nd buffer.
-- Lars
___
One more thing. I just dont find this "hbase.zookeeper.property.tickTime"
anywhere in the code base.
Also, I could not find ZooKeeper API that takes tickTime from client.
http://zookeeper.apache.org/doc/r3.3.3/api/org/apache/zookeeper/ZooKeeper.html
It takes sessionTime out value, but not tickTime.
Many apologies for forwarding this email again.
Could you let me know how can I be able to pull/export the real-time raw
logs (number of requests and their details in a particular regions) which
appears in the HBase Web UI like shown in below? I looked at pp. 277-283 of
Lars George's book and othe
Which tickTime is honored?
One in zoo.cfg or hbase.zookeeper.property.tickTime in hbase-site.xml?
My understanding now is, whichever tickTime is honored, session time can
not be more than 20 times the value.
I think this is whats happening on my cluster:
My hbase.zookeeper.property.tickTime val
Hi Shahab,
Thanks for the response. There was some misunderstanding between me and my
sysadmin's. By using the keytab similar to "hbase" user i can log-in.
Thanks,
Anil Gupta
On Wed, Jun 5, 2013 at 2:10 PM, Shahab Yunus wrote:
> Plus Anil, have you gone through this offical doc about Hbase an
bq. I thought this property in hbase-site.xml takes care of that:
zookeeper.session.timeout
From
http://zookeeper.apache.org/doc/current/zookeeperProgrammers.html#ch_zkSessions:
The client sends a requested timeout, the server responds with the timeout
that it can give the client. The current imp
In zoo.cfg I have not setup this value explicitly. My zoo.cfg looks like:
tickTime=2000
initLimit=10
syncLimit=5
We use common zoo keeper cluster for 2 of our HBase clusters. I'll try
increasing this value from zoo.cfg.
However is it possible to set this value cluster specific?
I thought this pro
Plus Anil, have you gone through this offical doc about Hbase and Kerberos:
http://hbase.apache.org/book/security.html
Regards,
Shahab
On Wed, Jun 5, 2013 at 5:02 PM, Shahab Yunus wrote:
> You should not have to enter a password for Kerberos authentication. You
> need to have a ticket against
You should not have to enter a password for Kerberos authentication. You
need to have a ticket against keytab file generated by the KDC server(s).
Did you configure Kerberos your self on the server?
Regards,
Shahab
On Wed, Jun 5, 2013 at 4:34 PM, anil gupta wrote:
> Hi All,
>
> I am running a
That's my understanding of how the current scan API works, yes. The
client calls next() to fetch a batch. While it's waiting for the response
from the server, it blocks. After the server responds to the next() call,
it does nothing for that scanner until the following next() call. That
makes fo
Ameya,
What does your zoo.cfg say for your timeout value?
On Wed, Jun 5, 2013 at 4:47 PM, Ameya Kantikar wrote:
> Hi,
>
> We have heavy map reduce write jobs running against our cluster. Every once
> in a while, we see a region server going down.
>
> We are on : 0.94.2-cdh4.2.0, r
>
> We hav
Hi,
We have heavy map reduce write jobs running against our cluster. Every once
in a while, we see a region server going down.
We are on : 0.94.2-cdh4.2.0, r
We have done some tuning for heavy map reduce jobs, and have increased
scanner timeouts, lease timeouts, have also tuned memstore as follo
Hi All,
I am running a secure Hadoop/HBase cluster. My user "hbase_user" has a
kerberos password. I am unable to figure out how to pass the value of
password for this user. Is there a way to provide the login password
through Java api(org.apache.hadoop.hbase.security.User)?
I am able to login to
Hi,
I've tried using a GraphiteContext I've found in GitHub, but I discovered I
only get a very partial list of metrics.
I've switched to using TimeStampingFileContext and I get the same list:
2013-06-05T18:24:59 hbase.master:
Master=a172-25-98-73.deploy.acme.com,6,1370455069499,
hostName=a172
Dear Sandy,
Thanks for your explanation.
However, what I don't get is your term "client", is this "client" means
MapReduce jobs? If I understand you right, this means Map function will
process the tuples and during this processing time, the regionserver did
nothing?
regards!
Yong
On Wed, Jun
Jon:
bq. we'll also be hosting an HBase Hackathon [4] during they day of June
12th
June 12th is approaching quickly. Can you give us some more information
about the HBase Hackathon ?
Thanks
On Mon, May 6, 2013 at 10:29 AM, Jonathan Hsieh wrote:
> HBaseCon 2013 is THE community event for HBase
Yong,
As a thought experiment, imagine how it impacts the throughput of TCP to
keep the window size at 1. That means there's only one packet in flight
at a time, and total throughput is a fraction of what it could be.
That's effectively what happens with RPC. The server sends a batch, then
does
Dear All,
We have production cluster that runs on hbase 0.94.1. The issue we are
facing is whenever one regionserver goes down, the cluster becomes
unresponsive until all the regions are allocated to another
regionserver(s). The transition is taking about 3-5 mins and during this
time we are unabl
Hi,
I was able to follow HBase tutorial. Now, I'd like to follow a best practice to
install HBase/Hadoop in the same cluster for maximum performance without using
Whirr or CDH as I will run it in a private cluster without sudo access. I can't
seem to find a good step by step tutorial on that. C
hi, Anoop,
Many thanks for the information. I misunderstand the concept between
'REPLICATION_SCOPE' and 'PEER'. Since I am able to set the
'REPLICATION_SCOPE' to 2, I thought the CF will be replicated to peer '2'
only, instead of peer '1'.
Demai
>There is no support like one CF can be replicat
bq. the Regionserver and Tasktracker are the same node when you use
MapReduce to scan the HBase table.
The scan performed by the Tasktracker on that node would very likely access
data hosted by region server on other node(s). So there would be RPC
involved.
There is some discussion on providing s
Hi,
Thanks to everyone that replied!
@Tariq: Not childish at all. We don't have direct access to the database, but
rather we will be going through a web service to obtain dumps of the data. This
is why we are not using sqoop (unless sqoop would support such an operation,
but not that I'm a
Can you go over http://hbase.apache.org/book.html#performance and then,
http://hbase.apache.org/book.html#trouble
Cheers
On Wed, Jun 5, 2013 at 8:32 AM, Ramasubramanian <
ramasubramanian.naraya...@gmail.com> wrote:
> Hi,
>
> Currently a Hbase table contains close to 60 million records and for
>
Hi,
Currently a Hbase table contains close to 60 million records and for loading
new records (no update.. new row key don't exists ) it is taking more than 3
hours to load 30,000 records… day by day when the volume of the records getting
increase it is going to 5 hours a day to load same 30,000
Thanks for the info Ted, I look to see if I can do the same for REST.
Simon
On Wed, Jun 5, 2013 at 4:21 PM, Ted Yu wrote:
> For thrift, there is already such support.
>
> Take a look at (0.94 codebase):
> src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java
>
> * HRegi
Can anyone explain why client + rpc + server will decrease the performance
of scanning? I mean the Regionserver and Tasktracker are the same node when
you use MapReduce to scan the HBase table. So, in my understanding, there
will be no rpc cost.
Thanks!
Yong
On Wed, Jun 5, 2013 at 10:09 AM, San
For thrift, there is already such support.
Take a look at (0.94 codebase):
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java
* HRegionThriftServer - this class starts up a Thrift server in the same
* JVM where the RegionServer is running. It inherits most of the
* fun
Do you expect setting this same value in one transaction ?
There is no such support yet.
Cheers
On Wed, Jun 5, 2013 at 6:18 AM, Bai Shen wrote:
> I want to set a column to be the same value for all rows from the shell. I
> can do it using put for a single row, but I'm not sure how to do it fo
No!
Just kidding, you can unsubscribe by going to the Apache site:
http://hbase.apache.org/mail-lists.html
On Wed, Jun 5, 2013 at 9:34 AM, Joseph Coleman <
joe.cole...@infinitecampus.com> wrote:
> Please remove me from this list
>
>
> On 6/5/13 8:32 AM, "Vimal Jain" wrote:
>
> >Ok.
> >I dont
Please remove me from this list
On 6/5/13 8:32 AM, "Vimal Jain" wrote:
>Ok.
>I dont have any batch read/write to hbase.
>
>
>On Wed, Jun 5, 2013 at 6:08 PM, Azuryy Yu wrote:
>
>> gc log cannot get by default. need some configuration. do you have some
>> batch read or write to hbase?
>>
>> --Se
Ok.
I dont have any batch read/write to hbase.
On Wed, Jun 5, 2013 at 6:08 PM, Azuryy Yu wrote:
> gc log cannot get by default. need some configuration. do you have some
> batch read or write to hbase?
>
> --Send from my Sony mobile.
> On Jun 5, 2013 8:25 PM, "Vimal Jain" wrote:
>
> > I dont h
I want to set a column to be the same value for all rows from the shell. I
can do it using put for a single row, but I'm not sure how to do it for all
rows.
How can I accomplish this?
Thanks.
Hello,
I wonder, why spawing an extra process for the different gateways (thrift,
stargate, avoc ...), when you can make your app discuss directly with
region servers (and zookeeper) ?
If your app is in Java you can use the native client so your app
communicates directly with the region servers.
There is no support like one CF can be replicated to one cluster while
another to another cluster.. In fact you can not specify peers where some
CF to be replicated. If the scope is given as 1 it gets replicated to all
peers..
See the issue HBASE-5002
-Anoop-
On Wed, Jun 5, 2013 at 6:04 PM, Sha
gc log cannot get by default. need some configuration. do you have some
batch read or write to hbase?
--Send from my Sony mobile.
On Jun 5, 2013 8:25 PM, "Vimal Jain" wrote:
> I dont have GC logs.Do you get it by default or it has to be configured ?
> After i came to know about crash , i checke
Anoop, Can you please clarify a bit.
-So we can specify replication at CF level but the scope 2 is not
supported, right? And we can replicate one CF to one slave cluster and
other CF to other slave cluster? Right? Thanks.
Regards,
Shahab
On Wed, Jun 5, 2013 at 12:37 AM, Anoop John wrote:
> Ye
I dont have GC logs.Do you get it by default or it has to be configured ?
After i came to know about crash , i checked which all processes are
running using "jps"
It displayed 4 processes , "namenode","datanode","secondarynamenode" and
"HQuorumpeer".
So i stopped dfs by running $HADOOP_HOME/bin/st
do you have GC log? and what you did during crash? and whats your gc
options?
for the dn error, thats net work issue generally, because dn received an
incomplete packet.
--Send from my Sony mobile.
On Jun 5, 2013 8:10 PM, "Vimal Jain" wrote:
> Yes.
> Thats true.
> There are some errors in all 3
Yes.
Thats true.
There are some errors in all 3 logs during same period , i.e. data , master
and region.
But i am unable to deduce the exact cause of error.
Can you please help in detecting the problem ?
So far i am suspecting following :-
I have 1GB heap (default) allocated for all 3 processes ,
there have errors in your dats node log, and the error time match with rs
log error time.
--Send from my Sony mobile.
On Jun 5, 2013 5:06 PM, "Vimal Jain" wrote:
> I don't think so , as i dont find any issues in data node logs.
> Also there are lot of exceptions like "session expired" , "slept m
I could not find any issue in namenode.
Here is the namenode log at time of the issue.
http://bin.cakephp.org/saved/103003
On Wed, Jun 5, 2013 at 5:00 PM, Ted Yu wrote:
> Have you looked at NameNode log ?
>
> The snippet you posted seems to imply issue with data block placement.
>
> Cheers
>
>
Have you looked at NameNode log ?
The snippet you posted seems to imply issue with data block placement.
Cheers
On Jun 5, 2013, at 4:12 AM, Vimal Jain wrote:
> I am running Hbase in pseudo distributed mode . So there is only one
> machine involved.
> I am using Hadoop version - 1.1.2 , Hbase
I am running Hbase in pseudo distributed mode . So there is only one
machine involved.
I am using Hadoop version - 1.1.2 , Hbase version - 0.94.7
On Wed, Jun 5, 2013 at 4:38 PM, Ted Yu wrote:
> How many region servers / data nodes do you have ?
>
> What Hadoop / HBase version are you using ?
>
How many region servers / data nodes do you have ?
What Hadoop / HBase version are you using ?
Thanks
On Jun 5, 2013, at 3:54 AM, Vimal Jain wrote:
> Yes.I did check those.
> But i am not sure if those parameter setting is the issue , as there are
> some other exceptions in logs ( "DFSOutputS
Yes.I did check those.
But i am not sure if those parameter setting is the issue , as there are
some other exceptions in logs ( "DFSOutputStream ResponseProcessor
exception " etc . )
On Wed, Jun 5, 2013 at 4:19 PM, Ted Yu wrote:
> There are a few tips under :
> http://hbase.apache.org/book.htm
There are a few tips under :
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
Can you check ?
Thanks
On Jun 5, 2013, at 2:05 AM, Vimal Jain wrote:
> I don't think so , as i dont find any issues in data node logs.
> Also there are lot of exceptions like "session expired" , "slept
I don't think so , as i dont find any issues in data node logs.
Also there are lot of exceptions like "session expired" , "slept more than
configured time" . what are these ?
On Wed, Jun 5, 2013 at 2:27 PM, Azuryy Yu wrote:
> Because your data node 192.168.20.30 broke down. which leads to RS do
Because your data node 192.168.20.30 broke down. which leads to RS down.
On Wed, Jun 5, 2013 at 3:19 PM, Vimal Jain wrote:
> Here is the complete log:
>
> http://bin.cakephp.org/saved/103001 - Hregion
> http://bin.cakephp.org/saved/103000 - Hmaster
> http://bin.cakephp.org/saved/103002 - Datano
Why there are so many miss for the index blocks? WHat is the block cache
mem you use?
On Wed, Jun 5, 2013 at 12:37 PM, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:
> I get your point Pankaj.
> Going thro the code to confirm it
> // Data index. We also read statistics about
https://issues.apache.org/jira/browse/HBASE-8691
On 6/4/13 6:11 PM, "Sandy Pratt" wrote:
>Haven't had a chance to write a JIRA yet, but I thought I'd pop in here
>with an update in the meantime.
>
>I tried a number of different approaches to eliminate latency and
>"bubbles" in the scan pipeline
Here is the complete log:
http://bin.cakephp.org/saved/103001 - Hregion
http://bin.cakephp.org/saved/103000 - Hmaster
http://bin.cakephp.org/saved/103002 - Datanode
On Wed, Jun 5, 2013 at 11:58 AM, Vimal Jain wrote:
> Hi,
> I have set up Hbase in pseudo-distributed mode.
> It was working fine
I get your point Pankaj.
Going thro the code to confirm it
// Data index. We also read statistics about the block index written
after
// the root level.
dataBlockIndexReader.readMultiLevelIndexRoot(
blockIter.nextBlockWithBlockType(BlockType.ROOT_INDEX),
trailer.getDataI
70 matches
Mail list logo