://issues.apache.org/jira/browse/ZEPPELIN-651
Commit:
https://github.com/apache/incubator-zeppelin/commit/1940388e3422b86a322fc82a0e7868ff25126804
Looking forward to feedback and suggestions for improvements.
Rajat Venkatesh
Engg. Lead
Qubole
Dear HBase experts,
I have a Hadoop cluster which has Hive, HBase installed along with other Hadoop
components. I am currently exploring ways to automate a data migration process
from Hive to HBase which involves new columns of data added ever so often. I
was successful in creating a HBase ta
ing-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/
J-D
On Mon, May 16, 2011 at 2:08 PM, Venkatesh wrote:
> Thanks J-D
>
> Using hbase-0.20.6, 49 node cluster
>
> The map reduce job involve a full table scan...(region size 4 gig)
> The job runs great for
scanner would expire. That's orthogonal tho.
You need to figure what you're blocking on, add logging and try to
jstack your Child processes for example.
J-D
On Thu, May 12, 2011 at 7:21 PM, Venkatesh wrote:
>
> Hi
> Using hbase-0.20.6
>
> mapreduce job starte
Hi
Using hbase-0.20.6
mapreduce job started failing in the map phase (using hbase table as input for
mapper)..(ran fine for a week or so starting with empty tables)..
task tracker log:
Task attempt_201105121141_0002_m_000452_0 failed to report status for 600
seconds. Killing
Region serve
s them (In his case he found that upping the handlers cured a
slow scan issue). You could study the 0.20.6 response times and then
when you upgrade to 0.90.2, check what its showing. That would at
least give us a clue as to where to start digging.
St.Ack
On Thu, Apr 21, 2011 at 8:21 PM, Ve
informative.
St.Ack
On Wed, Apr 20, 2011 at 10:41 AM, Venkatesh wrote:
> shell is no problems..ones/twos..i've tried mass puts from shell
> we cant handle our production load (even 1/3 of it)
> 700 mill per day is full load..same load we handled with absolutely no issues
i
suna
To: user@hbase.apache.org
Sent: Wed, Apr 20, 2011 4:30 pm
Subject: Re: java.lang.IndexOutOfBoundsException
On Wed, Apr 20, 2011 at 10:04 AM, Venkatesh wrote:
> On 0.90.2, do you all think using HTablePool would help with performance
problem?
What performance problems are you seein
ssage-
From: Stack
To: user@hbase.apache.org
Sent: Wed, Apr 20, 2011 1:30 pm
Subject: Re: hbase 0.90.2 - incredibly slow response
On Tue, Apr 19, 2011 at 11:58 AM, Venkatesh wrote:
>
> I was hoping that too..
> I don't have scripts to generate # requests from shell..I will t
specifying Integer.MAX_VALUE as maxSize along with config.
On Wed, Apr 20, 2011 at 10:17 AM, Venkatesh wrote:
> If I use default ..i can't share/pass my HBaseConfiguration object..atleast
> i don't see a constructor/setter..
> that would go against previous suggestion
>
>
>
ubject: Re: java.lang.IndexOutOfBoundsException
When using HTablePool, try not to define maxSize yourself - use the default.
On Wed, Apr 20, 2011 at 10:04 AM, Venkatesh wrote:
>
> Yeah you & J-D both hit it..
> I knew it's bad..I was trying anything & everything to solve the incredib
AM, Ted Yu wrote:
> I have seen this before.
> HTable isn't thread-safe.
>
> Please describe your usage.
>
> Thanks
>
>
> On Wed, Apr 20, 2011 at 6:03 AM, Venkatesh wrote:
>
>>
>> Using hbase-0.90.2..(sigh..) Any tip? thanks
>
gives you same lag?
St.Ack
On Tue, Apr 19, 2011 at 10:35 AM, Venkatesh wrote:
>
>
> Just upgraded to 0.90.2 from 0.20.6..Doing a simple put to table (< 100
> bytes
per put)..
> Only code change was to retrofit the HTable API to work with 0.90.2
>
> Init
Just upgraded to 0.90.2 from 0.20.6..Doing a simple put to table (< 100 bytes
per put)..
Only code change was to retrofit the HTable API to work with 0.90.2
Initializing HBaseConfiguration in servlet.init()...& reusing that config for
HTable constructor & doing put
Performance is very slow
deleteAllConnections works well for my case..I can live with this but not with
connection leaks
thanks for the idea
Venkatesh
-Original Message-
From: Ruben Quintero
To: user@hbase.apache.org
Sent: Wed, Apr 13, 2011 4:25 pm
Subject: Re: hbase -0.90.x upgrade - zookeeper
e -0.90.x upgrade - zookeeper exception in mapreduce job
Venkatesh, I guess the two quick and dirty solutions are:
- Call deleteAllConnections(bool) at the end of your MapReduce jobs, or
periodically. If you have no other tables or pools, etc. open, then no problem.
If you do, they'll
e hbase is running).
J-D
On Wed, Apr 13, 2011 at 12:04 PM, Venkatesh wrote:
> Reuben:
> Yes..I've the exact same issue now..& I'm also kicking off from another jvm
that runs for ever..
> I don't have an alternate solution..either modify hbase code (or) modify my
c
ions.
Am I missing a way to get that buried table's config, or another way to kill
the
orphaned connections?
- Ruben
From: Venkatesh
To: user@hbase.apache.org
Sent: Wed, April 13, 2011 10:20:50 AM
Subject: Re: hbase -0.90.x upgrade - zookeep
new Configuration is one new ZK
> connection). Upping your zookeeper max connection config should get
> rid of your issues since you only get it occasionally.
>
> J-D
>
> On Tue, Apr 12, 2011 at 7:59 AM, Venkatesh wrote:
>>
>> I get this occasion
I get this occasionally..(not all the time)..Upgrading from 0.20.6 to 0.90.2
Is this issue same as this JIRA
https://issues.apache.org/jira/browse/HBASE-3578
I'm using HBaseConfiguration.create() & setting that in job
thx
v
2011-04-12 02:13:06,870 ERROR Timer-0
org.apache.hadoop.hbase.mapre
Thanks St.Ack
Yes..I see these when map-reduce job is complete..but not always..I'll ignore
thanks..Getting close to 0.90.1 upgrade
-Original Message-
From: Stack
To: user@hbase.apache.org
Cc: Venkatesh
Sent: Thu, Apr 7, 2011 11:55 pm
Subject: Re: zookeper warning
I see lot of these warnings..everything seems to be working otherwise..Is this
something that can be ignored?
2011-04-07 21:29:15,032 WARN Timer-0-SendThread(..:2181)
org.apache.zookeeper.ClientCnxn - Session 0x0 for server :2181, unexpected
error, closing socket connection and attempti
Sorry about this..It was indeed an environment issue..my core-site.xml was
pointing to wrong hadoop
thanks for the tips
-Original Message-
From: Venkatesh
To: user@hbase.apache.org
Sent: Fri, Apr 1, 2011 4:51 pm
Subject: Re: row_counter map reduce job & 0.90.1
Yeah
A big thankyou from a hbase user (sorry for the spam..but deserves thanks)
-Original Message-
From: Jean-Daniel Cryans
To: user@hbase.apache.org
Sent: Sat, Apr 2, 2011 3:51 pm
Subject: Re: HBase wiki updated
2 Internets for you Doug, that's awesome!
Thx
J-D
On Apr 2,
TH.
But you shouldn't have to do the latter at least. Compare where it
works to where it doesn't. Something is different.
St.Ack
On Fri, Apr 1, 2011 at 9:26 AM, Venkatesh wrote:
> Definitely yes..It'all referenced in -classpath option of jvm of
tasktracker/jobtracker/da
stopping me in moving to 0.90.1
-Original Message-
From: Stack
To: user@hbase.apache.org
Sent: Fri, Apr 1, 2011 12:19 pm
Subject: Re: row_counter map reduce job & 0.90.1
On Fri, Apr 1, 2011 at 9:06 AM, Venkatesh wrote:
> I'm able to run this job from the hadoop
I'm able to run this job from the hadoop machine (where job & task tracker
also runs)
/hadoop jar /home/maryama/hbase-0.90.1/hbase-0.90.1.jar rowcounter
But, I'm not able to run the same job from
a) hbase client machine (full hbase & hadoop installed)
b) hbase server machines (ditto)
Get
20.6..
What are the region parameters? I tried encoded nam it did n't like..I tried
name of the form
tbl_name,st_key,,
That did n't work either..
thanks
-Original Message-
From: Stack
To: user@hbase.apache.org
Cc: Venkatesh
Sent: Thu, Mar 31, 2011 1:36 am
S
es it not complete?
St.Ack
On Wed, Mar 30, 2011 at 4:13 AM, Venkatesh wrote:
> Yes..st.ack..overlapping..& one of them has no data..
> there are too many of them about 800 or so..
> there are some with holes too..
>
>
>
>
>
>
>
>
>
&
n Wed, Mar 30, 2011 at 4:13 AM, Venkatesh wrote:
> Yes..st.ack..overlapping..& one of them has no data..
> there are too many of them about 800 or so..
> there are some with holes too..
>
>
>
>
>
>
>
>
>
>
> -Original Message-
>
Thanks Lucas..I'll give it a try
-Original Message-
From: Lukas
To: user@hbase.apache.org
Sent: Wed, Mar 30, 2011 4:19 am
Subject: Re: hole in META
Sorry for any inconvenience. This was in reply of
http://mail-archives.apache.org/mod_mbox/hbase-user/201103.mbox/%3c8cdbca99c
ping regions? Can you try merging them with
merge tool? Else, study whats in hdfs. One may have nothing in it
(check sizes). It might just be reference files only. If so, lets go
from there. And I describe how to merge.
St.Ack
On Tue, Mar 29, 2011 at 9:25 PM, Venkatesh wrote:
> I
From: Stack
To: user@hbase.apache.org
Sent: Tue, Mar 29, 2011 12:55 pm
Subject: Re: hole in META
On Tue, Mar 29, 2011 at 9:09 AM, Venkatesh wrote:
> I ran into missing jar with hadoop jar file when running a map reduce..which
i could n't fix it..That is the only known issue with up
for you.
J-D
On Tue, Mar 29, 2011 at 2:33 PM, Venkatesh wrote:
> Thanks J-D
>
> We have way too much data & it won't fit in 1 region.Is Import smart enough
create
> reqd # of regions?
>
> Cld u pl. elaborate on pre-split table creation? steps?
>
&g
to create a pre-split table.
J-D
On Tue, Mar 29, 2011 at 2:27 PM, Venkatesh wrote:
>
>
>
> Hi,
> If I export existing table using Export MR job, truncate the table, increase
region size, & do a Import
> will it make use of the new region size?
>
> thanks
> V
>
>
>
Hi,
If I export existing table using Export MR job, truncate the table, increase
region size, & do a Import
will it make use of the new region size?
thanks
V
s)..but it increases the # of regions
which is the down side..
(we are already way over recommended limit 2800 regions per node)
-Original Message-
From: Stack
To: user@hbase.apache.org
Sent: Tue, Mar 29, 2011 12:55 pm
Subject: Re: hole in META
On Tue, Mar 29, 2011 at 9:09 AM,
ll upgrade
BTW, is it better to fix existing holes using add_table.rb before the upgrade?
(or) upgrade takes care missing holes?
-Original Message-
From: Stack
To: user@hbase.apache.org
Sent: Tue, Mar 29, 2011 11:59 am
Subject: Re: hole in META
On Tue, Mar 29, 2011 at 7:3
Hi
Using hbase-0.20.6..This has happened quite often..Is this a known issue in
0.20.6 that
we would n't see in 0.90.1 (or) see less of?
..Attempt to fix/avoid this earlier times by truncating table, running
add_table.rb before
What is the best way to fix this in 0.20.6? Now it's there in mo
e below is pretty basic error. Reference the jar that is actually
present on your cluster.
St.Ack
On Wed, Mar 16, 2011 at 3:50 PM, Venkatesh wrote:
> yeah..i was aware of that..I removed that & tried with hadoop-0.20.2-core.jar
as I was n't ready to upgrade hadoop..
>
>
up its name in the lib/ directory of the distribution
(comes with a rev #) :)
On Thu, Mar 17, 2011 at 2:33 AM, Venkatesh wrote:
> Thanks St.Ack..I'm blind..Got past that..
> Now I get for hadoop-0.20.2-core.jar
>
> I've removed *append*.jar all over the place &am
0.1 ships with zookeeper-3.3.2, not with 3.2.2.
St.Ack
On Wed, Mar 16, 2011 at 8:05 AM, Venkatesh wrote:
>
> Does anyone how to get around this? Trying to run a mapreduce job in a
cluster..The one change was hbase upgraded to 0.90.1 (from 0.20.6)..No code
change
>
>
> java.i
Does anyone how to get around this? Trying to run a mapreduce job in a
cluster..The one change was hbase upgraded to 0.90.1 (from 0.20.6)..No code
change
java.io.FileNotFoundException: File
/data/servers/datastore/mapred/mapred/system/job_201103151601_0363/libjars/zookeeper-3.2.2.jar
does
Hi
When I upgraded to 0.90.1, mapreduce fails with exception..
system/job_201103151601_0121/libjars/hbase-0.90.1.jar does not exist.
I have the jar file in classpath (hadoop-env.sh)
any ideas?
thanks
hing
else than HBase, then you should only use 1 zk server and collocate it
with the master and the namenode, then use those 3 machines as region
servers to help spread the region load.
J-D
On Thu, Feb 10, 2011 at 12:35 PM, Venkatesh wrote:
> Thanks J-D..
> I was quite happy in the 1s
..but did help the real puts
-Original Message-
From: Ted Dunning
To: user@hbase.apache.org
Sent: Thu, Feb 10, 2011 3:45 pm
Subject: Re: region servers shutdown
Are your keys sequential or randomized?
On Thu, Feb 10, 2011 at 12:35 PM, Venkatesh wrote:
> iii) Process
d how many regions do they serve? Are you using lots
of families per table? Are you using LZO compression?
Thanks for helping us helping you :)
J-D
On Thu, Feb 10, 2011 at 11:32 AM, Venkatesh wrote:
> Thanks J-D..
> Can't believe i missed that..I have had it before ..i did l
node logs a the time
of the outage. Very often it's caused by either ulimit or xcievers
that weren't properly configured, checkout
http://hbase.apache.org/notsoquick.html#ulimit
J-D
On Thu, Feb 10, 2011 at 10:42 AM, Venkatesh wrote:
>
>
>
> Hi
> I've ha
Hi
I've had this before ..but not to 70% of the cluster..region servers all
dying..Any insight is helpful.
Using hbase-0.20.6, hadoop-0.20.2
I don't see any error in the datanode or the namenode
many thanks
Here's the relevant log entires
..in master...
Got while writing region XXlog j
g
regions. They might inspire. Also look at the Merge.java class. See
how it edits .META. after merging two adjacent regions to create a new
region that spans the key space of the two old adjacent regions.
St.Ack
On Fri, Jan 28, 2011 at 12:29 PM, Venkatesh wrote:
>
>
>
&
Is there a script?
thanks
G for
hbase and re-run the job to hopefully get more information.
J-D
On Wed, Jan 26, 2011 at 8:44 AM, Venkatesh wrote:
>
>
>
> Using 0.20.6..any solutions? Occurs during mapper phase..will increasing
retry count fix this?
> thanks
>
&
Hi Sean:
Thx
Size of column family is very small < 100 bytes
Investigating potential bottleneck spot..Our cluster is small (relatively
speaking)..10 node
Our hardware is high end (not commodity)
venkatesh
-Original Message-
From: Sean Bigdatafun
To: user@hbase.apache.
tablename)' in your client or are you doing
> 'new HTable(conf, tablename)' in your client code? Do the latter if
> not -- share the configuration with HTable instances.
>
> St.Ack
>
> On Mon, Oct 11, 2010 at 10:47 PM, Venkatesh wrote:
> >
> >
> >
&
..Is that reasonable for a table size
of 10 mill records?
hbase.client.scanner.caching - If set in hbase-site.xml, Scan calls should
pick that up correct?
thanks
venkatesh
-Original Message-
From: Jean-Daniel Cryans
To: user@hbase.apache.org
Sent: Thu, Oct 14, 2010
ered write without
code change)
thx
venkatesh
rvlet.java:617)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
-Original Message-
From: Venkatesh
To: user@hbase.apache.org
Sent: Mon, Oct 11, 2010 2:35 pm
Subject: hbase.client.retries.number
HBase was seamless for first couple of weeks..now all kinds of issues in
production :) fun
I can't do any put (new)
thanks
venkatesh
Some of the region servers suddenly dying..I've pasted relevant log lines..I
don't see any error in datanodes
Any ideas?
thanks
venkatsh
.
2010-10-10 12:55:36,664 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: java.io.IOException: Unable to create new block.at
o
his normal?
thanks
venkatesh
PS. setCaching(100) ..did n't make a dent in performance
Thanks J-D
I'll hookup Ganglia (wanting but kept pushing back..) & get back
V
-Original Message-
From: Jean-Daniel Cryans
To: user@hbase.apache.org
Sent: Wed, Oct 6, 2010 12:22 pm
Subject: Re: HBase map reduce job timing
> Also, do you think if I query using rowkey instead
Also, do you think if I query using rowkey instead of hbase time stamp..it
would not kick off that many tasks..
since region server knows the exact locations?
thanks
venkatesh
-Original Message-
From: Venkatesh
To: user@hbase.apache.org
Sent: Wed, Oct 6, 2010 8:53 am
Ahh ..ok..That makes sense
I've a 10 node cluster each with 36 gig..I've allocated 4gig for HBase Region
Servers..master.jsp
reports used heap is less than half on each region server.
I've close to 800 regions total..Guess it needs to kick off a jvm to see if
data exists
in all regions..
ts less free memory (couple of gig.) though box has 36 gig total.. I
don't quite trust
top since cached blocks don't show up under free column even if no process is
running..
venkatesh
-Original Message-
From: Jean-Daniel Cryans
To: user@hbase.apache.org
Sent:
.org
Sent: Tue, Oct 5, 2010 11:14 pm
Subject: Re: HBase map reduce job timing
It'd be more useful if we knew where that data is coming from, and
where it's going. Are you scanning HBase and/or writing to it?
J-D
On Tue, Oct 5, 2010 at 8:05 PM, Venkatesh wrote:
>
>
>
> So
t I
can do is pointing to the existing documentation
http://wiki.apache.org/hadoop/PerformanceTuning
J-D
On Tue, Oct 5, 2010 at 7:12 PM, Venkatesh wrote:
>
>
>
> I've a mapreduce job that is taking too long..over an hour..Trying to see
what can a tune
> to to bring it down
a why that would be?...
..map phase takes about couple of minutes..
..reduce phase takes the rest..
..i'll try increasing # of reduce tasks..Open to other other suggestion for
tunables..
thanks for your input
venkatesh
the web interface..reports 10gig
That seems odd..Any ideas on what could be taking up space?..I don't have
permission to look the entire hdfs..yet
Just thought i'll ask the group
thanks
venkatesh
Don't know if this helps..but here are couple of reasons when I had the issue
& how i resolved it
-> If zookeeper is not running (or do not have the quorum) in a cluster setup,
hbase does not go down..bring up zookeeper
-> Make sure pid file is not under /tmp...somtimes files get cleaned out of
application in HADOOP_CLASSPATH
..for the mapreduce to run in a cluster?
thanks
venkatesh
..to make this classpath effective
Is there a force shutdown option? (other than kill -9)..?
venkatesh
-Original Message-
From: Jean-Daniel Cryans
To: user@hbase.apache.org
Sent: Fri, Aug 27, 2010 12:10 am
Subject: Re: jobtracker.jsp
HBase needs to know about the job tracker,
hat's the log in your client side ?
On Thu, Aug 26, 2010 at 6:23 PM, Venkatesh wrote:
>
>
>
> I'm running map/reduce jobs from java app (table mapper & reducer) in true
distributed
> mode..I don't see anything in jobtracker page..Map/reduce job runs fine..Am I
mi
I'm running map/reduce jobs from java app (table mapper & reducer) in true
distributed
mode..I don't see anything in jobtracker page..Map/reduce job runs fine..Am I
missing some config?
thanks
venkatesh
;JMX" (tomcat)
It works fine..when I have periodic scheduler running automatically in the
background..
Also..I see this error on a 64-bit machine..& not on 32bit machines..
Wierd..wonder if anyone has any insights?..Setup looks identical..
thanks
venkatesh
I wrestled with that idea of time bounded tables..Would it make it harder to
write code/run map reduce
on multiple tables ? Also, how do u decide to when to do the cut over (start of
a new day, week/month..)
& if u do how to process data that cross those time boundaries efficiently..
Guess that
when do I call close()..upon every operation (put/get/..) ? to avoid
memory leaks
thanks
venkatesh
76 matches
Mail list logo
