Just to close the loop on this ordeal…
I started by clearing /hbase/splitWAL in ZK and restarting all the RS and
the HM. This didn’t change anything.
On Wed, Jan 27, 2016 at 8:42 AM, tsuna <tsuna...@gmail.com> wrote:
> 16/01/27 16:33:39 INFO namenode.FSNamesystem: Recovering [Lease.
Resending to u...@hadoop.apache.org now that I’m subscribed to that list.
On Thu, Jan 28, 2016 at 10:50 AM, tsuna <tsuna...@gmail.com> wrote:
> Just to close the loop on this ordeal…
>
> I started by clearing /hbase/splitWAL in ZK and restarting all the RS and
> the HM.
%252C9104%252C1452811286448.default.1453728137282
--
Benoit "tsuna" Sigoure
m one unlucky region.
> The BadVersion is probably because task has been assumed by another because
> lease recovery taking too long.
>
> I see 1453728374800 is still in the list you paste from Master later in
> this thread.
>
> Try moving it aside. Might have to restart master for it to notice the
> move-aside... Try not restarting first.
--
Benoit "tsuna" Sigoure
ideas / suggestions?
--
Benoit "tsuna" Sigoure
Wed Jan 27 05:51:58 UTC 2016 Starting regionserver on
r12s4.sjc.aristanetworks.com
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (
?
This has nothing to do with the OS cleaning up files or anything. I
can start HBase, kill it, restart it within the span of less than 2
minutes, and the restart fails.
--
Benoit tsuna Sigoure
from my iPhone
On 18 May 2015, at 05:55, tsuna tsuna...@gmail.com wrote:
Hi all,
For testing on my laptop (OSX with JDK 1.7.0_45) I usually build the
latest version from branch-1.0 and use the following config:
configuration
property
namehbase.rootdir/name
valuefile:///tmp
(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2002)
I noticed that this has something to do with the ZooKeeper data. If I
rm -rf $TMPDIR/hbase-tsuna/zookeeper then I can start HBase again.
But of course HBase won’t work properly because while
not schedule RPCs known to be NSRE'd for retry when they are
first issued.
Thanks to the people above who contributed to this release.
--
Benoit tsuna Sigoure
the first RC of any new major release. :P
--
Benoit tsuna Sigoure
If this RC sinks, can you please include this one in 0.98.0:
https://issues.apache.org/jira/browse/HBASE-10422
--
Benoit tsuna Sigoure
a value at the specified timestamp.
--
Benoit tsuna Sigoure
.
--
Benoit tsuna Sigoure
forcefully
removing the buggy coprocessor? Or should we maybe add some sort of a
cleanUp() method to give a chance to the coprocessor to save face and
die gracefully?
--
Benoit tsuna Sigoure
.
--
Benoit tsuna Sigoure
bugs prevented AsyncHBase from being able to scan -ROOT-
or hbase:meta.
- With 0.96, no user-generated RPC could be directed at hbase:meta
Pre-compiled JAR: http://tsunanet.net/~tsuna/asynchbase/asynchbase-1.5.0-rc2.jar
Also available for Maven users as 1.5.0-SNAPSHOT on OSS Snapshots.
Source
ipc.Client.CONNECTION_CONTEXT_CALL_ID is signed (i.e. -3). Contributed
by
Arun C. Murthy.
git-svn-id:
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.1-beta@152
3887
13f79535-47bb-0310-9956-ffa450edef68
Cosmin
On 29/10/13 01:19, tsuna tsuna...@gmail.com wrote
code doesn't belong in the
repository, except under rare circumstances. Developers are expected
to have the right tools at their disposal. If you have multiple
versions of protoc installed on your system, simply doing make
PROTOC=path/to/protobuf-2.5.0/protoc would do the trick.
--
Benoit tsuna
)
at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2768)
... 5 more
--
Benoit tsuna Sigoure
.cs1 unzip -l hadoop-common-2.1.0-beta.jar~ | grep
org.apache.hadoop.util.PlatformName
1707 08-15-2013 20:44 org/apache/hadoop/util/PlatformName.class
WTF.
--
Benoit tsuna Sigoure
On Mon, Oct 28, 2013 at 4:26 PM, Ted Yu yuzhih...@gmail.com wrote:
See this thread:
Yes, I've seen this thread (that's the one I referred to in my first
post). Why would it not work with 2.1.1?
--
Benoit tsuna Sigoure
…
--
Benoit tsuna Sigoure
the release?
--
Benoit tsuna Sigoure
release?
--
Benoit tsuna Sigoure
ZooKeeper.
Pre-compiled JAR: http://tsunanet.net/~tsuna/asynchbase/asynchbase-1.5.0-rc1.jar
Source: https://github.com/tsuna/asynchbase
Javadoc:
http://tsunanet.net/~tsuna/asynchbase/1.5.0/org/hbase/async/HBaseClient.html
$ git diff --stat v1.4.1.. | tail -n 1
70 files changed, 4824 insertions(+), 487
On Thu, May 2, 2013 at 1:49 PM, Andrew Purtell apurt...@apache.org wrote:
In that blog post Benoît does a fair amount of showing off to end up with
So this segfault remains a mystery so far :-)
Yeah I never got to the bottom of it. :-/
--
Benoit tsuna Sigoure
a clue.
--
Benoit tsuna Sigoure
(s) in 1.0830 seconds
I don't have any local changes. Anyone else seeing this? It doesn't
seem to impact functionality (i.e. my table was created properly).
--
Benoit tsuna Sigoure
wanna ask people on the asynchbase mailing list.
http://groups.google.com/group/asynchbase
If your application is multi-threaded, chances are you'll see a nice
speedup, because asynchbase is thread-safe and uses very few threads,
unlike HTable + HTablePool.
--
Benoit tsuna Sigoure
Software Engineer
of data per eventid, then putting the uuid in
between the eventid and the -mm-dd will screw up data locality
anyway. But the exact details depend on how you pick the uuid.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
are evenly spread across all your physical servers. You can
do this from the HBase shell. In the future, when HBase does proper
load balancing of regions, this won't be necessary anymore.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
. Also if you can
explain what led you to believe that the master is in the data path of
clients, maybe we can help address the source of this common
misconception.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
level might shed some
light on where the problem comes from.
https://github.com/stumbleupon/asynchbase
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
% of the cases, you're better
off with TCP anyway, because your reliable protocol is almost
certainly not going to be as reliable and efficient as, say, Linux's
TCP implementation.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
implementation. It certainly did for me.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
because our
cluster is bigger than 8 RS).
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
for realtime search?
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
this to build a river to stream updates from HBase to ES. An
another alternative would be to have a sink that uses HBase
replication to replicate edits to ES.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
, an alternative HBase
client that's fully asynchronous and non-blocking and written from the
ground-up to be thread safe. It performs much better than HTable for
high-throughput low-latency multi-threaded applications:
github.com/stumbleupon/asynchbase /plug
--
Benoit tsuna Sigoure
Software Engineer
applications, look at asynchbase:
http://github.com/stumbleupon/asynchbase
OpenTSDB uses it and I'm able to push 20 edits per second to 3
RegionServers.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
far fewer threads
and has less lock contention.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
of
versions appropriately for your needs.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
balancing
won't be as good as the insert balancing since some queries are much more
popular.
I believe this leads to better use of the blockcache though.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
?
No. You need to manually create a secondary index where the key is
made from `Long.MAX_VALUE - timestamp' so that you can scan that
secondary index backwards in time.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
and there, and that would be all.
http://www.elasticsearch.org/
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
on multiple points within one row) I recommend using a tall
table, since large rows will become unmanageable, especially if they
keep growing forever (and HBase cannot split a row that's become too
big).
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
it makes on HBase... You could literally run a JVM
with a 16GB heap on just 16 pages. It sounds almost ridiculous :)
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
like, I'm guessing you'll see that the slow
responses you're seeing are indeed really slow and that the problem
comes from the server-side.
What's your HBase cluster like? How many nodes, what hardware, what
configuration? What's you block cache hit ratio?
--
Benoit tsuna Sigoure
Software
.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
higher lock
contention (and causes more context switches). Other fields are:
syscalls = number of system calls executed by the process, rss =
actual memory used (resident set size) in KB before exiting main.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
and
uses a fixed number of threads (2 * N where N is the number of
hardware threads). That's how I solved my scalability problems in
OpenTSDB.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
/asynchbase) as it's an alternative
HBase client that is entirely asynchronous and non-blocking. Javadoc
is at http://su.pr/1PJCSY
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
small data items.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
Just to add on what others said, at StumbleUpon we typically see a
99th percentile lower than 100ms for simple get/put operations.
OpenTSDB also uses HBase with an interactive GWT interface and relies
on sub-second response times for queries involving many thousands of
rows.
--
Benoit tsuna
http://www.tsunanet.net/~tsuna/async/api/com/stumbleupon/async/Deferred.html
Deferred is a very powerful API for any kind of asynchronous processing.
So overall the code remains the same except that:
* You use callbacks to get the result of your operation asynchronously.
* You don't give a whole
interface. This way everything will keep using
127.0.0.1. This has worked well for me to develop on my laptop.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
client, while still
allowing for higher throughput through batching of edits.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
-effective than 10GbE or
Infiniband.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
are sorted, so sequential scanning is efficient (thanks to data
locality).
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
by creating different HBaseClient instances.
No configuration file needed. You only need a single HBaseClient
object per cluster, no matter how many tables you're going to use.
The Javadoc is at http://su.pr/1WGqav
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
/stumbleupon/asynchbase), an alternative HBase
client library that was written to be thread-safe from the ground up.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
in existing ones?
Is your pseudo-distributed HBase running on a single machine? If yes,
why not use a non-distributed HBase setup (without HDFS)?
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
contention?
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
machine is a very slow VM
so it doesn't have a lot of horsepower). This client is especially
good for throughput oriented workloads and was written to be
thread-safe from the ground up (unlike HTable).
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
worth of datapoints).
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
at the IO utilization on your boxes and maybe also
network utilization. How many disks per box do you have? What type
of disks are they? Have you looked at the output of iostat -xkd 1
while the cluster was performing poorly? Do you see a lot of iowait
(wa column in top)?
--
Benoit tsuna
companies that are doing it aren't willing to talk too much about it?
I also like ASA Computers Twin Servers (2 nodes in 1U):
http://www.asaservers.com/showpages.asp?pid=1280
Good bang for the buck.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
missing something?
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
. The code would also be simpler and more efficient.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
-blocking fashion.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
On Thu, Sep 9, 2010 at 5:10 AM, MauMau maumau...@gmail.com wrote:
- Original Message - From: tsuna tsuna...@gmail.com
Sent: Thursday, September 09, 2010 7:52 AM
If the server side latency consistently remains around 5-15ms but the
client side latency shoots up through the roof, you may
be interested in implementing
native HBase clients in non-Java languages, so I thought I'd pass this
around to save their time.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
On Wed, Sep 8, 2010 at 2:22 AM, tsuna tsuna...@gmail.com wrote:
I use Netty extensively, and for a long time I wanted to have a fully
asynchronous / non-blocking and thread-safe HBase client. So I ended
up writing one from scratch, which I just released at:
http://github.com/stumbleupon
to have
moving latency histograms for every RPC type I send and such, so I can
hook this up to my monitoring. I already have some code for
histograms and percentiles and whatnot, but it will be open sourced in
a separate package.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
pattern for non-blocking network programming
but would look to be useful beyond this use case, for example
implementing a server core.
If you would like to read more about Deferred, I encourage you to read
http://tsunanet.net/~tsuna/async/api/com/stumbleupon/async/Deferred.html
(which has pointers
.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
finally have a sane and reliable distributed
filesystem we can build upon.
--
Benoit tsuna Sigoure
Software Engineer @ www.StumbleUpon.com
79 matches
Mail list logo
