solutions and it worked fine for my application. However I'm not
sure what changing the behavior would do to other's applications especially
those that use SecureHadoop.
Thanks,
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
);
}
}
connection.addCall(call);
On Fri, Mar 29, 2013 at 11:41 AM, Jeff Whiting je...@qualtrics.com
wrote:
After noticing a lot of threads, I turned on debugging logging for
hbase
client and saw this many times counting up constantly:
HBaseClient:531 - IPC Client (687163870) connection
, Jeff Whiting je...@qualtrics.com
wrote:
I am using cdh4.1.3 which roughly maps to 0.92.1 with patches.
~Jeff
On Fri, Mar 29, 2013 at 1:40 PM, ramkrishna vasudevan
ramkrishna.s.vasude...@gmail.com wrote:
Nice one.. Good find.
On Sat, Mar 30, 2013 at 12:30 AM, Ted Yu
be getting a
NPE.
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
.
~Jeff
On 10/30/2012 10:45 PM, ramkrishna vasudevan wrote:
Are you writing fat cells?
Did you try raising the heap size? and see if still it is crashing?
Regards
Ram
On Wed, Oct 31, 2012 at 6:10 AM, Jeff Whiting je...@qualtrics.com mailto:je...@qualtrics.com
wrote:
So I'm looking
+ storefileindex. But in our environment, I have configured max heap to 20g
, heap used memory is sometimes up to 19g. At that moment, storefileindex
was 8g, blockcache was 1g, memstore was 1g. What's the rest 9g memory ?
Thanks in advance.
--
Jeff Whiting
Qualtrics Senior Software Engineer
je
/3822097/rolling-garbage-collector-logs-in-java
As for profiling memory dumps, jprofiler7, yourrkit, etc.. YMMV.
On Thu, Nov 1, 2012 at 10:01 AM, Jeff Whiting je...@qualtrics.com wrote:
We don't have GC logging enabled (we did but gc.log would begin filling up
the hdd and there was no way
of an
replication chain A - B - C where this cluster is B. So can we tell if it is running out of
memory because it is being replicated too? Or because it is trying to replicate somewhere else.
Thanks,
~Jeff
On 10/30/2012 11:39 PM, Stack wrote:
On Mon, Oct 29, 2012 at 3:55 PM, Jeff Whiting je
. I'm a little hazy when processData is called
and how it fits in the whole replication pipeline.
Since the region servers are just replaying wal logs to do the replication it seems like the memory
footprint could be made to be very minimal.
~Jeff
On 11/1/2012 5:44 PM, Jeff Whiting wrote:
So
in it as
defined by (ipc.server.max.callqueue.size).
~Jeff
On 11/1/2012 5:44 PM, Jeff Whiting wrote:
So this is some of what I'm seeing as I go through the profiles:
(a) 2GB - org.apache.hadoop.hbase.io.hfile.LruBlockCache
This looks like it is the block cache and we aren't having any
store files are
created?
The no of blocks getting cached will give you an idea too..
Regards
Ram
On Tue, Oct 30, 2012 at 4:25 AM, Jeff Whiting je...@qualtrics.com wrote:
We have 6 region server given 10G of memory for hbase. Each region server
has an average of about 100 regions and across
isn't very detailed so I'm guess it is like 98-99%).
Block cache size stays at about 2GB.
~Jeff
On 10/30/2012 6:21 PM, Jeff Whiting wrote:
We have no coprossesors. We are running replication from this cluster to
another one.
What is the best way to see how many store files we have? Or checking
is going. I also have jmx setup on each
region server and can connect to it that way.
Thanks,
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
On Wed, Sep 5, 2012 at 5:53 PM, Jeff Whiting je...@qualtrics.com wrote:
hmm. So if we are on 0.92 what suggestion would you have to prevent the
problem?
~Jeff
On 9/5/2012 5:23 PM, Himanshu Vashishtha wrote:
Number of PRI handlers are governed by
hbase.regionserver.metahandler.count; default
). Can you
throttle your load?
Replication is evolving; I'd say update if you can (or backport the jiras?).
Himanshu
On Wed, Sep 5, 2012 at 5:53 PM, Jeff Whiting je...@qualtrics.com
mailto:je...@qualtrics.com wrote:
hmm. So if we are on 0.92 what suggestion would you have
waiting on
condition
JNI global references: 1558
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
to look for other instances of the object monitor, I
don't find any. I see this for each instance of BLOCKED (Or at least,
the two or three I checked).
Whats your OS?
St.Ack
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
,
the two or three I checked).
Whats your OS?
St.Ack
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
occurring (i don't know about your load etc).
Another related jira is HBase-6550.
Some more context for your use case:
http://search-hadoop.com/m/WHkTxWj0MW/himanshu+vashisthasubj=Re+Long+running+replication+possible+improvements
On Wed, Sep 5, 2012 at 5:18 PM, Jeff Whiting je...@qualtrics.com
data nodes when running the export? Are you running too
many mappers per node and starving hbase?
On 7/24/12 2:52 PM, Jeff Whiting je...@qualtrics.com wrote:
What would cause a scanner timeout exception? Is hdfs too slow? Do I
just increase the scanner
timeout or is there a better approach
)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1000)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1100)
... 12 more
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
for porting it to
other branches.
On Wed, Jun 27, 2012 at 4:15 PM, Jeff Whiting je...@qualtrics.com mailto:je...@qualtrics.com
wrote:
Looking at HBASE-6284 it seems that deletes are not batched at the
regionserver level so that
is the reason for the performance degradation. Additionally
-archives.apache.org/mod_mbox/hbase-user/201206.mbox/%3CE83D30E8F408F94A96F992785FC29D82063395D6@s2k3mntaexc1.mentacapital.local%3E).
Any thoughts on why it is slower and how I can speed it up?
Thanks,
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
be the issue here.
Sent from my iPhone
On Jun 27, 2012, at 2:04 PM, Jeff Whiting je...@qualtrics.com wrote:
I'm struggling to understand why my deletes are taking longer than my
inserts. My understanding is that a delete is just an insertion of a
tombstone. And I'm deleting the entire row.
I do
to be created and old data imported
into this table.
Is this correct?
Thanks,
Otis
Performance Monitoring for Solr / ElasticSearch / HBase -
http://sematext.com/spm
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
Good to know thanks for the feedback!
On 5/10/2012 10:32 AM, Jean-Daniel Cryans wrote:
extensively, it's safe to use if your tables aren't
splitting like mad eg don't try to do it during a massive import
because you forgot to set the memstore size.
--
Jeff Whiting
Qualtrics Senior Software
.
--
View this message in context:
http://old.nabble.com/Speeding-up-HBase-read-response-tp33635226p33654666.html
Sent from the HBase User mailing list archive at Nabble.com.
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
the shell -- is it
still slow? If so, it means slowness is from accessing hfiles. Try
copying the region content out and rig up a little harness to bring
the region in a context free from the running cluser. See TestHRegion
for sample code on how to stand up a HRegion instance.
St.Ack
--
Jeff
Is there way to profile a specific get request to see where the time is spent (e.g. checking
memstore, reading from hdfs, etc)?
We are running into a problem where a get after a delete goes really slow. We have a row that has
between 100 to 256 MB of data in it across a couple hundred
)
... 2 more
Caused by: java.io.IOException: All datanodes 10.49.29.92:50010 are bad.
Aborting...
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145)
... 2 more
Caused by: java.io.IOException: All datanodes 10.49.29.92:50010 are bad.
Aborting...
--
Jeff Whiting
Qualtrics Senior Software Engineer
je
and successfully serving data for 15+ minutes? So when
I restart the last node it just gives up? fsck says that everything is healthy and fully replicated.
~Jeff
On 2/6/2012 11:28 AM, Jeff Whiting wrote:
I was increasing the storage on some of my data nodes and thus had to do a restart of the data
node
.**java:25)
at java.lang.reflect.Method.**invoke(Method.java:597)
at org.apache.hadoop.hbase.**regionserver.wal.**
SequenceFileLogWriter.sync(**SequenceFileLogWriter.java:**145)
... 2 more
Caused by: java.io.IOException: All datanodes 10.49.29.92:50010 are bad.
Aborting...
--
Jeff Whiting
.**java:25)
at java.lang.reflect.Method.**invoke(Method.java:597)
at org.apache.hadoop.hbase.**regionserver.wal.**
SequenceFileLogWriter.sync(**SequenceFileLogWriter.java:**145)
... 2 more
Caused by: java.io.IOException: All datanodes 10.49.29.92:50010 are bad.
Aborting...
--
Jeff Whiting
));
scan.setFilter(filter);
I am using the Cloudera 0.09.4 release, and a cluster of 3 machines on
EC2.
I think that this should be really fast, but it is not. Any advice on
how to debug/speed it up?
Thanks
Peter
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
15 12
http://www.gbif.org
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
. HDFS went away around this time? What version of hdfs?
St.Ack
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
we
aren't running anything, is hbase doing something that could cause those spikes?
Thanks,
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
(HBaseRPC.java:257)
... 3 more
Please note the hadoop fsck / says the file system is healthy.
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
be used to relay requests between the clusters?
Any thoughts would be appreciated.
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
to
work over a WAN or is it for LAN only? Can some kind of gateway machine be
used to relay requests between the clusters?
Any thoughts would be appreciated.
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
--
Jeff Whiting
Qualtrics Senior Software Engineer
je
it is gpl and cannot be included with hbase directly.
~Jeff
On 9/14/2011 11:34 AM, Joey Echeverria wrote:
ere was an earlier thread where some people noted
faster HBase performance using gzip compared to snappy. I haven't done
my own testing, so it might be worth trying out those two and
--
Jeff
(LinuxDebuggerLocal.java:51)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
--
Jeff Whiting
Qualtrics
)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
I should be looking to see why the balancer isn't running
automatically?
Thanks,
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
so
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
thread dumps on both the masters and region servers during this
time and did not see anything interesting. I'm using 0.90.3-CDH3U1.
Anyone have a suggestion on where to look next?
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
results in a failure, the
connection to the zookeeper ensamble is droped...
the work around to restart the whole application after 30 MR jobss is not
very elegant... :-(
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
.
No. The behavior you describe would come with replication of 2, not 1.
St.Ack
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
MapR to the tail of the last
paragraph (with the 'free as in free beer' caveat).
St.Ack
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
, because we are not doing multiple GETs as
most of the logic will be done on the backend.
-Jack
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
!
--
Laurent HATIER
Étudiant en 2e année du Cycle Ingénieur à l'EISTI
--
Laurent HATIER
Étudiant en 2e année du Cycle Ingénieur à l'EISTI
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
in diagnostic post mortem use cases.
Just so we're clear, we'd be talking about a new operation, right? Because
today's increment returns the incremented value, and some uses (like generating
unique values) do require that.
joe
--
Jeff Whiting
Qualtrics Senior Software Engineer
je
Put(Bytes.toBytes(id)));
arrayPut.get(arrayPut.size() - 1).add(FAMILY_GEOLOC, QUALIFIER_REGION,
Bytes.toBytes(regionCode)); table.put(arrayPut);
--
Laurent HATIER
Étudiant en 2e année du Cycle Ingénieur à l'EISTI
--
Laurent HATIER
Étudiant en 2e année du Cycle Ingénieur à l'EISTI
--
Jeff
!
Frank Liu
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
designing a data model which currently has
five column families. I expect each of those column families to have divergent
read and write patterns. Do you think I should look for ways to reduce the
number of CFs?
Thanks,
Leif Wickland
--
Jeff Whiting
Qualtrics Senior Software Engineer
je
and occupancy thresholds but that hasn't been necessary for the HBase
workloads so far:
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
.
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
,,1302141740486.010a5ae704ed53f656cbddb8e489162a.; next
sequenceid=1
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
out eventually?
Thanks,
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
or will it
figure
things out eventually?
Thanks,
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
namehbase.hregion.memstore.block.multiplier/name
value3/value
/property
property
namehbase.hstore.blockingStoreFiles/name
value14/value
/property
property
namehbase.hstore.compactionThreshold/name
value5/value
/property
Any suggestions would be great.
Thanks,
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
or will it figure
things out eventually?
Thanks,
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
the
rowcounter MR job. This page may be of help:
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#package_description
Good luck,
St.Ack
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
controller
2) Setup your raid controller with N devices using a 1 to 1 mapping
with physical disks
/dev/sda - disk1
/dev/sdb - disk2
...
Good luck.
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
A work college is having a really hard time posting his message to either the user or the dev
mailing list about his hbase replication problem. It says his spam score is too high. Why is
that? And is there any tips on what not to write...
Thanks,
~Jeff
--
Jeff Whiting
Qualtrics Senior
, storefileIndexSizeMB=0
Why would the region names be different between the master and the region
server?
~Jeff
On 11/12/2010 8:59 AM, Jeff Whiting wrote:
I'm unit testing some of our internal framework that interacts with hbase (version cdh3b3 =
0.89.20100924+28). The unit test is small
INFO org.apache.hadoop.hbase.master.BaseScanner: All 1
.META. region(s) scanned
2010-11-11 16:46:06,623 INFO org.apache.hadoop.hbase.master.ServerManager: 4 region servers, 0 dead,
average load 5.5
~Jeff
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
queuedRequest is returned to the caller of take(). Later, this
line in take():
removeFromRegionsInQueue(cr.getHRegion());
would remove the newly added, higher priority request from regionsInQueue.
Please comment.
On Fri, Sep 24, 2010 at 11:32 AM, jeff whiting je...@qualtrics.com wrote:
I'm sure
two cents.
On Tue, Sep 28, 2010 at 11:45 AM, jeff whiting je...@qualtrics.com wrote:
That is an astute observation. Stepping through the code with the threads
stopping execution at the points in code you suggest would indeed make it so
take() would return the lower priority compactionRequest
period of unavailability, you get
a 30 second period of slower performance, which is usually preferable.
Where is this brainstorming taking place? Could we open a JIRA issue to
capture the brainstorming in public and searchable fashion?
--
Jeff Whiting
Qualtrics Senior Software Engineer
je
/ 1282848897469
requests=0, regions=0, usedHeap=30, maxHeap=996
Total:
servers: 24 requests=0, regions=5
Load is requests per second and count of regions loaded
--
Jeff Whiting
Qualtrics Senior Software Engineer
je...@qualtrics.com
72 matches
Mail list logo
