Hi Mario,
First off, look for any JVM GC pause warnings in the TabletServer logs.
There's a callback which should fire at a given interval and will warn
when it does not. Typically, this is because of a stop-the-world GC
cycle. These cycles also block the ZK heartbeat action as Dylan
described. They're the most common culprit.
Other related lower-lever configurations may be relevant:
* swappiness -- the OS swaps memory pages of the tserver to disk which
drastically increase the latency of the process. D
* I/O saturation - Dramatic pressure on the i/o subsystem can break the
system to its knees without ample memory headroom
* CPU saturation by other processes on the same physical host.
Other stuff is also covered
http://accumulo.apache.org/1.8/accumulo_user_manual.html#_keeping_the_tablet_server_lock
Also, if there are many services running on the same hosts that talk to
ZooKeeper, you might bump into ZooKeeper's rate-limiting via maxClientCnxns.
https://community.hortonworks.com/articles/51191/understanding-apache-zookeeper-connection-rate-lim.html
Regarding your first point, unless you're falling behind on minor
compactions, I wouldn't be too worried about the "running low on
memory". In sustained ingest situations, you *want* memory to be
saturated :). It's a hint as to the system's state if other things are
going wrong, but it's not directly a problem by itself.
Mario Pastorelli wrote:
Hi all,
I have two recurring errors with Accumulo in my cluster and I would like
to know more about them. The first, usually happening at ingestion time
when I write with the batch writers many records, is the "Running low on
memory". We keep adding memory to Accumulo but this is a blind guess and
I was wondering if there is a way to understand how much memory Accumulo
would need considering the amount of data that will be written. Should
we write slowly to Accumulo to avoid this? What is filling all the
memory at ingestion time?
Secondly, we have these zookeeper session expired and other zookeeper
timeouts. Zookeeper on our cluster works quite well, we have many
systems using it. How can I debug a "zookeeper session expired" in Accumulo?
Thanks,
Mario
--
Mario Pastorelli| TERALYTICS
*software engineer*
Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
phone:+41794381682
email: [email protected]
<mailto:[email protected]>
www.teralytics.net <http://www.teralytics.net/>
Company registration number: CH-020.3.037.709-7 | Trade register Canton
Zurich
Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz,
Yann de Vries
This e-mail message contains confidential information which is for the
sole attention and use of the intended recipient. Please notify us at
once if you think that it may not be intended for you and delete it
immediately.
