Hi Claudio,
I think I have fixed the problem.
HBase runs with its own copy of ZooKeeper which listens on port 2181. So,
when I tried to start ZooKeeper for Giraph it also tried to listen on port 2181
and found it was already in use, and then it terminated - which is why Giraph
That should in principle not be the case, as the zookeeper started by
Giraph listens on a different port than the default. See
parameter giraph.zkServerPort, which defaults to 22181.
On Wed, Sep 4, 2013 at 11:40 AM, Ken Williams zoo9...@hotmail.com wrote:
Hi Claudio,
I think I have fixed
Thanks,
I was not sure if it really works as I described.
Facebook can't be using it like this if, as described, they have
billions of vertices and a trillion edges.
Yes, its strange. I guess configuration does not help so much on large
cluster. What might help are properties of input
H. Interesting.
Is Giraph (1.0.0) supposed to come with its own version of ZooKeeper ?
The only version of ZooKeeper I have installed is the one that came with
HBase,and the config file it uses /etc/zookeeper/conf/zoo.cfg specifies
clientPort=2181This is the only zoo.cfg file on my
Giraph is shipped with Zookeeper 3.3.3, and it is run, if an existing
zookeeper is not used through the giraph.zkServerList parameter, with its
own configuration listening on port 22181.
On Wed, Sep 4, 2013 at 7:11 PM, Ken Williams zoo9...@hotmail.com wrote:
H. Interesting.
Is Giraph
Ok thanks Avery. But I still have two questions, the first a really dumb
newbie question. Why are there ever any number of partitions other than
exactly one per worker thread (one per worker in our case)? And a deeper
question. Even if I shrink the cache I would suppose that if Facebook has
We have caches per every compute threads. Then we have w worker caches
per compute thread. So the total amount of memory consumed by message
caches per worker =
Compute threads * workers * size of cache. The best thing is to tune
down the size of the cache from MAX_MSG_REQUEST_SIZE to a size
