RE: Exception with Large Graphs
Hi Avery, Thanks for your response. The data I am loading is almost 9 GB, and I have 10 nodes, each has a 4G of ram. Best, Yasser From: Avery Ching [mailto:ach...@apache.org] Sent: Friday, August 30, 2013 4:43 PM To: user@giraph.apache.org Subject: Re: Exception with Large Graphs That error is from the master dying (likely due to the results of another worker dying). Can you do a rough calculation of the size of data that you expect to be loaded and check if the memory is enough? On 8/30/13 11:19 AM, Yasser Altowim wrote: Guys, Can someone please help me with this issue? Thanks. Best, Yasser From: Yasser Altowim Sent: Thursday, August 29, 2013 11:16 AM To: user@giraph.apache.orgmailto:user@giraph.apache.org Subject: Exception with Large Graphs Hi, I am implementing an algorithm using Giraph, and I was able to run my algorithm on relatively small datasets (64,000,000 vertices and 128,000,000 edges). However, when I increase the size of the dataset to 128,000,000 vertices and 256,000,000 edges, the job takes so much time to load the vertices, and then it gives me the following exception. I have tried to increase the heap size and the task timeout value in the mapred-site.xml configuration file, and even vary the number of workers from 1 to 10, but still getting the same exceptions. I have a cluster of 10 nodes, and each node has a 4G of ram. Thanks in advance. 2013-08-29 10:22:53,150 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@1a129460mailto:java.util.concurrent.FutureTask@1a129460 2013-08-29 10:22:53,151 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@30d320e4mailto:org.apache.giraph.utils.ProgressableUtils$FutureWaitable@30d320e4 2013-08-29 10:23:07,938 INFO org.apache.giraph.worker.VertexInputSplitsCallable: readVertexInputSplit: Loaded 7769685 vertices at 14250.953615591572 vertices/sec 15539370 edges at 28500.77593053654 edges/sec Memory (free/total/max) = 680.21M / 3207.44M / 3555.56M 2013-08-29 10:23:14,538 INFO org.apache.giraph.worker.VertexInputSplitsCallable: readVertexInputSplit: Loaded 8019685 vertices at 14533.557468366102 vertices/sec 16039370 edges at 29065.97491865343 edges/sec Memory (free/total/max) = 906.80M / 3242.75M / 3555.56M 2013-08-29 10:23:21,888 INFO org.apache.giraph.worker.InputSplitsCallable: loadFromInputSplit: Finished loading /_hadoopBsp/job_201308290837_0003/_vertexInputSplitDir/9 (v=1212852, e=2425704) 2013-08-29 10:23:37,911 INFO org.apache.giraph.worker.InputSplitsHandler: reserveInputSplit: Reserved input split path /_hadoopBsp/job_201308290837_0003/_vertexInputSplitDir/19, overall roughly 7.518797% input splits reserved 2013-08-29 10:23:37,923 INFO org.apache.giraph.worker.InputSplitsCallable: getInputSplit: Reserved /_hadoopBsp/job_201308290837_0003/_vertexInputSplitDir/19 from ZooKeeper and got input split 'org.apache.giraph.io.formats.multi.InputSplitWithInputFormatIndex@24004559' 2013-08-29 10:23:44,313 INFO org.apache.giraph.worker.VertexInputSplitsCallable: readVertexInputSplit: Loaded 8482537 vertices at 14585.340134636266 vertices/sec 16965074 edges at 29169.59449002283 edges/sec Memory (free/total/max) = 538.93M / 3186.13M / 3555.56M 2013-08-29 10:23:49,963 INFO org.apache.giraph.worker.VertexInputSplitsCallable: readVertexInputSplit: Loaded 8732537 vertices at 14870.726503632277 vertices/sec 17465074 edges at 29740.356341344923 edges/sec Memory (free/total/max) = 489.84M / 3222.56M / 3555.56M 2013-08-29 10:34:28,371 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@1a129460mailto:java.util.concurrent.FutureTask@1a129460 2013-08-29 10:34:34,847 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@30d320e4mailto:org.apache.giraph.utils.ProgressableUtils$FutureWaitable@30d320e4 2013-08-29 10:34:34,850 INFO org.apache.giraph.comm.netty.handler.RequestDecoder: decode: Server window metrics MBytes/sec sent = 0, MBytes/sec received = 0.0161, MBytesSent = 0.0002, MBytesReceived = 12.3175, ave sent req MBytes = 0, ave received req MBytes = 0.0587, secs waited = 765.881 2013-08-29 10:34:35,698 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 649805ms for sessionid 0x140cb1140540006, closing socket connection and attempting reconnect 2013-08-29 10:34:42,471 WARN org.apache.giraph.bsp.BspService: process: Disconnected from ZooKeeper (will automatically try to recover) WatchedEvent state:Disconnected type:None path:null 2013-08-29 10:34:42,472 WARN org.apache.giraph.worker.InputSplitsHandler: process: Problem with zookeeper, got event with path null, state Disconnected, event type None 2013-08-29 10:34:43,819 INFO
RE: Exception with Large Graphs
Guys, Can someone please help me with this issue? Thanks. Best, Yasser From: Yasser Altowim Sent: Thursday, August 29, 2013 11:16 AM To: user@giraph.apache.org Subject: Exception with Large Graphs Hi, I am implementing an algorithm using Giraph, and I was able to run my algorithm on relatively small datasets (64,000,000 vertices and 128,000,000 edges). However, when I increase the size of the dataset to 128,000,000 vertices and 256,000,000 edges, the job takes so much time to load the vertices, and then it gives me the following exception. I have tried to increase the heap size and the task timeout value in the mapred-site.xml configuration file, and even vary the number of workers from 1 to 10, but still getting the same exceptions. I have a cluster of 10 nodes, and each node has a 4G of ram. Thanks in advance. 2013-08-29 10:22:53,150 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@1a129460mailto:java.util.concurrent.FutureTask@1a129460 2013-08-29 10:22:53,151 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@30d320e4mailto:org.apache.giraph.utils.ProgressableUtils$FutureWaitable@30d320e4 2013-08-29 10:23:07,938 INFO org.apache.giraph.worker.VertexInputSplitsCallable: readVertexInputSplit: Loaded 7769685 vertices at 14250.953615591572 vertices/sec 15539370 edges at 28500.77593053654 edges/sec Memory (free/total/max) = 680.21M / 3207.44M / 3555.56M 2013-08-29 10:23:14,538 INFO org.apache.giraph.worker.VertexInputSplitsCallable: readVertexInputSplit: Loaded 8019685 vertices at 14533.557468366102 vertices/sec 16039370 edges at 29065.97491865343 edges/sec Memory (free/total/max) = 906.80M / 3242.75M / 3555.56M 2013-08-29 10:23:21,888 INFO org.apache.giraph.worker.InputSplitsCallable: loadFromInputSplit: Finished loading /_hadoopBsp/job_201308290837_0003/_vertexInputSplitDir/9 (v=1212852, e=2425704) 2013-08-29 10:23:37,911 INFO org.apache.giraph.worker.InputSplitsHandler: reserveInputSplit: Reserved input split path /_hadoopBsp/job_201308290837_0003/_vertexInputSplitDir/19, overall roughly 7.518797% input splits reserved 2013-08-29 10:23:37,923 INFO org.apache.giraph.worker.InputSplitsCallable: getInputSplit: Reserved /_hadoopBsp/job_201308290837_0003/_vertexInputSplitDir/19 from ZooKeeper and got input split 'org.apache.giraph.io.formats.multi.InputSplitWithInputFormatIndex@24004559' 2013-08-29 10:23:44,313 INFO org.apache.giraph.worker.VertexInputSplitsCallable: readVertexInputSplit: Loaded 8482537 vertices at 14585.340134636266 vertices/sec 16965074 edges at 29169.59449002283 edges/sec Memory (free/total/max) = 538.93M / 3186.13M / 3555.56M 2013-08-29 10:23:49,963 INFO org.apache.giraph.worker.VertexInputSplitsCallable: readVertexInputSplit: Loaded 8732537 vertices at 14870.726503632277 vertices/sec 17465074 edges at 29740.356341344923 edges/sec Memory (free/total/max) = 489.84M / 3222.56M / 3555.56M 2013-08-29 10:34:28,371 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@1a129460mailto:java.util.concurrent.FutureTask@1a129460 2013-08-29 10:34:34,847 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@30d320e4mailto:org.apache.giraph.utils.ProgressableUtils$FutureWaitable@30d320e4 2013-08-29 10:34:34,850 INFO org.apache.giraph.comm.netty.handler.RequestDecoder: decode: Server window metrics MBytes/sec sent = 0, MBytes/sec received = 0.0161, MBytesSent = 0.0002, MBytesReceived = 12.3175, ave sent req MBytes = 0, ave received req MBytes = 0.0587, secs waited = 765.881 2013-08-29 10:34:35,698 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 649805ms for sessionid 0x140cb1140540006, closing socket connection and attempting reconnect 2013-08-29 10:34:42,471 WARN org.apache.giraph.bsp.BspService: process: Disconnected from ZooKeeper (will automatically try to recover) WatchedEvent state:Disconnected type:None path:null 2013-08-29 10:34:42,472 WARN org.apache.giraph.worker.InputSplitsHandler: process: Problem with zookeeper, got event with path null, state Disconnected, event type None 2013-08-29 10:34:43,819 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server slave5.ericsson-magic.net/10.126.72.165:22181 2013-08-29 10:34:44,077 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to slave5.ericsson-magic.net/10.126.72.165:22181, initiating session 2013-08-29 10:34:44,220 WARN org.apache.giraph.bsp.BspService: process: Got unknown null path event WatchedEvent state:Expired type:None path:null 2013-08-29 10:34:44,220 WARN org.apache.giraph.worker.InputSplitsHandler: process: Problem with zookeeper, got event with path null, state Expired, event type None 2013-08-29
Re: Exception with Large Graphs
That error is from the master dying (likely due to the results of another worker dying). Can you do a rough calculation of the size of data that you expect to be loaded and check if the memory is enough? On 8/30/13 11:19 AM, Yasser Altowim wrote: Guys, Can someone please help me with this issue? Thanks. Best, Yasser *From:*Yasser Altowim *Sent:* Thursday, August 29, 2013 11:16 AM *To:* user@giraph.apache.org *Subject:* Exception with Large Graphs Hi, I am implementing an algorithm using Giraph, and I was able to run my algorithm on relatively small datasets (64,000,000 vertices and 128,000,000 edges). However, when I increase the size of the dataset to 128,000,000 vertices and 256,000,000 edges, the job takes so much time to load the vertices, and then it gives me the following exception. I have tried to increase the heap size and the task timeout value in the mapred-site.xml configuration file, and even vary the number of workers from 1 to 10, but still getting the same exceptions. I have a cluster of 10 nodes, and each node has a 4G of ram. Thanks in advance. 2013-08-29 10:22:53,150 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@1a129460 mailto:java.util.concurrent.FutureTask@1a129460 2013-08-29 10:22:53,151 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@30d320e4 mailto:org.apache.giraph.utils.ProgressableUtils$FutureWaitable@30d320e4 2013-08-29 10:23:07,938 INFO org.apache.giraph.worker.VertexInputSplitsCallable: readVertexInputSplit: Loaded 7769685 vertices at 14250.953615591572 vertices/sec 15539370 edges at 28500.77593053654 edges/sec Memory (free/total/max) = 680.21M / 3207.44M / 3555.56M 2013-08-29 10:23:14,538 INFO org.apache.giraph.worker.VertexInputSplitsCallable: readVertexInputSplit: Loaded 8019685 vertices at 14533.557468366102 vertices/sec 16039370 edges at 29065.97491865343 edges/sec Memory (free/total/max) = 906.80M / 3242.75M / 3555.56M 2013-08-29 10:23:21,888 INFO org.apache.giraph.worker.InputSplitsCallable: loadFromInputSplit: Finished loading /_hadoopBsp/job_201308290837_0003/_vertexInputSplitDir/9 (v=1212852, e=2425704) 2013-08-29 10:23:37,911 INFO org.apache.giraph.worker.InputSplitsHandler: reserveInputSplit: Reserved input split path /_hadoopBsp/job_201308290837_0003/_vertexInputSplitDir/19, overall roughly 7.518797% input splits reserved 2013-08-29 10:23:37,923 INFO org.apache.giraph.worker.InputSplitsCallable: getInputSplit: Reserved /_hadoopBsp/job_201308290837_0003/_vertexInputSplitDir/19 from ZooKeeper and got input split 'org.apache.giraph.io.formats.multi.InputSplitWithInputFormatIndex@24004559' 2013-08-29 10:23:44,313 INFO org.apache.giraph.worker.VertexInputSplitsCallable: readVertexInputSplit: Loaded 8482537 vertices at 14585.340134636266 vertices/sec 16965074 edges at 29169.59449002283 edges/sec Memory (free/total/max) = 538.93M / 3186.13M / 3555.56M 2013-08-29 10:23:49,963 INFO org.apache.giraph.worker.VertexInputSplitsCallable: readVertexInputSplit: Loaded 8732537 vertices at 14870.726503632277 vertices/sec 17465074 edges at 29740.356341344923 edges/sec Memory (free/total/max) = 489.84M / 3222.56M / 3555.56M 2013-08-29 10:34:28,371 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@1a129460 mailto:java.util.concurrent.FutureTask@1a129460 2013-08-29 10:34:34,847 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@30d320e4 mailto:org.apache.giraph.utils.ProgressableUtils$FutureWaitable@30d320e4 2013-08-29 10:34:34,850 INFO org.apache.giraph.comm.netty.handler.RequestDecoder: decode: Server window metrics MBytes/sec sent = 0, MBytes/sec received = 0.0161, MBytesSent = 0.0002, MBytesReceived = 12.3175, ave sent req MBytes = 0, ave received req MBytes = 0.0587, secs waited = 765.881 2013-08-29 10:34:35,698 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 649805ms for sessionid 0x140cb1140540006, closing socket connection and attempting reconnect 2013-08-29 10:34:42,471 WARN org.apache.giraph.bsp.BspService: process: Disconnected from ZooKeeper (will automatically try to recover) WatchedEvent state:Disconnected type:None path:null 2013-08-29 10:34:42,472 WARN org.apache.giraph.worker.InputSplitsHandler: process: Problem with zookeeper, got event with path null, state Disconnected, event type None 2013-08-29 10:34:43,819 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server slave5.ericsson-magic.net/10.126.72.165:22181 2013-08-29 10:34:44,077 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to slave5.ericsson-magic.net/10.126.72.165:22181, initiating session 2013-08-29
