Hi
Thanks for the reply. I am using the HAMA version from the TRUNK and I am
running my own developed algorithm. I am trying to work with a grapg consists
of 10 million vertices. Did someone experienced working with big graphs
(millions of vertices) using HAMA? can you please share your experience?
I am trying now to use:
Conf.setClass("
hama.graph.vertices.info",org.apache.hama.graph.
OffHeapVerticesInfo.class,org.apache.hama.graph.VerticesInfo.class);
I get the error:
14/01/20 17:42:16 ERROR bsp.LocalBSPRunner: Exception during BSP execution!
java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError:
org/apache/directmemory/utils/CacheValuesIterable
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.hama.bsp.LocalBSPRunner$ThreadObserver.run(LocalBSPRunner.java:313)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.NoClassDefFoundError:
org/apache/directmemory/utils/CacheValuesIterable
at
org.apache.hama.graph.OffHeapVerticesInfo.skippingIterator(OffHeapVerticesInfo.java:112)
at org.apache.hama.graph.GraphJobRunner.cleanup(GraphJobRunner.java:163)
at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:262)
at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
at
org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
The log of my master is as following:
/************************************************************
STARTUP_MSG: Starting BSPMaster
STARTUP_MSG: host = c3-large1-master/10.255.255.2
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.2.0
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1479473;
compiled by 'hortonfo' on Mon May 6 18:29:07 UTC 2013
STARTUP_MSG: java = 1.7.0_25
************************************************************/
2014-01-14 21:27:35,808 INFO org.apache.hama.bsp.BSPMaster: RPC BSPMaster: host
master port 40000
2014-01-14 21:27:37,200 INFO org.apache.hama.ipc.Server: Starting Socket Reader
#1 for port 40000
2014-01-14 21:27:37,732 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2014-01-14 21:27:38,147 INFO org.apache.hama.http.HttpServer: Port returned by
webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the
listener on 40013
2014-01-14 21:27:38,168 INFO org.apache.hama.http.HttpServer:
listener.getLocalPort() returned 40013
webServer.getConnectors()[0].getLocalPort() returned 40013
2014-01-14 21:27:38,168 INFO org.apache.hama.http.HttpServer: Jetty bound to
port 40013
2014-01-14 21:27:38,168 INFO org.mortbay.log: jetty-6.1.14
2014-01-14 21:27:38,446 INFO org.mortbay.log: Extract
jar:file:/usr/local/hama-0.6.3/hama-core-0.6.3.jar!/webapp/bspmaster/ to
/tmp/Jetty_master_40013_bspmaster____ge2lxf/webapp
2014-01-14 21:27:40,162 INFO org.mortbay.log: Started
SelectChannelConnector@master:40013
2014-01-14 21:27:40,734 INFO org.apache.hama.bsp.BSPMaster: Cleaning up the
system directory
2014-01-14 21:27:40,734 INFO org.apache.hama.bsp.BSPMaster:
hdfs://master:54310/tmp/hama-hadoop/bsp/system
2014-01-14 21:27:40,991 INFO org.apache.hama.bsp.sync.ZKSyncBSPMasterClient:
Initialized ZK false
2014-01-14 21:27:40,991 INFO org.apache.hama.bsp.sync.ZKSyncClient:
Initializing ZK Sync Client
2014-01-14 21:27:41,073 INFO org.apache.hama.ipc.Server: IPC Server Responder:
starting
2014-01-14 21:27:41,077 INFO org.apache.hama.ipc.Server: IPC Server listener on
40000: starting
2014-01-14 21:27:41,085 INFO org.apache.hama.ipc.Server: IPC Server handler 0
on 40000: starting
2014-01-14 21:27:41,088 INFO org.apache.hama.bsp.BSPMaster: Starting RUNNING
2014-01-14 21:27:41,168 INFO org.apache.hama.bsp.BSPMaster: groomd_slave2_50000
is added.
2014-01-14 21:27:49,634 INFO org.apache.hama.bsp.BSPMaster: groomd_slave1_50000
is added.
2014-01-14 21:28:15,943 INFO org.apache.hama.bsp.BSPMaster: groomd_master_50000
is added.
On Sunday, January 19, 2014 7:58 AM, Tommaso Teofili
<[email protected]> wrote:
yes, the correct way of setting OffHeapVI is: conf.setClass("
hama.graph.vertices.info",org.apache.hama.graph.
OffHeapVerticesInfo.class,org.apache.hama.graph.VerticesInfo.class);
Apart from that, what Hama version are you running on?
Looking at the code in trunk it shouldn't be possible to have a NPE on the
currentVertex if the iterator is consumed correctly, instead if one doesn't
call hasNext before next and / or calls next even if hasNext returns false
then it's possible to have that NPE.
Also what algorithm / example are you running? Any useful information (like
environment, execution mode, logs, version, etc.) would be useful to help
you.
Tommaso
2014/1/19 步青云 <[email protected]>
> I got the same problem about loading vertices into RAM.And I try to use
> off OffHeapVerticesInfo.
> You may use the method setClass like this:
> conf.setClass("hama.graph.vertices.info
> ",org.apache.hama.graph.OffHeapVerticesInfo.class,org.apache.hama.graph.VerticesInfo.class);
> However,I got the Nullexception using OffHeapVerticesInfo.The errors are
> as follows:
>
> 14/01/18 20:54:23 ERROR bsp.LocalBSPRunner: Exception during BSP execution!
> java.lang.NullPointerException
> at
> org.apache.hama.graph.OffHeapVerticesInfo$1.next(OffHeapVerticesInfo.java:139)
> at
> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:251)
> at org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:145)
> at
> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256)
> at
> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
>
> Anyone could help me to solve this problem?Thanks a lot.
>
>
>
>
> ------------------ Original ------------------
> From: "Ammar Sahib"<[email protected]>;
> Date: Jan 17, 2014
> To: "[email protected]"<[email protected]>;
>
> Subject: Re: loading vertices into RAM
>
>
>
> I think we are getting close now, However now I have runtime exception:
>
> Exception in thread "main" java.lang.RuntimeException: interface
> org.apache.hama.graph.VerticesInfo not
> org.apache.hama.graph.ListVerticesInfo
> at
> org.apache.hadoop.conf.Configuration.setClass(Configuration.java:858)
>
>
>
>
>
> On Friday, January 17, 2014 2:30 PM, Tommaso Teofili <
> [email protected]> wrote:
>
> ah yes, sorry, you also have to specify the interface, I don't have the
> code in front of me but it should be :
>
> conf.setClass("hama.graph.vertices.info",
> org.apache.hama.graph.VerticesInfo.class, org.apache.
> hama.graph.ListVerticesInfo.class);
>
> Tommaso
>
>
>
> 2014/1/17 Ammar Sahib <[email protected]>
>
> > Hi
> >
> > Thanks for your reply. I used now:
> >
> > conf.setClass("hama.graph.vertices.info
> > ",org.apache.hama.graph.ListVerticesInfo.class);
> >
> > Now I get this error:
> > The method setClass(String, Class<?>, Class<?>) in the type Configuration
> > is not applicable for the arguments (String, Class<ListVerticesInfo>)
> >
> > I am using HAMA 0.6.3
> >
> >
> >
> >
> >
> > On Friday, January 17, 2014 12:59 PM, Tommaso Teofili <
> > [email protected]> wrote:
> >
> > you're passing the fully qualified name of the Class as a String to a
> > method setClass(String, Class) while you should pass the Class itself,
> > e.g.:
> > HamaConfiguration conf = new HamaConfiguration();
> > conf.setClass("hama.graph.vertices.info",org.apache.
> > hama.graph.ListVerticesInfo.class);
> >
> > Hope this helps,
> > Tommaso
> >
> >
> >
> >
> > 2014/1/17 Ammar Sahib <[email protected]>
> >
> > > Hi
> > >
> > > I am trying to evaluate the different implementation below:
> > >
> > >
> > > - ListVerticesinfo: loads vertices into array list.
> > > - MapVerticesinfo: loads vertices into tree map.
> > > - DiskVerticesInfo: loads vertices into a local file.
> > >
> > > When using the conf.setClass method I got an error. Below is sample of
> my
> > > code:
> > > HamaConfiguration conf = new HamaConfiguration();
> > > conf.setClass("hama.graph.vertices.info
> > > ","org.apache.hama.graph.ListVerticesInfo");
> > >
> > > The error I am getting is:
> > > The method setClass(String, Class<?>, Class<?>) in the type
> Configuration
> > > is not applicable for the arguments (String, String).
> > >
> > > However I found that I can use conf.set method.
> > >
> > >
> > > Can someone tell me what is I am doing wrong?
> > >
> > >
> > >
> > >
> > >
> > > On Wednesday, January 15, 2014 8:01 AM, Tommaso Teofili <
> > > [email protected]> wrote:
> > >
> > > and OffHeapVerticesInfo for loading vertices off heap, which is
> available
> > > with 0.6.3 as well if I recall correctly.
> > > Tommaso
> > >
> > >
> > >
> > > 2014/1/15 Edward J. Yoon <[email protected]>
> > >
> > > > There are few implementations.
> > > >
> > > > - ListVerticesinfo: loads vertices into array list.
> > > > - MapVerticesinfo: loads vertices into tree map.
> > > > - DiskVerticesInfo: loads vertices into a local file.
> > > >
> > > > You can choose one of them by setting the "hama.graph.vertices.info"
> > > > in job configuration.
> > > >
> > > > > conf.setClass("hama.graph.vertices.info",
> > > > "org.apache.hama.graph.ListVerticesInfo".
> > > >
> > > > With the latest 0.6.3 version, you can use only ListVerticesInfo.
> > > > Please use the TRUNK.
> > > >
> > > >
> > > > On Tue, Jan 14, 2014 at 11:18 PM, Ammar Sahib <[email protected]
> >
> > > > wrote:
> > > > > Hi
> > > > >
> > > > > According to the BSP model, the data is processed in the RAM and
> that
> > > is
> > > > the reason why Pregel model is faster than the MapReduce (MapReduce
> > > > writedown to disk). Can someone explains to me how to be sure that
> all
> > > the
> > > > graph vertices are actually been loaded in RAM?
> > > > >
> > > > >
> > > > > How would HAMA behave if the vertices values are so big such that
> the
> > > > available RAM memory is not enough to contains all of the vertices?
> > > > >
> > > > > Regards
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards, Edward J. Yoon
> > > > @eddieyoon
> > > >
> > >
> >
>
