RE: Problem Building Apache Giraph Formats

[Rui Sarmento]

How much memory do those tests need? Is there a way/command to allocate more 
memory to Giraph?

From: nitay.jo...@gmail.com
Subject: Re: Problem Building Apache Giraph Formats
Date: Tue, 27 Nov 2012 15:53:42 -0800
To: user@giraph.apache.org

>From cursory look it seems to me like all the failures are due to memory 
>issues (OOME)
On Nov 27, 2012, at 1:54 PM, Rui Sarmento  
wrote:Sorry not 800, its more than 80 :-) .  I'm on the right dir.

> Date: Tue, 27 Nov 2012 22:16:08 +0100
> Subject: Re: Problem Building Apache Giraph Formats
> From: efeshundert...@googlemail.com
> To: user@giraph.apache.org
> 
> Hi Rui!
> 
> from where are you running this? The test-results are on
> target/munged/surefire-reports. Please go into that directory and grep
> for ERROR.
> 
> --André
> 
> 2012/11/27 Rui Sarmento :
> > I'm using "grep -R -l" command but is taking to long, any advise for another
> > command? Is there a faster one available, its more than 800 documents.
> >
> > Thanks,
> >
> > Rui
> >
> > 
> > From: alessan...@fb.com
> > To: user@giraph.apache.org
> > Subject: Re: Problem Building Apache Giraph Formats
> > Date: Tue, 27 Nov 2012 02:05:18 +
> >
> >
> > You could take a look at surefire-reports and post the failing test(s).
> > Search for the string "error" in that folder.
> >
> > From: Rui Sarmento 
> > Reply-To: "user@giraph.apache.org" 
> > Date: Monday, November 26, 2012 5:58 PM
> > To: Giraph Support 
> > Subject: RE: Problem Building Apache Giraph Formats
> >
> > running "mvn install" on trunk dir gives this result:
> >
> > [INFO]
> > 
> > [INFO] Reactor Summary:
> > [INFO]
> > [INFO] Apache Giraph Parent .. SUCCESS [0.563s]
> > [INFO] Apache Giraph . FAILURE
> > [2:23.047s]
> > [INFO] Apache Giraph Formats . SKIPPED
> > [INFO]
> > 
> > [INFO] BUILD FAILURE
> > [INFO]
> > 
> > [INFO] Total time: 2:23.882s
> > [INFO] Finished at: Tue Nov 27 01:54:17 WET 2012
> > [INFO] Final Memory: 27M/811M
> > [INFO]
> > 
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-surefire-plugin:2.6:test (default-test) on
> > project giraph: There are test failures.
> > [ERROR]
> > [ERROR] Please refer to
> > /home/110414015/trunk/giraph/target/munged/surefire-reports for the
> > individual test results.
> > [ERROR] -> [Help 1]
> > [ERROR]
> > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> > switch.
> > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > [ERROR]
> > [ERROR] For more information about the errors and possible solutions, please
> > read the following articles:
> > [ERROR] [Help 1]
> > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> > [ERROR]
> > [ERROR] After correcting the problems, you can resume the build with the
> > command
> > [ERROR] mvn  -rf :giraph
> >
> > Do you need any test result in particular?
> >
> > Regards
> >
> > 
> > From: alessan...@fb.com
> > To: user@giraph.apache.org
> > Subject: Re: Problem Building Apache Giraph Formats
> > Date: Tue, 27 Nov 2012 01:37:39 +
> >
> > The actual error seems to be:
> >
> > [ERROR] Failed to execute goal on project giraph-formats-contrib: Could not
> > resolve dependencies for project
> > org.apache.giraph:giraph-formats-contrib:jar:0.2-SNAPSHOT: Failure to find
> > org.apache.giraph:giraph:jar:tests:0.2-SNAPSHOT
> > inhttp://repo1.maven.org/maven2 was cached in the local repository,
> > resolution will not be reattempted until the update interval of central has
> > elapsed or updates are forced -> [Help 1]
> >
> > Try running "mvn install" at the top level first. That should put the test
> > jar in your local repository.
> >
> > From: Rui Sarmento 
> > Reply-To: "user@giraph.apache.org" 
> > Date: Monday, November 26, 2012 5:15 PM
> > To: Giraph Support 
> > Subject: RE: Problem Building Apache Giraph Formats
> >
> > Hi Alessandro,
> >
> > The command is "mvn compile", Everything compiles with success except Apache
> > Giraph Formats
> >
> > [INFO]
> > 
> > [INFO] Building Apache Giraph Formats 0.2-SNAPSHOT
> > [INFO]
> > 
> > [WARNING] The POM for org.apache.hadoop:hadoop-core:jar:0.20.1 is missing,
> > no dependency information available
> > [WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to
> > local.repository (file:../../local.repository/trunk): No connector available
> > to access repository local.repository (file:../../local.repository/trunk) of
> > 

Re: Problem Building Apache Giraph Formats

[Nitay Joffe]

From cursory look it seems to me like all the failures are due to memory issues 
(OOME)

On Nov 27, 2012, at 1:54 PM, Rui Sarmento  wrote:

> Sorry not 800, its more than 80 :-) .  I'm on the right dir.
> 
> 
> > Date: Tue, 27 Nov 2012 22:16:08 +0100
> > Subject: Re: Problem Building Apache Giraph Formats
> > From: efeshundert...@googlemail.com
> > To: user@giraph.apache.org
> > 
> > Hi Rui!
> > 
> > from where are you running this? The test-results are on
> > target/munged/surefire-reports. Please go into that directory and grep
> > for ERROR.
> > 
> > --André
> > 
> > 2012/11/27 Rui Sarmento :
> > > I'm using "grep -R -l" command but is taking to long, any advise for 
> > > another
> > > command? Is there a faster one available, its more than 800 documents.
> > >
> > > Thanks,
> > >
> > > Rui
> > >
> > > 
> > > From: alessan...@fb.com
> > > To: user@giraph.apache.org
> > > Subject: Re: Problem Building Apache Giraph Formats
> > > Date: Tue, 27 Nov 2012 02:05:18 +
> > >
> > >
> > > You could take a look at surefire-reports and post the failing test(s).
> > > Search for the string "error" in that folder.
> > >
> > > From: Rui Sarmento 
> > > Reply-To: "user@giraph.apache.org" 
> > > Date: Monday, November 26, 2012 5:58 PM
> > > To: Giraph Support 
> > > Subject: RE: Problem Building Apache Giraph Formats
> > >
> > > running "mvn install" on trunk dir gives this result:
> > >
> > > [INFO]
> > > 
> > > [INFO] Reactor Summary:
> > > [INFO]
> > > [INFO] Apache Giraph Parent .. SUCCESS 
> > > [0.563s]
> > > [INFO] Apache Giraph . FAILURE
> > > [2:23.047s]
> > > [INFO] Apache Giraph Formats . SKIPPED
> > > [INFO]
> > > 
> > > [INFO] BUILD FAILURE
> > > [INFO]
> > > 
> > > [INFO] Total time: 2:23.882s
> > > [INFO] Finished at: Tue Nov 27 01:54:17 WET 2012
> > > [INFO] Final Memory: 27M/811M
> > > [INFO]
> > > 
> > > [ERROR] Failed to execute goal
> > > org.apache.maven.plugins:maven-surefire-plugin:2.6:test (default-test) on
> > > project giraph: There are test failures.
> > > [ERROR]
> > > [ERROR] Please refer to
> > > /home/110414015/trunk/giraph/target/munged/surefire-reports for the
> > > individual test results.
> > > [ERROR] -> [Help 1]
> > > [ERROR]
> > > [ERROR] To see the full stack trace of the errors, re-run Maven with the 
> > > -e
> > > switch.
> > > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > > [ERROR]
> > > [ERROR] For more information about the errors and possible solutions, 
> > > please
> > > read the following articles:
> > > [ERROR] [Help 1]
> > > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> > > [ERROR]
> > > [ERROR] After correcting the problems, you can resume the build with the
> > > command
> > > [ERROR] mvn  -rf :giraph
> > >
> > > Do you need any test result in particular?
> > >
> > > Regards
> > >
> > > 
> > > From: alessan...@fb.com
> > > To: user@giraph.apache.org
> > > Subject: Re: Problem Building Apache Giraph Formats
> > > Date: Tue, 27 Nov 2012 01:37:39 +
> > >
> > > The actual error seems to be:
> > >
> > > [ERROR] Failed to execute goal on project giraph-formats-contrib: Could 
> > > not
> > > resolve dependencies for project
> > > org.apache.giraph:giraph-formats-contrib:jar:0.2-SNAPSHOT: Failure to find
> > > org.apache.giraph:giraph:jar:tests:0.2-SNAPSHOT
> > > inhttp://repo1.maven.org/maven2 was cached in the local repository,
> > > resolution will not be reattempted until the update interval of central 
> > > has
> > > elapsed or updates are forced -> [Help 1]
> > >
> > > Try running "mvn install" at the top level first. That should put the test
> > > jar in your local repository.
> > >
> > > From: Rui Sarmento 
> > > Reply-To: "user@giraph.apache.org" 
> > > Date: Monday, November 26, 2012 5:15 PM
> > > To: Giraph Support 
> > > Subject: RE: Problem Building Apache Giraph Formats
> > >
> > > Hi Alessandro,
> > >
> > > The command is "mvn compile", Everything compiles with success except 
> > > Apache
> > > Giraph Formats
> > >
> > > [INFO]
> > > 
> > > [INFO] Building Apache Giraph Formats 0.2-SNAPSHOT
> > > [INFO]
> > > 
> > > [WARNING] The POM for org.apache.hadoop:hadoop-core:jar:0.20.1 is missing,
> > > no dependency information available
> > > [WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to
> > > local.repository (file:../../local.repository/trunk): No connector 
> > > available
> > > to access reposi

Problem Building Apache Giraph Formats

[Rui Sarmento]

Here goes the files (with command grep "ERROR" *.txt -l)
--Rui


  ---
Test set: org.apache.giraph.comm.RequestTest
---
Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 0.194 sec <<< 
FAILURE!
sendVertexPartition(org.apache.giraph.comm.RequestTest)  Time elapsed: 0.053 
sec  <<< ERROR!
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at 
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at 
org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
at 
org.jboss.netty.channel.socket.nio.AbstractNioWorker.start(AbstractNioWorker.java:179)
at 
org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:325)
at 
org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:35)
at 
org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:308)
at 
org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:35)
at 
org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at 
org.jboss.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:635)
at 
org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:504)
at 
org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:47)
at 
org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:657)
at 
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)
at 
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:170)
at 
org.jboss.netty.channel.socket.nio.NioClientSocketChannel.(NioClientSocketChannel.java:79)
at 
org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.newChannel(NioClientSocketChannelFactory.java:176)
at 
org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.newChannel(NioClientSocketChannelFactory.java:82)
at 
org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:213)
at 
org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:183)
at 
org.apache.giraph.comm.netty.NettyClient.connectAllAddresses(NettyClient.java:408)
at org.apache.giraph.comm.RequestTest.setUp(RequestTest.java:102)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at 
org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
at 
org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:120)
at 
org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:103)
at org.apache.maven.surefire.Surefire.run(Surefire.java:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java

RE: Problem Building Apache Giraph Formats

[Rui Sarmento]

Sorry not 800, its more than 80 :-) .  I'm on the right dir.

> Date: Tue, 27 Nov 2012 22:16:08 +0100
> Subject: Re: Problem Building Apache Giraph Formats
> From: efeshundert...@googlemail.com
> To: user@giraph.apache.org
> 
> Hi Rui!
> 
> from where are you running this? The test-results are on
> target/munged/surefire-reports. Please go into that directory and grep
> for ERROR.
> 
> --André
> 
> 2012/11/27 Rui Sarmento :
> > I'm using "grep -R -l" command but is taking to long, any advise for another
> > command? Is there a faster one available, its more than 800 documents.
> >
> > Thanks,
> >
> > Rui
> >
> > 
> > From: alessan...@fb.com
> > To: user@giraph.apache.org
> > Subject: Re: Problem Building Apache Giraph Formats
> > Date: Tue, 27 Nov 2012 02:05:18 +
> >
> >
> > You could take a look at surefire-reports and post the failing test(s).
> > Search for the string "error" in that folder.
> >
> > From: Rui Sarmento 
> > Reply-To: "user@giraph.apache.org" 
> > Date: Monday, November 26, 2012 5:58 PM
> > To: Giraph Support 
> > Subject: RE: Problem Building Apache Giraph Formats
> >
> > running "mvn install" on trunk dir gives this result:
> >
> > [INFO]
> > 
> > [INFO] Reactor Summary:
> > [INFO]
> > [INFO] Apache Giraph Parent .. SUCCESS [0.563s]
> > [INFO] Apache Giraph . FAILURE
> > [2:23.047s]
> > [INFO] Apache Giraph Formats . SKIPPED
> > [INFO]
> > 
> > [INFO] BUILD FAILURE
> > [INFO]
> > 
> > [INFO] Total time: 2:23.882s
> > [INFO] Finished at: Tue Nov 27 01:54:17 WET 2012
> > [INFO] Final Memory: 27M/811M
> > [INFO]
> > 
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-surefire-plugin:2.6:test (default-test) on
> > project giraph: There are test failures.
> > [ERROR]
> > [ERROR] Please refer to
> > /home/110414015/trunk/giraph/target/munged/surefire-reports for the
> > individual test results.
> > [ERROR] -> [Help 1]
> > [ERROR]
> > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> > switch.
> > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > [ERROR]
> > [ERROR] For more information about the errors and possible solutions, please
> > read the following articles:
> > [ERROR] [Help 1]
> > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> > [ERROR]
> > [ERROR] After correcting the problems, you can resume the build with the
> > command
> > [ERROR]   mvn  -rf :giraph
> >
> > Do you need any test result in particular?
> >
> > Regards
> >
> > 
> > From: alessan...@fb.com
> > To: user@giraph.apache.org
> > Subject: Re: Problem Building Apache Giraph Formats
> > Date: Tue, 27 Nov 2012 01:37:39 +
> >
> > The actual error seems to be:
> >
> > [ERROR] Failed to execute goal on project giraph-formats-contrib: Could not
> > resolve dependencies for project
> > org.apache.giraph:giraph-formats-contrib:jar:0.2-SNAPSHOT: Failure to find
> > org.apache.giraph:giraph:jar:tests:0.2-SNAPSHOT
> > inhttp://repo1.maven.org/maven2 was cached in the local repository,
> > resolution will not be reattempted until the update interval of central has
> > elapsed or updates are forced -> [Help 1]
> >
> > Try running "mvn install" at the top level first. That should put the test
> > jar in your local repository.
> >
> > From: Rui Sarmento 
> > Reply-To: "user@giraph.apache.org" 
> > Date: Monday, November 26, 2012 5:15 PM
> > To: Giraph Support 
> > Subject: RE: Problem Building Apache Giraph Formats
> >
> > Hi Alessandro,
> >
> > The command is "mvn compile", Everything compiles with success except Apache
> > Giraph Formats
> >
> > [INFO]
> > 
> > [INFO] Building Apache Giraph Formats 0.2-SNAPSHOT
> > [INFO]
> > 
> > [WARNING] The POM for org.apache.hadoop:hadoop-core:jar:0.20.1 is missing,
> > no dependency information available
> > [WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to
> > local.repository (file:../../local.repository/trunk): No connector available
> > to access repository local.repository (file:../../local.repository/trunk) of
> > type legacy using the available factories WagonRepositoryConnectorFactory
> > [WARNING] Failure to transfer asm:asm/maven-metadata.xml from
> > file:../../local.repository/trunk was cached in the local repository,
> > resolution will not be reattempted until the update interval of
> > local.repository has elapsed or updates are forced. Original error: Could
> > not transfer meta

Re: Minimum superstep time

[Jonathan Bishop]

Sebastian,

I am evaluating a directed acyclic graph (DAG). So the vertices in one
superstep are not dependent on each other, only upon their predecessors.

What do you mean "execute the messaging simultaneously"? I am simply using
BasicVertex.sendMsg() during BasicVertex.compute(). Is there another way to
do this?

Jon




On Tue, Nov 27, 2012 at 10:19 AM, Sebastian Schelter  wrote:

> Are the vertices which are active in supersteps dependent on each other?
> If this is not the case, you could try to execute the messaging
> simultaneously.
>
> Could you give a little more details about the problem, which you are
> trying to solve?
>
> /s
>
>
> On 27.11.2012 19:07, Jonathan Bishop wrote:
> > Hi,
> >
> > I am involved in a project which requires thousands of supersteps. In
> each
> > superstep there are about 1 thousand vertices, each sending a message.
> The
> > message size is not large, about 40 bytes or so.
> >
> > I am seeing abouth 3000-5000ms per superstep. I am curious what others
> with
> > similar problem sizes are seeing.
> >
> > Thanks,
> >
> > Jon
> >
>
>

Re: Problem Building Apache Giraph Formats

[André Kelpe]

Hi Rui!

from where are you running this? The test-results are on
target/munged/surefire-reports. Please go into that directory and grep
for ERROR.

--André

2012/11/27 Rui Sarmento :
> I'm using "grep -R -l" command but is taking to long, any advise for another
> command? Is there a faster one available, its more than 800 documents.
>
> Thanks,
>
> Rui
>
> 
> From: alessan...@fb.com
> To: user@giraph.apache.org
> Subject: Re: Problem Building Apache Giraph Formats
> Date: Tue, 27 Nov 2012 02:05:18 +
>
>
> You could take a look at surefire-reports and post the failing test(s).
> Search for the string "error" in that folder.
>
> From: Rui Sarmento 
> Reply-To: "user@giraph.apache.org" 
> Date: Monday, November 26, 2012 5:58 PM
> To: Giraph Support 
> Subject: RE: Problem Building Apache Giraph Formats
>
> running "mvn install" on trunk dir gives this result:
>
> [INFO]
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Giraph Parent .. SUCCESS [0.563s]
> [INFO] Apache Giraph . FAILURE
> [2:23.047s]
> [INFO] Apache Giraph Formats . SKIPPED
> [INFO]
> 
> [INFO] BUILD FAILURE
> [INFO]
> 
> [INFO] Total time: 2:23.882s
> [INFO] Finished at: Tue Nov 27 01:54:17 WET 2012
> [INFO] Final Memory: 27M/811M
> [INFO]
> 
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.6:test (default-test) on
> project giraph: There are test failures.
> [ERROR]
> [ERROR] Please refer to
> /home/110414015/trunk/giraph/target/munged/surefire-reports for the
> individual test results.
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR]   mvn  -rf :giraph
>
> Do you need any test result in particular?
>
> Regards
>
> 
> From: alessan...@fb.com
> To: user@giraph.apache.org
> Subject: Re: Problem Building Apache Giraph Formats
> Date: Tue, 27 Nov 2012 01:37:39 +
>
> The actual error seems to be:
>
> [ERROR] Failed to execute goal on project giraph-formats-contrib: Could not
> resolve dependencies for project
> org.apache.giraph:giraph-formats-contrib:jar:0.2-SNAPSHOT: Failure to find
> org.apache.giraph:giraph:jar:tests:0.2-SNAPSHOT
> inhttp://repo1.maven.org/maven2 was cached in the local repository,
> resolution will not be reattempted until the update interval of central has
> elapsed or updates are forced -> [Help 1]
>
> Try running "mvn install" at the top level first. That should put the test
> jar in your local repository.
>
> From: Rui Sarmento 
> Reply-To: "user@giraph.apache.org" 
> Date: Monday, November 26, 2012 5:15 PM
> To: Giraph Support 
> Subject: RE: Problem Building Apache Giraph Formats
>
> Hi Alessandro,
>
> The command is "mvn compile", Everything compiles with success except Apache
> Giraph Formats
>
> [INFO]
> 
> [INFO] Building Apache Giraph Formats 0.2-SNAPSHOT
> [INFO]
> 
> [WARNING] The POM for org.apache.hadoop:hadoop-core:jar:0.20.1 is missing,
> no dependency information available
> [WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to
> local.repository (file:../../local.repository/trunk): No connector available
> to access repository local.repository (file:../../local.repository/trunk) of
> type legacy using the available factories WagonRepositoryConnectorFactory
> [WARNING] Failure to transfer asm:asm/maven-metadata.xml from
> file:../../local.repository/trunk was cached in the local repository,
> resolution will not be reattempted until the update interval of
> local.repository has elapsed or updates are forced. Original error: Could
> not transfer metadata asm:asm/maven-metadata.xml from/to local.repository
> (file:../../local.repository/trunk): No connector available to access
> repository local.repository (file:../../local.repository/trunk) of type
> legacy using the available factories WagonRepositoryConnectorFactory
> [INFO]
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Giraph Parent .. SUCCESS [0.002s]
> [I

RE: Problem Building Apache Giraph Formats

[Rui Sarmento]

I'm using "grep -R -l" command but is taking to long, any advise for another 
command? Is there a faster one available, its more than 800 documents.
Thanks,
Rui

From: alessan...@fb.com
To: user@giraph.apache.org
Subject: Re: Problem Building Apache Giraph Formats
Date: Tue, 27 Nov 2012 02:05:18 +






You could take a look at surefire-reports and post the failing test(s). Search 
for the string "error" in that folder.





From: Rui Sarmento 

Reply-To: "user@giraph.apache.org" 

Date: Monday, November 26, 2012 5:58 PM

To: Giraph Support 

Subject: RE: Problem Building Apache Giraph Formats








running "mvn install" on trunk dir gives this result:



[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Giraph Parent .. SUCCESS [0.563s]
[INFO] Apache Giraph . FAILURE [2:23.047s]
[INFO] Apache Giraph Formats . SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 2:23.882s
[INFO] Finished at: Tue Nov 27 01:54:17 WET 2012
[INFO] Final Memory: 27M/811M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.6:test (default-test) on 
project giraph: There are test failures.
[ERROR]
[ERROR] Please refer to 
/home/110414015/trunk/giraph/target/munged/surefire-reports for the individual 
test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :giraph



Do you need any test result in particular?



Regards





From: alessan...@fb.com

To: user@giraph.apache.org

Subject: Re: Problem Building Apache Giraph Formats

Date: Tue, 27 Nov 2012 01:37:39 +



The actual error seems to be:



[ERROR] Failed to execute goal on project giraph-formats-contrib: Could not 
resolve dependencies for project 
org.apache.giraph:giraph-formats-contrib:jar:0.2-SNAPSHOT: Failure to find 
org.apache.giraph:giraph:jar:tests:0.2-SNAPSHOT
 inhttp://repo1.maven.org/maven2 was
 cached in the local repository, resolution will not be reattempted until the 
update interval of central has elapsed or updates are forced -> [Help 1]



Try running "mvn install" at the top level first. That should put the test jar 
in your local repository.





From: Rui Sarmento 

Reply-To: "user@giraph.apache.org" 

Date: Monday, November 26, 2012 5:15 PM

To: Giraph Support 

Subject: RE: Problem Building Apache Giraph Formats







Hi Alessandro,



The command is "mvn compile", Everything compiles with success except Apache 
Giraph Formats




[INFO] 
[INFO] Building Apache Giraph Formats 0.2-SNAPSHOT
[INFO] 
[WARNING] The POM for org.apache.hadoop:hadoop-core:jar:0.20.1 is missing, no 
dependency information available
[WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to 
local.repository (file:../../local.repository/trunk): No connector available to 
access repository local.repository (file:../../local.repository/trunk)
 of type legacy using the available factories WagonRepositoryConnectorFactory
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of local.repository
 has elapsed or updates are forced. Original error: Could not transfer metadata 
asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk)
 of type legacy using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Giraph Parent .. SUCCESS [0.002s]
[INFO] Apache Giraph . SUCCESS [10.759s]
[INFO] Apache Giraph Formats . FAILURE [0.362s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 11.381s
[INFO] Finished at: Wed Nov 21 15:34:

Re: What a "worker" really is and other interesting runtime information

[Avery Ching]

Hi Alexandros,

The extra task is for the master process (a coordination task). In your 
case, since you are using a single machine, you can use a single task.


-Dgiraph.SplitMasterWorker=false

and you can try multithreading instead of multiple workers.

-Dgiraph.numComputeThreads=12

The reason why cpu usage increases is due to netty threads to handle 
network requests.  By using multithreading instead, you should bypass this.


Avery

On 11/27/12 9:40 AM, Alexandros Daglis wrote:

Hello everybody,

I went through most of the documentation I could find for Giraph and 
also most of the messages in this email list, but still I have not 
figured out precisely what a "worker" really is. I would really 
appreciate it if you could help me understand how the framework works.


At first I thought that a worker has a one-to-one correspondence to a 
map task. Apparently this is not exactly the case, since I have 
noticed that if I ask for x workers, the job finishes after having 
used x+1 map tasks. What is this extra task for?


I have been trying out the example SSSP application on a single node 
with 12 cores. Giving an input graph of ~400MB and using 1 worker, 
around 10 GBs of memory are used during execution. What intrigues me 
is that if I use 2 workers for the same input (and without limiting 
memory per map task), double the memory will be used. Furthermore, 
there will be no improvement in performance. I rather notice a 
slowdown. Are these observations normal?


Might it be the case that 1 and 2 workers are very few and I should go 
to the 30-100 range that is the proposed number of mappers for a 
conventional MapReduce job?


Finally, a last observation. Even though I use only 1 worker, I see 
that there are significant periods during execution where up to 90% of 
the 12 cores computing power is consumed, that is, almost 10 cores are 
used in parallel. Does each worker spawn multiple threads and 
dynamically balances the load to utilize the available hardware?


Thanks a lot in advance!

Best,
Alexandros

Re: Minimum superstep time

[Sebastian Schelter]

Are the vertices which are active in supersteps dependent on each other?
If this is not the case, you could try to execute the messaging
simultaneously.

Could you give a little more details about the problem, which you are
trying to solve?

/s


On 27.11.2012 19:07, Jonathan Bishop wrote:
> Hi,
> 
> I am involved in a project which requires thousands of supersteps. In each
> superstep there are about 1 thousand vertices, each sending a message. The
> message size is not large, about 40 bytes or so.
> 
> I am seeing abouth 3000-5000ms per superstep. I am curious what others with
> similar problem sizes are seeing.
> 
> Thanks,
> 
> Jon
> 

Re: Running Giraph with secure Hadoop

[Eugene Koontz]

Hi Prajakta,
Are you sure that your KDC is running? It looks like from your 
stack that it might not be, or that you can't connect to it. Can you 
telnet or nc to port 88 on the server that the KDC is running?

-Eugene

On 11/19/12 6:48 PM, Prajakta Kalmegh wrote:

Hi

Re-posting my previous problem:
I am trying to configure Hadoop with Kerberos following the
instructions given at
.

12/11/19 18:30:26 ERROR namenode.NameNode: java.io.IOException: Login
failure for pkalmegh@HADOOP.LOCALDOMAIN from keytab
/home/pkalmegh/kerb-setup/pkalmegh.keytab
at 
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:630)



Caused by: javax.security.auth.login.LoginException: Connection refused
at 
com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:700)
at 
com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)

What a "worker" really is and other interesting runtime information

[Alexandros Daglis]

Hello everybody,

I went through most of the documentation I could find for Giraph and also
most of the messages in this email list, but still I have not figured out
precisely what a "worker" really is. I would really appreciate it if you
could help me understand how the framework works.

At first I thought that a worker has a one-to-one correspondence to a map
task. Apparently this is not exactly the case, since I have noticed that if
I ask for x workers, the job finishes after having used x+1 map tasks. What
is this extra task for?

I have been trying out the example SSSP application on a single node with
12 cores. Giving an input graph of ~400MB and using 1 worker, around 10 GBs
of memory are used during execution. What intrigues me is that if I use 2
workers for the same input (and without limiting memory per map task),
double the memory will be used. Furthermore, there will be no improvement
in performance. I rather notice a slowdown. Are these observations normal?

Might it be the case that 1 and 2 workers are very few and I should go to
the 30-100 range that is the proposed number of mappers for a conventional
MapReduce job?

Finally, a last observation. Even though I use only 1 worker, I see that
there are significant periods during execution where up to 90% of the 12
cores computing power is consumed, that is, almost 10 cores are used in
parallel. Does each worker spawn multiple threads and dynamically balances
the load to utilize the available hardware?

Thanks a lot in advance!

Best,
Alexandros