[jira] [Commented] (GIRAPH-981) hadoop_2.6 profile
[ https://issues.apache.org/jira/browse/GIRAPH-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287059#comment-14287059 ] Eugene Koontz commented on GIRAPH-981: -- (no test failures either). > hadoop_2.6 profile > -- > > Key: GIRAPH-981 > URL: https://issues.apache.org/jira/browse/GIRAPH-981 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz >Assignee: Eugene Koontz > Fix For: 1.2.0-SNAPSHOT > > Attachments: GIRAPH-981.patch, GIRAPH-981.patch, GIRAPH-981.patch > > > Create a profile to build against the latest Hadoop release: > http://hadoop.apache.org/#18+November%2C+2014%3A+release+2.6.0+available -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GIRAPH-981) hadoop_2.6 profile
[
https://issues.apache.org/jira/browse/GIRAPH-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287056#comment-14287056
]
Eugene Koontz commented on GIRAPH-981:
--
{{mvn -Phadoop_2 clean test}}
runs without errors for me with this patch.
> hadoop_2.6 profile
> --
>
> Key: GIRAPH-981
> URL: https://issues.apache.org/jira/browse/GIRAPH-981
> Project: Giraph
> Issue Type: Improvement
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Fix For: 1.2.0-SNAPSHOT
>
> Attachments: GIRAPH-981.patch, GIRAPH-981.patch, GIRAPH-981.patch
>
>
> Create a profile to build against the latest Hadoop release:
> http://hadoop.apache.org/#18+November%2C+2014%3A+release+2.6.0+available
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (GIRAPH-981) hadoop_2.6 profile
[ https://issues.apache.org/jira/browse/GIRAPH-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-981: - Attachment: GIRAPH-981.patch add comment to hadoop_2 profile: 'this profile (hadoop_2) should track the latest Apache Hadoop release, per: http://hadoop.apache.org/#News' > hadoop_2.6 profile > -- > > Key: GIRAPH-981 > URL: https://issues.apache.org/jira/browse/GIRAPH-981 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz >Assignee: Eugene Koontz > Fix For: 1.2.0-SNAPSHOT > > Attachments: GIRAPH-981.patch, GIRAPH-981.patch, GIRAPH-981.patch > > > Create a profile to build against the latest Hadoop release: > http://hadoop.apache.org/#18+November%2C+2014%3A+release+2.6.0+available -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GIRAPH-981) hadoop_2.6 profile
[ https://issues.apache.org/jira/browse/GIRAPH-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287037#comment-14287037 ] Eugene Koontz commented on GIRAPH-981: -- Thanks for the comments Lee and Roman! I agree with you both that hadoop_2 should simply track the latest Hadoop 2 release. I'll put in a comment to that effect in the patch. > hadoop_2.6 profile > -- > > Key: GIRAPH-981 > URL: https://issues.apache.org/jira/browse/GIRAPH-981 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz >Assignee: Eugene Koontz > Fix For: 1.2.0-SNAPSHOT > > Attachments: GIRAPH-981.patch, GIRAPH-981.patch > > > Create a profile to build against the latest Hadoop release: > http://hadoop.apache.org/#18+November%2C+2014%3A+release+2.6.0+available -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GIRAPH-981) hadoop_2.6 profile
[ https://issues.apache.org/jira/browse/GIRAPH-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-981: - Attachment: GIRAPH-981.patch update hadoop_2 profile: Roman Shaposhnik's proposal to simply update hadoop_2 to use latest Hadoop release (2.6.0) > hadoop_2.6 profile > -- > > Key: GIRAPH-981 > URL: https://issues.apache.org/jira/browse/GIRAPH-981 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz >Assignee: Eugene Koontz > Fix For: 1.2.0-SNAPSHOT > > Attachments: GIRAPH-981.patch, GIRAPH-981.patch > > > Create a profile to build against the latest Hadoop release: > http://hadoop.apache.org/#18+November%2C+2014%3A+release+2.6.0+available -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GIRAPH-981) hadoop_2.6 profile
[ https://issues.apache.org/jira/browse/GIRAPH-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273082#comment-14273082 ] Eugene Koontz commented on GIRAPH-981: -- Sounds like a good idea, Roman. I'll update the patch. > hadoop_2.6 profile > -- > > Key: GIRAPH-981 > URL: https://issues.apache.org/jira/browse/GIRAPH-981 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz >Assignee: Eugene Koontz > Fix For: 1.2.0-SNAPSHOT > > Attachments: GIRAPH-981.patch > > > Create a profile to build against the latest Hadoop release: > http://hadoop.apache.org/#18+November%2C+2014%3A+release+2.6.0+available -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GIRAPH-982) 'mvn -Phadoop_2 clean test' fails with 7 errors in giraph-rexster
[ https://issues.apache.org/jira/browse/GIRAPH-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272674#comment-14272674 ] Eugene Koontz commented on GIRAPH-982: -- Transcript of attempt: https://gist.github.com/ekoontz/07657cb998025b94de6e#file-gistfile1-txt > 'mvn -Phadoop_2 clean test' fails with 7 errors in giraph-rexster > - > > Key: GIRAPH-982 > URL: https://issues.apache.org/jira/browse/GIRAPH-982 > Project: Giraph > Issue Type: Improvement >Affects Versions: 1.2.0-SNAPSHOT >Reporter: Eugene Koontz >Assignee: Eugene Koontz > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GIRAPH-982) 'mvn -Phadoop_2 clean test' fails with 7 errors in giraph-rexster
Eugene Koontz created GIRAPH-982: Summary: 'mvn -Phadoop_2 clean test' fails with 7 errors in giraph-rexster Key: GIRAPH-982 URL: https://issues.apache.org/jira/browse/GIRAPH-982 Project: Giraph Issue Type: Improvement Affects Versions: 1.2.0-SNAPSHOT Reporter: Eugene Koontz Assignee: Eugene Koontz -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GIRAPH-981) hadoop_2.6 profile
[ https://issues.apache.org/jira/browse/GIRAPH-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272669#comment-14272669 ] Eugene Koontz commented on GIRAPH-981: -- Also made a small, unrelated correction in the top-level README. > hadoop_2.6 profile > -- > > Key: GIRAPH-981 > URL: https://issues.apache.org/jira/browse/GIRAPH-981 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz >Assignee: Eugene Koontz > Fix For: 1.2.0-SNAPSHOT > > Attachments: GIRAPH-981.patch > > > Create a profile to build against the latest Hadoop release: > http://hadoop.apache.org/#18+November%2C+2014%3A+release+2.6.0+available -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GIRAPH-981) hadoop_2.6 profile
[ https://issues.apache.org/jira/browse/GIRAPH-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-981: - Attachment: GIRAPH-981.patch > hadoop_2.6 profile > -- > > Key: GIRAPH-981 > URL: https://issues.apache.org/jira/browse/GIRAPH-981 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz >Assignee: Eugene Koontz > Fix For: 1.2.0-SNAPSHOT > > Attachments: GIRAPH-981.patch > > > Create a profile to build against the latest Hadoop release: > http://hadoop.apache.org/#18+November%2C+2014%3A+release+2.6.0+available -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GIRAPH-981) hadoop_2.6 profile
Eugene Koontz created GIRAPH-981: Summary: hadoop_2.6 profile Key: GIRAPH-981 URL: https://issues.apache.org/jira/browse/GIRAPH-981 Project: Giraph Issue Type: Improvement Reporter: Eugene Koontz Assignee: Eugene Koontz Create a profile to build against the latest Hadoop release: http://hadoop.apache.org/#18+November%2C+2014%3A+release+2.6.0+available -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GIRAPH-840) Upgrade to netty 4
[ https://issues.apache.org/jira/browse/GIRAPH-840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900564#comment-13900564 ] Eugene Koontz commented on GIRAPH-840: -- Hi Pavan, Currently there are no SASL unit tests - I added GIRAPH-852 for these. I have not had a chance to test your patch with an existing SASL setup. However the changes you made related to SASL look fine to me. > Upgrade to netty 4 > -- > > Key: GIRAPH-840 > URL: https://issues.apache.org/jira/browse/GIRAPH-840 > Project: Giraph > Issue Type: Improvement >Reporter: Pavan Kumar >Assignee: Pavan Kumar > Fix For: 1.1.0 > > Attachments: GIRAPH-840.patch, GIRAPH-840.patch > > > Off late netty 4 has earned so much praise in the community. For example, > https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead > A switch to netty 4 enables a significant reduction in gc pressure and also > huge performance gains. I started working on this last Sunday and have a > patch that shows performance gains on the order of 15-25% (total execution > time) for some applications at Facebook. However, I only tested it with > hadoop_facebook. So there might be issues with SASL path. > I will release the patch today and want to open up a discussion if anyone is > using the secure feature anymore. If not we can just deprecate it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (GIRAPH-852) Add SASL unit tests
Eugene Koontz created GIRAPH-852: Summary: Add SASL unit tests Key: GIRAPH-852 URL: https://issues.apache.org/jira/browse/GIRAPH-852 Project: Giraph Issue Type: Improvement Reporter: Eugene Koontz Currently there are no unit tests for the SASL functionality. See linked JIRAs in GIRAPH-211 for some references to how to create these tests such as HADOOP-8078. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (GIRAPH-840) Upgrade to netty 4
[ https://issues.apache.org/jira/browse/GIRAPH-840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-840: - Attachment: GIRAPH-840.patch Re-created Pavan's patch, works for me now. > Upgrade to netty 4 > -- > > Key: GIRAPH-840 > URL: https://issues.apache.org/jira/browse/GIRAPH-840 > Project: Giraph > Issue Type: Improvement >Reporter: Pavan Kumar >Assignee: Pavan Kumar > Fix For: 1.1.0 > > Attachments: GIRAPH-840.patch, GIRAPH-840.patch > > > Off late netty 4 has earned so much praise in the community. For example, > https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead > A switch to netty 4 enables a significant reduction in gc pressure and also > huge performance gains. I started working on this last Sunday and have a > patch that shows performance gains on the order of 15-25% (total execution > time) for some applications at Facebook. However, I only tested it with > hadoop_facebook. So there might be issues with SASL path. > I will release the patch today and want to open up a discussion if anyone is > using the secure feature anymore. If not we can just deprecate it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GIRAPH-840) Upgrade to netty 4
[ https://issues.apache.org/jira/browse/GIRAPH-840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13898819#comment-13898819 ] Eugene Koontz commented on GIRAPH-840: -- Pavan thanks for the updating the review. It looks good and I recognize the significant work it took; especially having to mess with the SASL channel handlers :) My only problem is that I cannot apply the latest patch: https://gist.github.com/ekoontz/8950523 Am I doing something wrong? Do I have the wrong URL for the latest patch? > Upgrade to netty 4 > -- > > Key: GIRAPH-840 > URL: https://issues.apache.org/jira/browse/GIRAPH-840 > Project: Giraph > Issue Type: Improvement >Reporter: Pavan Kumar >Assignee: Pavan Kumar > Fix For: 1.1.0 > > > Off late netty 4 has earned so much praise in the community. For example, > https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead > A switch to netty 4 enables a significant reduction in gc pressure and also > huge performance gains. I started working on this last Sunday and have a > patch that shows performance gains on the order of 15-25% (total execution > time) for some applications at Facebook. However, I only tested it with > hadoop_facebook. So there might be issues with SASL path. > I will release the patch today and want to open up a discussion if anyone is > using the secure feature anymore. If not we can just deprecate it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GIRAPH-840) Upgrade to netty 4
[
https://issues.apache.org/jira/browse/GIRAPH-840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890477#comment-13890477
]
Eugene Koontz commented on GIRAPH-840:
--
Gettting a failure with {{mvn -Phadoop_1.0 clean test}}:
https://gist.github.com/ekoontz/8799669
> Upgrade to netty 4
> --
>
> Key: GIRAPH-840
> URL: https://issues.apache.org/jira/browse/GIRAPH-840
> Project: Giraph
> Issue Type: Improvement
>Reporter: Pavan Kumar
>Assignee: Pavan Kumar
>
> Off late netty 4 has earned so much praise in the community. For example,
> https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead
> A switch to netty 4 enables a significant reduction in gc pressure and also
> huge performance gains. I started working on this last Sunday and have a
> patch that shows performance gains on the order of 15-25% (total execution
> time) for some applications at Facebook. However, I only tested it with
> hadoop_facebook. So there might be issues with SASL path.
> I will release the patch today and want to open up a discussion if anyone is
> using the secure feature anymore. If not we can just deprecate it.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (GIRAPH-845) Fix builds on builds.apache.org
[ https://issues.apache.org/jira/browse/GIRAPH-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-845: - Attachment: Screen Shot 2014-02-03 at 11.46.15 PM.png > Fix builds on builds.apache.org > --- > > Key: GIRAPH-845 > URL: https://issues.apache.org/jira/browse/GIRAPH-845 > Project: Giraph > Issue Type: Bug > Components: build >Reporter: Eugene Koontz >Assignee: Eugene Koontz > Attachments: Screen Shot 2014-02-03 at 11.46.15 PM.png > > > Looks like all of the giraph builds are failing right now (see screenshot). > Let's get these blue again. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (GIRAPH-845) Fix builds on builds.apache.org
[ https://issues.apache.org/jira/browse/GIRAPH-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz reassigned GIRAPH-845: Assignee: Eugene Koontz > Fix builds on builds.apache.org > --- > > Key: GIRAPH-845 > URL: https://issues.apache.org/jira/browse/GIRAPH-845 > Project: Giraph > Issue Type: Bug > Components: build >Reporter: Eugene Koontz >Assignee: Eugene Koontz > Attachments: Screen Shot 2014-02-03 at 11.46.15 PM.png > > > Looks like all of the giraph builds are failing right now (see screenshot). > Let's get these blue again. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (GIRAPH-845) Fix builds on builds.apache.org
Eugene Koontz created GIRAPH-845: Summary: Fix builds on builds.apache.org Key: GIRAPH-845 URL: https://issues.apache.org/jira/browse/GIRAPH-845 Project: Giraph Issue Type: Bug Components: build Reporter: Eugene Koontz Looks like all of the giraph builds are failing right now (see screenshot). Let's get these blue again. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (GIRAPH-841) GIRAPH-841: fix missing munge symbols error for non-munge-using profiles
[ https://issues.apache.org/jira/browse/GIRAPH-841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-841: - Summary: GIRAPH-841: fix missing munge symbols error for non-munge-using profiles (was: "mvn -P2.2.0 compile" fails with "The parameters 'symbols' for goal org.sonatype.plugins:munge-maven-plugin:1.0:munge are missing or invalid") > GIRAPH-841: fix missing munge symbols error for non-munge-using profiles > > > Key: GIRAPH-841 > URL: https://issues.apache.org/jira/browse/GIRAPH-841 > Project: Giraph > Issue Type: Bug > Components: conf and scripts >Affects Versions: 1.1.0 >Reporter: Eugene Koontz > Fix For: 1.1.0 > > > A simple fix is to globally define munge.symbols to a value that is never > used otherwise, e.g. "NO_MUNGE_SYMBOLS". This will be overridden by profiles > that need munge.symbols to be set to a specific value. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GIRAPH-840) Upgrade to netty 4
[
https://issues.apache.org/jira/browse/GIRAPH-840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890441#comment-13890441
]
Eugene Koontz commented on GIRAPH-840:
--
Pavan,
Impressive, looks like a lot of work went into this!
Avery,
{{mvn -P2.2.0 test}}
and
{{mvn -Phadoop_yarn -Dhadoop.version=2.2.0 test}}
work for me, provided I use Pavan's patch from this issue together with my
GIRAPH-841 patch.
I have not yet looked at the SASL-specific tests, but I assume those passed if
the whole thing passes. Will look more at it later tonight.
> Upgrade to netty 4
> --
>
> Key: GIRAPH-840
> URL: https://issues.apache.org/jira/browse/GIRAPH-840
> Project: Giraph
> Issue Type: Improvement
>Reporter: Pavan Kumar
>Assignee: Pavan Kumar
>
> Off late netty 4 has earned so much praise in the community. For example,
> https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead
> A switch to netty 4 enables a significant reduction in gc pressure and also
> huge performance gains. I started working on this last Sunday and have a
> patch that shows performance gains on the order of 15-25% (total execution
> time) for some applications at Facebook. However, I only tested it with
> hadoop_facebook. So there might be issues with SASL path.
> I will release the patch today and want to open up a discussion if anyone is
> using the secure feature anymore. If not we can just deprecate it.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (GIRAPH-839) NettyWorkerAggregatorRequestProcessor tries to reuse request objects
[
https://issues.apache.org/jira/browse/GIRAPH-839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890413#comment-13890413
]
Eugene Koontz commented on GIRAPH-839:
--
Hm, looks like the build failed because a test couldn't connect to ZooKeeper?
https://builds.apache.org/job/Giraph-trunk-Commit/1403/consoleText (warning:
big output)
{code}
14/02/03 17:53:01 INFO zookeeper.ClientCnxn: Opening socket connection to
server localhost/127.0.0.1:22182
14/02/03 17:53:01 WARN zookeeper.ClientCnxn: Session 0x143f8e19d350002 for
server null, unexpected error, closing socket connection and attempting
reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
14/02/03 17:53:02 INFO server.SessionTrackerImpl: SessionTrackerImpl exited
loop!
{code}
> NettyWorkerAggregatorRequestProcessor tries to reuse request objects
>
>
> Key: GIRAPH-839
> URL: https://issues.apache.org/jira/browse/GIRAPH-839
> Project: Giraph
> Issue Type: Bug
>Reporter: Pavan Kumar
>Assignee: Pavan Kumar
> Attachments: GIRAPH-839.patch
>
>
> I was working on upgrading netty to 4.0.14.Final. It is mostly done, after
> some polishing I will submit a patch for that soon.
> While working on the above, I discovered a bug in
> NettyWorkerAggregatorRequestProcessor. The distributeAggregators method is
> reusing SendAggregatorsToWorkerRequest objects, which is incorrect since each
> task has a different task id & (potentially a different) request id. It does
> not cause any problem with current netty but in the new version I guess due
> to delayed serialization the object cannot be reused.
> Anyway, moving it inside the for loop does not cause any issues with current
> netty as well.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (GIRAPH-841) "mvn -P2.2.0 compile" fails with "The parameters 'symbols' for goal org.sonatype.plugins:munge-maven-plugin:1.0:munge are missing or invalid"
[ https://issues.apache.org/jira/browse/GIRAPH-841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889034#comment-13889034 ] Eugene Koontz commented on GIRAPH-841: -- https://reviews.apache.org/r/17646/ > "mvn -P2.2.0 compile" fails with "The parameters 'symbols' for goal > org.sonatype.plugins:munge-maven-plugin:1.0:munge are missing or invalid" > - > > Key: GIRAPH-841 > URL: https://issues.apache.org/jira/browse/GIRAPH-841 > Project: Giraph > Issue Type: Bug > Components: conf and scripts >Affects Versions: 1.1.0 >Reporter: Eugene Koontz > Fix For: 1.1.0 > > > A simple fix is to globally define munge.symbols to a value that is never > used otherwise, e.g. "NO_MUNGE_SYMBOLS". This will be overridden by profiles > that need munge.symbols to be set to a specific value. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (GIRAPH-841) "mvn -P2.2.0 compile" fails with "The parameters 'symbols' for goal org.sonatype.plugins:munge-maven-plugin:1.0:munge are missing or invalid"
Eugene Koontz created GIRAPH-841: Summary: "mvn -P2.2.0 compile" fails with "The parameters 'symbols' for goal org.sonatype.plugins:munge-maven-plugin:1.0:munge are missing or invalid" Key: GIRAPH-841 URL: https://issues.apache.org/jira/browse/GIRAPH-841 Project: Giraph Issue Type: Bug Components: conf and scripts Affects Versions: 1.1.0 Reporter: Eugene Koontz Fix For: 1.1.0 A simple fix is to globally define munge.symbols to a value that is never used otherwise, e.g. "NO_MUNGE_SYMBOLS". This will be overridden by profiles that need munge.symbols to be set to a specific value. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GIRAPH-840) Upgrade to netty 4
[ https://issues.apache.org/jira/browse/GIRAPH-840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889013#comment-13889013 ] Eugene Koontz commented on GIRAPH-840: -- Hi Pavan, Looking forward to seeing your patch. Let's see if we can get it to work with other profiles and with SASL. -Eugene > Upgrade to netty 4 > -- > > Key: GIRAPH-840 > URL: https://issues.apache.org/jira/browse/GIRAPH-840 > Project: Giraph > Issue Type: Improvement >Reporter: Pavan Kumar >Assignee: Pavan Kumar > > Off late netty 4 has earned so much praise in the community. For example, > https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead > A switch to netty 4 enables a significant reduction in gc pressure and also > huge performance gains. I started working on this last Sunday and have a > patch that shows performance gains on the order of 15-25% (total execution > time) for some applications at Facebook. However, I only tested it with > hadoop_facebook. So there might be issues with SASL path. > I will release the patch today and want to open up a discussion if anyone is > using the secure feature anymore. If not we can just deprecate it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GIRAPH-629) YARN profile is broken when compiled against hadoop-2.0.4
[
https://issues.apache.org/jira/browse/GIRAPH-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631930#comment-13631930
]
Eugene Koontz commented on GIRAPH-629:
--
I'm able to successfully compile Giraph using Hadoop 2.0.4 by using -alpha
rather than -SNAPSHOT, that is:
{code}
mvn -Phadoop_yarn -DskipTests -Dhadoop.version=2.0.4-alpha clean package
{code}
Perhaps the published 2.0.4-SNAPSHOT jars are old and do not contain the
AMResponse class.
Full instructions that worked for me:
# {{cd ~}}
# {{git clone git://git.apache.org/hadoop-common.git}}
# {{cd hadoop-common}}
# git checkout apache branch-2.0.4-alpha (commit e8286885032142 as of now)
# {{mvn -Pdist -DskipTests install}}
# {{cd ~/giraph}}
# git checkout apache trunk (commit b77acf2ef54d14b1566e2e as of now)
# {{mvn -Phadoop_yarn -Dhadoop.version=2.0.4-alpha clean package}}
Final step yields:
{code}
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Giraph Parent .. SUCCESS [0.728s]
[INFO] Apache Giraph Core SUCCESS [1:05.842s]
[INFO] Apache Giraph Hive I/O SUCCESS [27.309s]
[INFO] Apache Giraph Examples SUCCESS [1:22.994s]
[INFO]
[INFO] BUILD SUCCESS
{code}
> YARN profile is broken when compiled against hadoop-2.0.4
> -
>
> Key: GIRAPH-629
> URL: https://issues.apache.org/jira/browse/GIRAPH-629
> Project: Giraph
> Issue Type: Bug
> Components: build
>Affects Versions: 1.0.0
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>
> {noformat}
> $ mvn -Phadoop_yarn -DskipTests -Dhadoop.version=2.0.4-SNAPSHOT clean package
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Giraph Parent .. SUCCESS [1.359s]
> [INFO] Apache Giraph Core FAILURE [15.319s]
> [INFO] Apache Giraph Hive I/O SKIPPED
> [INFO] Apache Giraph Examples SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 17.374s
> [INFO] Finished at: Fri Apr 12 17:21:11 PDT 2013
> [INFO] Final Memory: 39M/481M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile)
> on project giraph-core: Compilation failure: Compilation failure:
> [ERROR]
> /tmp/giraph/giraph-core/target/munged/main/org/apache/giraph/yarn/GiraphApplicationMaster.java:[46,42]
> cannot find symbol
> [ERROR] symbol : class AMResponse
> [ERROR] location: package org.apache.hadoop.yarn.api.records
> [ERROR]
> /tmp/giraph/giraph-core/target/munged/main/org/apache/giraph/yarn/GiraphApplicationMaster.java:[206,42]
> cannot find symbol
> [ERROR] symbol : class AMResponse
> [ERROR] location: class org.apache.giraph.yarn.GiraphApplicationMaster
> [ERROR]
> /tmp/giraph/giraph-core/target/munged/main/org/apache/giraph/yarn/GiraphApplicationMaster.java:[291,47]
> cannot find symbol
> [ERROR] symbol : class AMResponse
> [ERROR] location: class org.apache.giraph.yarn.GiraphApplicationMaster
> [ERROR]
> /tmp/giraph/giraph-core/target/munged/main/org/apache/giraph/yarn/GiraphApplicationMaster.java:[368,11]
> cannot find symbol
> [ERROR] symbol : class AMResponse
> [ERROR] location: class org.apache.giraph.yarn.GiraphApplicationMaster
> [ERROR]
> /tmp/giraph/giraph-core/target/munged/main/org/apache/giraph/yarn/GiraphApplicationMaster.java:[398,35]
> cannot find symbol
> [ERROR] symbol : class AMResponse
> [ERROR] location: class org.apache.giraph.yarn.GiraphApplicationMaster
> [ERROR]
> /tmp/giraph/giraph-core/target/munged/main/org/apache/giraph/yarn/GiraphApplicationMaster.java:[178,7]
> cannot find symbol
> [ERROR] symbol : class AMResponse
> [ERROR] location: class org.apache.giraph.yarn.GiraphApplicationMaster
> [ERROR]
> /tmp/giraph/giraph-core/target/munged/main/org/apache/giraph/yarn/GiraphApplicationMaster.java:[255,26]
> cannot find symbol
> [ERROR] symbol : method getAMResponse()
> [ERROR] location: interface
> org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse
> [ERROR]
> /tmp/giraph/giraph-core/target/munged/main/org/apache/giraph/yarn/GiraphApplicationMaster.java:[296,37]
> cannot find symbol
> [ERROR] symbol : method getAMResponse()
> [ERROR] location: interface
> org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse
> [ERROR]
> /tmp/giraph/giraph-core/target/munged/main/org/apache/giraph/yarn/GiraphApplicationMaster.java:[32
[jira] [Commented] (GIRAPH-601) Exception when running pagerank benchmark with 6 or more workers on a pseudodistributed setup: SendVertexRequest cannot be cast to MasterRequest
[
https://issues.apache.org/jira/browse/GIRAPH-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620320#comment-13620320
]
Eugene Koontz commented on GIRAPH-601:
--
Also will run with your patch, Maja, thanks for supplying it.
> Exception when running pagerank benchmark with 6 or more workers on a
> pseudodistributed setup: SendVertexRequest cannot be cast to MasterRequest
>
>
> Key: GIRAPH-601
> URL: https://issues.apache.org/jira/browse/GIRAPH-601
> Project: Giraph
> Issue Type: Bug
>Reporter: Eugene Koontz
> Attachments: instrumentation.patch, print_addresses.patch
>
>
> Building Giraph with:
> {code}
> mvn -DskipTests -Phadoop_2.0.3 clean compile
> {code}
> Running pagerank like this:
> {code}
> $HADOOP_RUNTIME/bin/hadoop jar $JAR \
> org.apache.giraph.benchmark.PageRankBenchmark \
> -e 10 -s 10 -v -V 10 -w 6
> {code}
> I see this in
> /tmp/userlogs/application_1364578380737_0003/container_1364578380737_0003_01_02/
> :
> {code}
> 2013-03-29 10:58:06,371 DEBUG [org.apache.giraph.master.MasterThread]
> org.apache.giraph.master.BspServiceMaster: barrierOnWorkerList: Got finished
> worker list = [Eugenes-MacBook-Pro.local_1, Eugenes-MacBook-Pro.local_3],
> size = 2, worker list = [Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=2, port=30002), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=1, port=30001), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=4, port=30004), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=3, port=30003), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=5, port=30005), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=0, port=30010)], size = 6 from
> /_hadoopBsp/job_1364578380737_0003/_vertexInputSplitDoneDir
> 2013-03-29 10:58:06,373 WARN [netty-server-exec-3]
> org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
> Channel failed with remote address /172.16.175.1:56236
> java.lang.ClassCastException:
> org.apache.giraph.comm.requests.SendVertexRequest cannot be cast to
> org.apache.giraph.comm.requests.MasterRequest
> at
> org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:27)
> at
> org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:106)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:680)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-601) Exception when running pagerank benchmark with 6 or more workers on a pseudodistributed setup: SendVertexRequest cannot be cast to MasterRequest
[
https://issues.apache.org/jira/browse/GIRAPH-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620314#comment-13620314
]
Eugene Koontz commented on GIRAPH-601:
--
Hi Maja and Eli,
Thanks for helping with this. It seems to only happen with -w N with N > 5. So
I can run with no problem:
{code}
$HADOOP_RUNTIME/bin/hadoop jar $JAR \
org.apache.giraph.benchmark.PageRankBenchmark \
-e 10 -s 10 -v -V 10 -w 5
{code}
Also, other MR jobs like pi work fine. I've updated the description to reflect
the workers > 5 condition.
I think the port increment that is used to attempt a re-bind after binding
failure is related. Will try to dig more soon. There's also a small
case-sensitivity-of-hostnames bug which I'll file separately.
> Exception when running pagerank benchmark with 6 or more workers on a
> pseudodistributed setup: SendVertexRequest cannot be cast to MasterRequest
>
>
> Key: GIRAPH-601
> URL: https://issues.apache.org/jira/browse/GIRAPH-601
> Project: Giraph
> Issue Type: Bug
>Reporter: Eugene Koontz
> Attachments: instrumentation.patch, print_addresses.patch
>
>
> Building Giraph with:
> {code}
> mvn -DskipTests -Phadoop_2.0.3 clean compile
> {code}
> Running pagerank like this:
> {code}
> $HADOOP_RUNTIME/bin/hadoop jar $JAR \
> org.apache.giraph.benchmark.PageRankBenchmark \
> -e 10 -s 10 -v -V 10 -w 6
> {code}
> I see this in
> /tmp/userlogs/application_1364578380737_0003/container_1364578380737_0003_01_02/
> :
> {code}
> 2013-03-29 10:58:06,371 DEBUG [org.apache.giraph.master.MasterThread]
> org.apache.giraph.master.BspServiceMaster: barrierOnWorkerList: Got finished
> worker list = [Eugenes-MacBook-Pro.local_1, Eugenes-MacBook-Pro.local_3],
> size = 2, worker list = [Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=2, port=30002), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=1, port=30001), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=4, port=30004), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=3, port=30003), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=5, port=30005), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=0, port=30010)], size = 6 from
> /_hadoopBsp/job_1364578380737_0003/_vertexInputSplitDoneDir
> 2013-03-29 10:58:06,373 WARN [netty-server-exec-3]
> org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
> Channel failed with remote address /172.16.175.1:56236
> java.lang.ClassCastException:
> org.apache.giraph.comm.requests.SendVertexRequest cannot be cast to
> org.apache.giraph.comm.requests.MasterRequest
> at
> org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:27)
> at
> org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:106)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:680)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-601) Exception when running pagerank benchmark with 6 or more workers on a pseudodistributed setup: SendVertexRequest cannot be cast to MasterRequest
[
https://issues.apache.org/jira/browse/GIRAPH-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-601:
-
Summary: Exception when running pagerank benchmark with 6 or more workers
on a pseudodistributed setup: SendVertexRequest cannot be cast to MasterRequest
(was: Exception when running pagerank benchmark: SendVertexRequest cannot be
cast to MasterRequest)
> Exception when running pagerank benchmark with 6 or more workers on a
> pseudodistributed setup: SendVertexRequest cannot be cast to MasterRequest
>
>
> Key: GIRAPH-601
> URL: https://issues.apache.org/jira/browse/GIRAPH-601
> Project: Giraph
> Issue Type: Bug
>Reporter: Eugene Koontz
> Attachments: instrumentation.patch, print_addresses.patch
>
>
> Building Giraph with:
> {code}
> mvn -DskipTests -Phadoop_2.0.3 clean compile
> {code}
> Running pagerank like this:
> {code}
> $HADOOP_RUNTIME/bin/hadoop jar $JAR \
> org.apache.giraph.benchmark.PageRankBenchmark \
> -e 10 -s 10 -v -V 10 -w 6
> {code}
> I see this in
> /tmp/userlogs/application_1364578380737_0003/container_1364578380737_0003_01_02/
> :
> {code}
> 2013-03-29 10:58:06,371 DEBUG [org.apache.giraph.master.MasterThread]
> org.apache.giraph.master.BspServiceMaster: barrierOnWorkerList: Got finished
> worker list = [Eugenes-MacBook-Pro.local_1, Eugenes-MacBook-Pro.local_3],
> size = 2, worker list = [Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=2, port=30002), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=1, port=30001), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=4, port=30004), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=3, port=30003), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=5, port=30005), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=0, port=30010)], size = 6 from
> /_hadoopBsp/job_1364578380737_0003/_vertexInputSplitDoneDir
> 2013-03-29 10:58:06,373 WARN [netty-server-exec-3]
> org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
> Channel failed with remote address /172.16.175.1:56236
> java.lang.ClassCastException:
> org.apache.giraph.comm.requests.SendVertexRequest cannot be cast to
> org.apache.giraph.comm.requests.MasterRequest
> at
> org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:27)
> at
> org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:106)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:680)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-601) Exception when running pagerank benchmark: SendVertexRequest cannot be cast to MasterRequest
[
https://issues.apache.org/jira/browse/GIRAPH-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617924#comment-13617924
]
Eugene Koontz commented on GIRAPH-601:
--
Thanks for the feedback Eli! This is where I was waylaid while hoping to take a
look at GIRAPH-13. Once I get this sorted out I hope to continue with the
latter!
Here's some output enhanced with taskId information for each worker and master:
{code}
ekoontz@Eugenes-MacBook-Pro ~$ perl format.pl < postprocessed.txt
application_1364578380737_0027/container_1364578380737_0027_01_02/syslog
=
INFO [main] org.apache.giraph.graph.GraphTaskManager: setup: Log level remains
at info
INFO [main] org.apache.giraph.graph.GraphTaskManager: Distributed cache is
empty. Assuming fatjar.
INFO [main] org.apache.giraph.graph.GraphTaskManager: setup: classpath @
/tmp/hadoop-yarn/staging/ekoontz/.staging/job_1364578380737_0027/job.jar for
job org.apache.giraph.benchmark.PageRankBenchmark
DEBUG [main] org.apache.giraph.graph.GraphTaskManager: splitMasterWorker: true
DEBUG [main] org.apache.giraph.graph.GraphTaskManager: taskPartition: 0
DEBUG [main] org.apache.giraph.graph.GraphTaskManager: splitMasterWorker is
true.
DEBUG [main] org.apache.giraph.graph.GraphTaskManager: splitMasterWorker is
true and zkAlreadyProvided=true.
DEBUG [main] org.apache.giraph.graph.GraphTaskManager: splitMasterWorker is
true and taskPartition (0) is less than masterCount (1), so MASTER_ONLY.
INFO [main] org.apache.giraph.graph.GraphTaskManager: setup: Starting up
BspServiceMaster (master thread)...
INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection,
connectString=localhost:2181 sessionTimeout=6
watcher=org.apache.giraph.master.BspServiceMaster@47875da7
INFO [main] org.apache.giraph.graph.GraphTaskManager: map: No need to do
anything when not a worker
INFO [main] org.apache.giraph.graph.GraphTaskManager: cleanup: Starting for
MASTER_ONLY
INFO [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: becomeMaster: First child is
'/_hadoopBsp/job_1364578380737_0027/_masterElectionDir/eugenes-macbook-pro.local_000'
and my bid is
'/_hadoopBsp/job_1364578380737_0027/_masterElectionDir/eugenes-macbook-pro.local_000'
INFO [org.apache.giraph.master.MasterThread]
org.apache.giraph.comm.netty.NettyServer: NettyServer: Using execution handler
with 8 threads after requestFrameDecoder.
WARN [org.apache.giraph.master.MasterThread]
org.apache.hadoop.conf.Configuration: mapred.map.tasks is deprecated. Instead,
use mapreduce.job.maps
INFO [org.apache.giraph.master.MasterThread]
org.apache.giraph.comm.netty.NettyServer: start: Started server communication
server: Eugenes-MacBook-Pro.local/172.16.175.1:3 with up to 16 threads on
bind attempt 0 with sendBufferSize = 32768 receiveBufferSize = 524288 backlog =
6
INFO [org.apache.giraph.master.MasterThread]
org.apache.giraph.comm.netty.NettyClient: NettyClient: Using execution handler
with 8 threads after requestEncoder.
INFO [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: becomeMaster: I am now the master!
DEBUG [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: checkWorkers: Got event that health
registration changed, not using poll attempt
DEBUG [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: checkWorkers: Got event that health
registration changed, not using poll attempt
DEBUG [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: checkWorkers: Got event that health
registration changed, not using poll attempt
DEBUG [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: checkWorkers: Got event that health
registration changed, not using poll attempt
DEBUG [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: checkWorkers: Got event that health
registration changed, not using poll attempt
INFO [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: checkWorkers: Only found 5 responses
of 6 needed to start superstep -1. Reporting every 3 msecs, 565085 more
msecs left before giving up.
INFO [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: logMissingWorkersOnSuperstep: No
response from partition 6 (could be master)
INFO [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: checkWorkers: Only found 5 responses
of 6 needed to start superstep -1. Reporting every 3 msecs, 535076 more
msecs left before giving up.
INFO [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: logMissingWorkersOnSuperstep: No
response from partition 6 (could be master)
INFO [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: check
[jira] [Commented] (GIRAPH-601) Exception when running pagerank benchmark: SendVertexRequest cannot be cast to MasterRequest
[
https://issues.apache.org/jira/browse/GIRAPH-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617859#comment-13617859
]
Eugene Koontz commented on GIRAPH-601:
--
Using debug logging with instrumentation.patch (see attachments) shows that, in
fact, we are correctly respecting SplitMasterWorker's setting : that is, Master
is running in its own separate task, as expected:
{code}
application_1364578380737_0019/container_1364578380737_0019_01_02/syslog
29:2013-03-29 15:50:07,620 INFO [main]
org.apache.giraph.graph.GraphTaskManager: setup: Log level remains at info
30:2013-03-29 15:50:07,639 INFO [main]
org.apache.giraph.graph.GraphTaskManager: Distributed cache is empty. Assuming
fatjar.
31:2013-03-29 15:50:07,639 INFO [main]
org.apache.giraph.graph.GraphTaskManager: setup: classpath @
/tmp/hadoop-yarn/staging/ekoontz/.staging/job_1364578380737_0019/job.jar for
job org.apache.giraph.benchmark.PageRankBenchmark
32:2013-03-29 15:50:07,639 DEBUG [main]
org.apache.giraph.graph.GraphTaskManager: splitMasterWorker: true
34:2013-03-29 15:50:07,640 DEBUG [main]
org.apache.giraph.graph.GraphTaskManager: taskPartition: 0
35:2013-03-29 15:50:07,640 DEBUG [main]
org.apache.giraph.graph.GraphTaskManager: splitMasterWorker is true.
36:2013-03-29 15:50:07,640 DEBUG [main]
org.apache.giraph.graph.GraphTaskManager: splitMasterWorker is true and
zkAlreadyProvided=true.
37:2013-03-29 15:50:07,640 DEBUG [main]
org.apache.giraph.graph.GraphTaskManager: splitMasterWorker is true and
taskPartition (0) is less than masterCount (1), so MASTER_ONLY.
38:2013-03-29 15:50:07,640 INFO [main]
org.apache.giraph.graph.GraphTaskManager: setup: Starting up BspServiceMaster
(master thread)...
61:2013-03-29 15:50:07,709 INFO [main]
org.apache.giraph.graph.GraphTaskManager: map: No need to do anything when not
a worker
62:2013-03-29 15:50:07,709 INFO [main]
org.apache.giraph.graph.GraphTaskManager: cleanup: Starting for MASTER_ONLY
application_1364578380737_0019/container_1364578380737_0019_01_03/syslog
29:2013-03-29 15:50:09,090 INFO [main]
org.apache.giraph.graph.GraphTaskManager: setup: Log level remains at info
30:2013-03-29 15:50:09,110 INFO [main]
org.apache.giraph.graph.GraphTaskManager: Distributed cache is empty. Assuming
fatjar.
31:2013-03-29 15:50:09,110 INFO [main]
org.apache.giraph.graph.GraphTaskManager: setup: classpath @
/tmp/hadoop-yarn/staging/ekoontz/.staging/job_1364578380737_0019/job.jar for
job org.apache.giraph.benchmark.PageRankBenchmark
32:2013-03-29 15:50:09,110 DEBUG [main]
org.apache.giraph.graph.GraphTaskManager: splitMasterWorker: true
34:2013-03-29 15:50:09,110 DEBUG [main]
org.apache.giraph.graph.GraphTaskManager: taskPartition: 1
35:2013-03-29 15:50:09,110 DEBUG [main]
org.apache.giraph.graph.GraphTaskManager: splitMasterWorker is true.
36:2013-03-29 15:50:09,110 DEBUG [main]
org.apache.giraph.graph.GraphTaskManager: splitMasterWorker is true and
zkAlreadyProvided=true.
37:2013-03-29 15:50:09,111 DEBUG [main]
org.apache.giraph.graph.GraphTaskManager: splitMasterWorker is true and
taskPartition (1) is NOT less than masterCount (1), so WORKER_ONLY.
38:2013-03-29 15:50:09,111 INFO [main]
org.apache.giraph.graph.GraphTaskManager: setup: Starting up BspServiceWorker...
66:2013-03-29 15:50:09,323 INFO [main]
org.apache.giraph.graph.GraphTaskManager: setup: Registering health of this
worker...
application_1364578380737_0019/container_1364578380737_0019_01_04/syslog
29:2013-03-29 15:50:10,222 INFO [main]
org.apache.giraph.graph.GraphTaskManager: setup: Log level remains at info
30:2013-03-29 15:50:10,241 INFO [main]
org.apache.giraph.graph.GraphTaskManager: Distributed cache is empty. Assuming
fatjar.
31:2013-03-29 15:50:10,242 INFO [main]
org.apache.giraph.graph.GraphTaskManager: setup: classpath @
/tmp/hadoop-yarn/staging/ekoontz/.staging/job_1364578380737_0019/job.jar for
job org.apache.giraph.benchmark.PageRankBenchmark
32:2013-03-29 15:50:10,242 DEBUG [main]
org.apache.giraph.graph.GraphTaskManager: splitMasterWorker: true
34:2013-03-29 15:50:10,242 DEBUG [main]
org.apache.giraph.graph.GraphTaskManager: taskPartition: 2
35:2013-03-29 15:50:10,242 DEBUG [main]
org.apache.giraph.graph.GraphTaskManager: splitMasterWorker is true.
36:2013-03-29 15:50:10,242 DEBUG [main]
org.apache.giraph.graph.GraphTaskManager: splitMasterWorker is true and
zkAlreadyProvided=true.
37:2013-03-29 15:50:10,242 DEBUG [main]
org.apache.giraph.graph.GraphTaskManager: splitMasterWorker is true and
taskPartition (2) is NOT less than masterCount (1), so WORKER_ONLY.
38:2013-03-29 15:50:10,242 INFO [main]
org.apache.giraph.graph.GraphTaskManager: setup: Starting up BspServiceWorker...
66:2013-03-29 15:50:10,444 INFO [main]
org.apache.giraph.graph.GraphTaskManager: setup: Registering health of this
worker...
application_1364578380737_0019/container_1364578380737_0019_01_05/syslog
29:2013-0
[jira] [Updated] (GIRAPH-601) Exception when running pagerank benchmark: SendVertexRequest cannot be cast to MasterRequest
[
https://issues.apache.org/jira/browse/GIRAPH-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-601:
-
Attachment: instrumentation.patch
more DEBUG logging to report task => what-to-start* mapping, which is based on
configuration of splitMasterWorker and other parameters.
what-to-start = one or more of the set of {master,worker,zookeeper}.
> Exception when running pagerank benchmark: SendVertexRequest cannot be cast
> to MasterRequest
>
>
> Key: GIRAPH-601
> URL: https://issues.apache.org/jira/browse/GIRAPH-601
> Project: Giraph
> Issue Type: Bug
>Reporter: Eugene Koontz
> Attachments: instrumentation.patch
>
>
> Building Giraph with:
> {code}
> mvn -DskipTests -Phadoop_2.0.3 clean compile
> {code}
> Running pagerank like this:
> {code}
> $HADOOP_RUNTIME/bin/hadoop jar $JAR \
> org.apache.giraph.benchmark.PageRankBenchmark \
> -e 10 -s 10 -v -V 10 -w 6
> {code}
> I see this in
> /tmp/userlogs/application_1364578380737_0003/container_1364578380737_0003_01_02/
> :
> {code}
> 2013-03-29 10:58:06,371 DEBUG [org.apache.giraph.master.MasterThread]
> org.apache.giraph.master.BspServiceMaster: barrierOnWorkerList: Got finished
> worker list = [Eugenes-MacBook-Pro.local_1, Eugenes-MacBook-Pro.local_3],
> size = 2, worker list = [Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=2, port=30002), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=1, port=30001), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=4, port=30004), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=3, port=30003), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=5, port=30005), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=0, port=30010)], size = 6 from
> /_hadoopBsp/job_1364578380737_0003/_vertexInputSplitDoneDir
> 2013-03-29 10:58:06,373 WARN [netty-server-exec-3]
> org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
> Channel failed with remote address /172.16.175.1:56236
> java.lang.ClassCastException:
> org.apache.giraph.comm.requests.SendVertexRequest cannot be cast to
> org.apache.giraph.comm.requests.MasterRequest
> at
> org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:27)
> at
> org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:106)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:680)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-601) Exception when running pagerank benchmark: SendVertexRequest cannot be cast to MasterRequest
[
https://issues.apache.org/jira/browse/GIRAPH-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617812#comment-13617812
]
Eugene Koontz commented on GIRAPH-601:
--
Hi Maja,
Everything's on one machine, and the master and worker are the same task, which
seems incorrect since I have SplitMasterWorker set to true.
I need to track down why they are being started on the same task.
-Eugene
> Exception when running pagerank benchmark: SendVertexRequest cannot be cast
> to MasterRequest
>
>
> Key: GIRAPH-601
> URL: https://issues.apache.org/jira/browse/GIRAPH-601
> Project: Giraph
> Issue Type: Bug
>Reporter: Eugene Koontz
>
> Building Giraph with:
> {code}
> mvn -DskipTests -Phadoop_2.0.3 clean compile
> {code}
> Running pagerank like this:
> {code}
> $HADOOP_RUNTIME/bin/hadoop jar $JAR \
> org.apache.giraph.benchmark.PageRankBenchmark \
> -e 10 -s 10 -v -V 10 -w 6
> {code}
> I see this in
> /tmp/userlogs/application_1364578380737_0003/container_1364578380737_0003_01_02/
> :
> {code}
> 2013-03-29 10:58:06,371 DEBUG [org.apache.giraph.master.MasterThread]
> org.apache.giraph.master.BspServiceMaster: barrierOnWorkerList: Got finished
> worker list = [Eugenes-MacBook-Pro.local_1, Eugenes-MacBook-Pro.local_3],
> size = 2, worker list = [Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=2, port=30002), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=1, port=30001), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=4, port=30004), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=3, port=30003), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=5, port=30005), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=0, port=30010)], size = 6 from
> /_hadoopBsp/job_1364578380737_0003/_vertexInputSplitDoneDir
> 2013-03-29 10:58:06,373 WARN [netty-server-exec-3]
> org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
> Channel failed with remote address /172.16.175.1:56236
> java.lang.ClassCastException:
> org.apache.giraph.comm.requests.SendVertexRequest cannot be cast to
> org.apache.giraph.comm.requests.MasterRequest
> at
> org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:27)
> at
> org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:106)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:680)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-601) Exception when running pagerank benchmark: SendVertexRequest cannot be cast to MasterRequest
[
https://issues.apache.org/jira/browse/GIRAPH-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617717#comment-13617717
]
Eugene Koontz commented on GIRAPH-601:
--
Yes, I think somehow the configuration setting is not being respected even
though I'm seeing "split master/worker = true" in the console output:
{code}
.
013-03-29 13:47:13,094 INFO security.TokenCache
(TokenCache.java:obtainTokensForNamenodesInternal(125)) - Got dt for
hdfs://ekoontz1:8020; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ekoontz1,
Ident: (HDFS_DELEGATION_TOKEN token 17 for ekoontz)
2013-03-29 13:47:13,331 DEBUG bsp.BspInputFormat
(BspInputFormat.java:getMaxTasks(60)) - getMaxTasks: Max workers = 6, split
master/worker = true, total max tasks = 7
2013-03-29 13:47:13,374 INFO mapreduce.JobSubmitter
(JobSubmitter.java:submitJobInternal(369)) - number of splits:7
2013-03-29 13:47:13,388 WARN conf.Configuration
(Configuration.java:warnOnceIfDeprecated(817)) - mapred.jar is deprecated.
Instead, use mapreduce.job.jar
2013-03-29 13:47:13,389 WARN conf.Configuration
(Configuration.java:warnOnceIfDeprecated(817)) - mapred.reduce.tasks is
deprecated. Instead, use mapreduce.job.reduces
2013-03-29 13:47:13,389 WARN conf.Configuration
(Configuration.java:warnOnceIfDeprecated(817)) - mapreduce.map.class is
deprecated. Instead, use mapreduce.job.map.class
2013-03-29 13:47:13,390 WARN conf.Configuration
(Configuration.java:warnOnceIfDeprecated(817)) - mapred.job.name is deprecated.
Instead, use mapreduce.job.name
2013-03-29 13:47:13,390 WARN conf.Configuration
(Configuration.java:warnOnceIfDeprecated(817)) - mapreduce.inputformat.class is
deprecated. Instead, use mapreduce.job.inputformat.class
2013-03-29 13:47:13,390 WARN conf.Configuration
(Configuration.java:warnOnceIfDeprecated(817)) - mapreduce.outputformat.class
is deprecated. Instead, use mapreduce.job.outputformat.class
2013-03-29 13:47:13,390 WARN conf.Configuration
(Configuration.java:warnOnceIfDeprecated(817)) - mapred.map.tasks is
deprecated. Instead, use mapreduce.job.maps
...
{code}
> Exception when running pagerank benchmark: SendVertexRequest cannot be cast
> to MasterRequest
>
>
> Key: GIRAPH-601
> URL: https://issues.apache.org/jira/browse/GIRAPH-601
> Project: Giraph
> Issue Type: Bug
>Reporter: Eugene Koontz
>
> Building Giraph with:
> {code}
> mvn -DskipTests -Phadoop_2.0.3 clean compile
> {code}
> Running pagerank like this:
> {code}
> $HADOOP_RUNTIME/bin/hadoop jar $JAR \
> org.apache.giraph.benchmark.PageRankBenchmark \
> -e 10 -s 10 -v -V 10 -w 6
> {code}
> I see this in
> /tmp/userlogs/application_1364578380737_0003/container_1364578380737_0003_01_02/
> :
> {code}
> 2013-03-29 10:58:06,371 DEBUG [org.apache.giraph.master.MasterThread]
> org.apache.giraph.master.BspServiceMaster: barrierOnWorkerList: Got finished
> worker list = [Eugenes-MacBook-Pro.local_1, Eugenes-MacBook-Pro.local_3],
> size = 2, worker list = [Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=2, port=30002), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=1, port=30001), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=4, port=30004), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=3, port=30003), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=5, port=30005), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=0, port=30010)], size = 6 from
> /_hadoopBsp/job_1364578380737_0003/_vertexInputSplitDoneDir
> 2013-03-29 10:58:06,373 WARN [netty-server-exec-3]
> org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
> Channel failed with remote address /172.16.175.1:56236
> java.lang.ClassCastException:
> org.apache.giraph.comm.requests.SendVertexRequest cannot be cast to
> org.apache.giraph.comm.requests.MasterRequest
> at
> org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:27)
> at
> org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:106)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:680)
> {code}
--
Thi
[jira] [Commented] (GIRAPH-601) Exception when running pagerank benchmark: SendVertexRequest cannot be cast to MasterRequest
[
https://issues.apache.org/jira/browse/GIRAPH-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617701#comment-13617701
]
Eugene Koontz commented on GIRAPH-601:
--
Hi Maja,
Thanks for the suggestion. It seems that the job still doesn't finish even with
removing that line. Also, I'm running with no giraph.SplitMasterWorker setting,
so it should default to true.
Still digging.. :)
-Eugene
> Exception when running pagerank benchmark: SendVertexRequest cannot be cast
> to MasterRequest
>
>
> Key: GIRAPH-601
> URL: https://issues.apache.org/jira/browse/GIRAPH-601
> Project: Giraph
> Issue Type: Bug
>Reporter: Eugene Koontz
>
> Building Giraph with:
> {code}
> mvn -DskipTests -Phadoop_2.0.3 clean compile
> {code}
> Running pagerank like this:
> {code}
> $HADOOP_RUNTIME/bin/hadoop jar $JAR \
> org.apache.giraph.benchmark.PageRankBenchmark \
> -e 10 -s 10 -v -V 10 -w 6
> {code}
> I see this in
> /tmp/userlogs/application_1364578380737_0003/container_1364578380737_0003_01_02/
> :
> {code}
> 2013-03-29 10:58:06,371 DEBUG [org.apache.giraph.master.MasterThread]
> org.apache.giraph.master.BspServiceMaster: barrierOnWorkerList: Got finished
> worker list = [Eugenes-MacBook-Pro.local_1, Eugenes-MacBook-Pro.local_3],
> size = 2, worker list = [Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=2, port=30002), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=1, port=30001), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=4, port=30004), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=3, port=30003), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=5, port=30005), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=0, port=30010)], size = 6 from
> /_hadoopBsp/job_1364578380737_0003/_vertexInputSplitDoneDir
> 2013-03-29 10:58:06,373 WARN [netty-server-exec-3]
> org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
> Channel failed with remote address /172.16.175.1:56236
> java.lang.ClassCastException:
> org.apache.giraph.comm.requests.SendVertexRequest cannot be cast to
> org.apache.giraph.comm.requests.MasterRequest
> at
> org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:27)
> at
> org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:106)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:680)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-601) Exception when running pagerank benchmark: SendVertexRequest cannot be cast to MasterRequest
[
https://issues.apache.org/jira/browse/GIRAPH-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617662#comment-13617662
]
Eugene Koontz commented on GIRAPH-601:
--
Note that in one container, the master and the worker are both running. The
Master binds to 3:
{code}
2013-03-29 12:35:45,085 INFO [org.apache.giraph.master.MasterThread]
org.apache.giraph.comm.netty.NettyServer: start: Started server communication
server: Eugenes-MacBook-Pro.local/172.16.175.1:3 with up to 16 threads on
bind attempt 0 with sendBufferSize = 32768 receiveBufferSize = 524288 backlog =
6
2013-03-29 12:35:45,092 INFO [org.apache.giraph.master.MasterThread]
org.apache.giraph.comm.netty.NettyClient: NettyClient: Using execution handler
with 8 threads after requestEncoder.
2013-03-29 12:35:45,097 INFO [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: becomeMaster: I am now the master!
{code}
The worker NettyServer tries to take the same port to listen on (3), and
fails because the Master's NettyServer already acquired it. So, the worker
tries again with a new port number (30010) and succeeds on listening on that:
{code}
2013-03-29 12:35:45,097 WARN [main] org.apache.giraph.comm.netty.NettyServer:
start: Likely failed to bind on attempt 0 to port 3
org.jboss.netty.channel.ChannelException: Failed to bind to:
Eugenes-MacBook-Pro.local/172.16.175.1:3
at
org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:298)
at org.apache.giraph.comm.netty.NettyServer.start(NettyServer.java:324)
at
org.apache.giraph.comm.netty.NettyWorkerServer.(NettyWorkerServer.java:101)
at
org.apache.giraph.worker.BspServiceWorker.(BspServiceWorker.java:184)
at
org.apache.giraph.graph.GraphTaskManager.instantiateBspService(GraphTaskManager.java:557)
at
org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:206)
at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:60)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:90)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:757)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.bind(NioServerSocketPipelineSink.java:138)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleServerSocket(NioServerSocketPipelineSink.java:90)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:64)
at org.jboss.netty.channel.Channels.bind(Channels.java:569)
at
org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:187)
at
org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen(ServerBootstrap.java:343)
at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:170)
at
org.jboss.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:80)
at
org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:158)
at
org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:86)
at
org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:277)
... 14 more
2013-03-29 12:35:45,101 INFO [main] org.apache.giraph.comm.netty.NettyServer:
start: Started server communication server:
Eugenes-MacBook-Pro.local/172.16.175.1:30010 with up to 16 threads on bind
attempt 1 with sendBufferSize = 32768 receiveBufferSize = 524288 backlog = 6
{code}
> Exception when running pagerank benchmark: SendVertexRequest cannot be cast
> to MasterRequest
>
>
> Key: GIRAPH-601
> URL: https://issues.apache.org/jira/browse/GIRAPH-601
> Project: Giraph
> Issue Type: Bug
>Reporter: Eugene Koontz
>
> Building Giraph with:
> {code}
> mvn -DskipTests -Phadoop_2.0.3 clean compile
> {code}
> Running pagerank like this:
> {code}
> $HADOOP_RUNTIME/bin/hadoop jar $JAR \
> org.apache
[jira] [Commented] (GIRAPH-601) Exception when running pagerank benchmark: SendVertexRequest cannot be cast to MasterRequest
[
https://issues.apache.org/jira/browse/GIRAPH-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617648#comment-13617648
]
Eugene Koontz commented on GIRAPH-601:
--
All containers *except one* are able to finish their client messages:
{code}
2013-03-29 12:35:49,934 INFO [main] org.apache.giraph.worker.BspServiceWorker:
loadInputSplits: Using 1 thread(s), originally 1 threads(s) for 6 total splits.
2013-03-29 12:35:49,938 INFO [main] org.apache.giraph.comm.SendPartitionCache:
SendPartitionCache: maxVerticesPerTransfer = 1
2013-03-29 12:35:49,938 INFO [main] org.apache.giraph.comm.SendPartitionCache:
SendPartitionCache: maxEdgesPerTransfer = 8
2013-03-29 12:35:49,949 INFO [load-0]
org.apache.giraph.worker.InputSplitsCallable: call: Loaded 0 input splits in
0.015249 secs, (v=0, e=0) 0.0 vertices/sec, 0.0 edges/sec
2013-03-29 12:35:49,951 INFO [main] org.apache.giraph.comm.netty.NettyClient:
waitAllRequests: Finished all requests. MBytes/sec sent = 0, MBytes/sec
received = 0, MBytesSent = 0, MBytesReceived = 0, ave sent req MBytes = 0, ave
received req MBytes = 0, secs waited = 2.807
2013-03-29 12:35:49,952 INFO [main] org.apache.giraph.worker.BspServiceWorker:
setup: Finally loaded a total of (v=0, e=0)
{code}
.. but this one container never finishes its open requests:
{code}
2013-03-29 12:37:34,988 INFO [main] org.apache.giraph.comm.netty.NettyClient:
logInfoAboutOpenRequests: Waiting for request (destTask=0,reqId=0) -
(reqId=0,destAddr=Eugenes-MacBook-Pro.local:3,elapsedNanos=105029398000,started=Fri
Mar 29 12:35:49 PDT 2013,writeDone=true,writeSuccess=true)
2013-03-29 12:37:34,988 INFO [main] org.apache.giraph.comm.netty.NettyClient:
logInfoAboutOpenRequests: 1 requests for taskId=0,
{code}
> Exception when running pagerank benchmark: SendVertexRequest cannot be cast
> to MasterRequest
>
>
> Key: GIRAPH-601
> URL: https://issues.apache.org/jira/browse/GIRAPH-601
> Project: Giraph
> Issue Type: Bug
>Reporter: Eugene Koontz
>
> Building Giraph with:
> {code}
> mvn -DskipTests -Phadoop_2.0.3 clean compile
> {code}
> Running pagerank like this:
> {code}
> $HADOOP_RUNTIME/bin/hadoop jar $JAR \
> org.apache.giraph.benchmark.PageRankBenchmark \
> -e 10 -s 10 -v -V 10 -w 6
> {code}
> I see this in
> /tmp/userlogs/application_1364578380737_0003/container_1364578380737_0003_01_02/
> :
> {code}
> 2013-03-29 10:58:06,371 DEBUG [org.apache.giraph.master.MasterThread]
> org.apache.giraph.master.BspServiceMaster: barrierOnWorkerList: Got finished
> worker list = [Eugenes-MacBook-Pro.local_1, Eugenes-MacBook-Pro.local_3],
> size = 2, worker list = [Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=2, port=30002), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=1, port=30001), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=4, port=30004), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=3, port=30003), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=5, port=30005), Worker(hostname=Eugenes-MacBook-Pro.local,
> MRtaskID=0, port=30010)], size = 6 from
> /_hadoopBsp/job_1364578380737_0003/_vertexInputSplitDoneDir
> 2013-03-29 10:58:06,373 WARN [netty-server-exec-3]
> org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
> Channel failed with remote address /172.16.175.1:56236
> java.lang.ClassCastException:
> org.apache.giraph.comm.requests.SendVertexRequest cannot be cast to
> org.apache.giraph.comm.requests.MasterRequest
> at
> org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:27)
> at
> org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:106)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:680)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-362) Address master task id for communication for master (known issue from GIRAPH-211)
[
https://issues.apache.org/jira/browse/GIRAPH-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617637#comment-13617637
]
Eugene Koontz commented on GIRAPH-362:
--
Sorry to take a long time to respond, by the way. I will be looking into this
in connection with GIRAPH-601.
> Address master task id for communication for master (known issue from
> GIRAPH-211)
> -
>
> Key: GIRAPH-362
> URL: https://issues.apache.org/jira/browse/GIRAPH-362
> Project: Giraph
> Issue Type: Improvement
>Reporter: Avery Ching
>
> There is a workaround from GIRAPH-211 to handle requests a little differently
> due to issues communicating to the master. We should fix this to be a
> regular request in the future.
> {code}
> public void sendWritableRequest(Integer destWorkerId,
> InetSocketAddress remoteServer,
> WritableRequest request) {
> if (clientRequestIdRequestInfoMap.isEmpty()) {
> byteCounter.resetAll();
> }
> boolean registerRequest = true;
> /*if[HADOOP_NON_SECURE]
> else[HADOOP_NON_SECURE]*/
> if (request.getType() == RequestType.SASL_TOKEN_MESSAGE_REQUEST) {
> registerRequest = false;
> }
> /*end[HADOOP_NON_SECURE]*/
> Channel channel = getNextChannel(remoteServer);
> RequestInfo newRequestInfo = new RequestInfo(remoteServer, request);
> if (registerRequest) {
> request.setClientId(clientId);
> request.setRequestId(
> addressRequestIdGenerator.getNextRequestId(remoteServer));
> ClientRequestId clientRequestId =
> new ClientRequestId(destWorkerId, request.getRequestId());
> RequestInfo oldRequestInfo = clientRequestIdRequestInfoMap.putIfAbsent(
> clientRequestId, newRequestInfo);
> if (oldRequestInfo != null) {
> throw new IllegalStateException("sendWritableRequest: Impossible to "
> +
> "have a previous request id = " + request.getRequestId() + ", " +
> "request info of " + oldRequestInfo);
> }
> }
> ChannelFuture writeFuture = channel.write(request);
> newRequestInfo.setWriteFuture(writeFuture);
> if (limitNumberOfOpenRequests &&
> clientRequestIdRequestInfoMap.size() > maxNumberOfOpenRequests) {
> waitSomeRequests(maxNumberOfOpenRequests);
> }
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-362) Address master task id for communication for master (known issue from GIRAPH-211)
[
https://issues.apache.org/jira/browse/GIRAPH-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617627#comment-13617627
]
Eugene Koontz commented on GIRAPH-362:
--
Hi Maja,
I think this might be related to GIRAPH-601. Something to do with the master's
task id.
-Eugene
> Address master task id for communication for master (known issue from
> GIRAPH-211)
> -
>
> Key: GIRAPH-362
> URL: https://issues.apache.org/jira/browse/GIRAPH-362
> Project: Giraph
> Issue Type: Improvement
>Reporter: Avery Ching
>
> There is a workaround from GIRAPH-211 to handle requests a little differently
> due to issues communicating to the master. We should fix this to be a
> regular request in the future.
> {code}
> public void sendWritableRequest(Integer destWorkerId,
> InetSocketAddress remoteServer,
> WritableRequest request) {
> if (clientRequestIdRequestInfoMap.isEmpty()) {
> byteCounter.resetAll();
> }
> boolean registerRequest = true;
> /*if[HADOOP_NON_SECURE]
> else[HADOOP_NON_SECURE]*/
> if (request.getType() == RequestType.SASL_TOKEN_MESSAGE_REQUEST) {
> registerRequest = false;
> }
> /*end[HADOOP_NON_SECURE]*/
> Channel channel = getNextChannel(remoteServer);
> RequestInfo newRequestInfo = new RequestInfo(remoteServer, request);
> if (registerRequest) {
> request.setClientId(clientId);
> request.setRequestId(
> addressRequestIdGenerator.getNextRequestId(remoteServer));
> ClientRequestId clientRequestId =
> new ClientRequestId(destWorkerId, request.getRequestId());
> RequestInfo oldRequestInfo = clientRequestIdRequestInfoMap.putIfAbsent(
> clientRequestId, newRequestInfo);
> if (oldRequestInfo != null) {
> throw new IllegalStateException("sendWritableRequest: Impossible to "
> +
> "have a previous request id = " + request.getRequestId() + ", " +
> "request info of " + oldRequestInfo);
> }
> }
> ChannelFuture writeFuture = channel.write(request);
> newRequestInfo.setWriteFuture(writeFuture);
> if (limitNumberOfOpenRequests &&
> clientRequestIdRequestInfoMap.size() > maxNumberOfOpenRequests) {
> waitSomeRequests(maxNumberOfOpenRequests);
> }
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-601) Exception when running pagerank benchmark: SendVertexRequest cannot be cast to MasterRequest
Eugene Koontz created GIRAPH-601:
Summary: Exception when running pagerank benchmark:
SendVertexRequest cannot be cast to MasterRequest
Key: GIRAPH-601
URL: https://issues.apache.org/jira/browse/GIRAPH-601
Project: Giraph
Issue Type: Bug
Reporter: Eugene Koontz
Building Giraph with:
{code}
mvn -DskipTests -Phadoop_2.0.3 clean compile
{code}
Running pagerank like this:
{code}
$HADOOP_RUNTIME/bin/hadoop jar $JAR \
org.apache.giraph.benchmark.PageRankBenchmark \
-e 10 -s 10 -v -V 10 -w 6
{code}
I see this in
/tmp/userlogs/application_1364578380737_0003/container_1364578380737_0003_01_02/
:
{code}
2013-03-29 10:58:06,371 DEBUG [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: barrierOnWorkerList: Got finished
worker list = [Eugenes-MacBook-Pro.local_1, Eugenes-MacBook-Pro.local_3], size
= 2, worker list = [Worker(hostname=Eugenes-MacBook-Pro.local, MRtaskID=2,
port=30002), Worker(hostname=Eugenes-MacBook-Pro.local, MRtaskID=1,
port=30001), Worker(hostname=Eugenes-MacBook-Pro.local, MRtaskID=4,
port=30004), Worker(hostname=Eugenes-MacBook-Pro.local, MRtaskID=3,
port=30003), Worker(hostname=Eugenes-MacBook-Pro.local, MRtaskID=5,
port=30005), Worker(hostname=Eugenes-MacBook-Pro.local, MRtaskID=0,
port=30010)], size = 6 from
/_hadoopBsp/job_1364578380737_0003/_vertexInputSplitDoneDir
2013-03-29 10:58:06,373 WARN [netty-server-exec-3]
org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
Channel failed with remote address /172.16.175.1:56236
java.lang.ClassCastException: org.apache.giraph.comm.requests.SendVertexRequest
cannot be cast to org.apache.giraph.comm.requests.MasterRequest
at
org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:27)
at
org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:106)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
at
org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
at
org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:680)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-590) Missing class org.apache.commons.net.util.Base64
[
https://issues.apache.org/jira/browse/GIRAPH-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13614093#comment-13614093
]
Eugene Koontz commented on GIRAPH-590:
--
Hi Gianmarco,
I see in the top-level pom.xml that the 0.23 profile has:
commons-net
commons-net
whereas 2.0-series Hadoops have:
commons-net
commons-net
provided
3.1
Perhaps changing the 0.23 profile to use the latter would fix this?
Or try adding to your yarn-site.xml:
yarn.application.classpath
(path to the commons-net jar)
-Eugene
> Missing class org.apache.commons.net.util.Base64
>
>
> Key: GIRAPH-590
> URL: https://issues.apache.org/jira/browse/GIRAPH-590
> Project: Giraph
> Issue Type: Bug
> Environment: Hadoop_0.23, -Dgiraph.useNetty=true,
> -Dgiraph.authenticate=true
>Reporter: Gianmarco De Francisci Morales
>
> I get the following exception when my mappers start:
> {code}
> 2013-03-26 10:31:18,176 FATAL [main] org.apache.hadoop.mapred.YarnChild:
> Error running child : java.lang.NoClassDefFoundError:
> org/apache/commons/net/util/Base64
> at
> org.apache.giraph.comm.netty.SaslNettyServer.encodeIdentifier(SaslNettyServer.java:124)
> at
> org.apache.giraph.comm.netty.SaslNettyClient$SaslClientCallbackHandler.(SaslNettyClient.java:167)
> at
> org.apache.giraph.comm.netty.SaslNettyClient.(SaslNettyClient.java:77)
> at
> org.apache.giraph.comm.netty.NettyClient.authenticateOnChannel(NettyClient.java:504)
> at
> org.apache.giraph.comm.netty.NettyClient.authenticate(NettyClient.java:479)
> at
> org.apache.giraph.comm.netty.NettyWorkerClient.authenticate(NettyWorkerClient.java:176)
> at
> org.apache.giraph.comm.netty.NettyWorkerClient.setup(NettyWorkerClient.java:167)
> at
> org.apache.giraph.worker.BspServiceWorker.setup(BspServiceWorker.java:504)
> at
> org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:225)
> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:726)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1221)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.commons.net.util.Base64
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> ... 17 more
> {code}
> Indeed, the class is not included in the giraph-core or giraph-examples fat
> jars, even though I see commons-net listed in the pom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (GIRAPH-583) Problem with authentication on Hadoop 0.23
[
https://issues.apache.org/jira/browse/GIRAPH-583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz resolved GIRAPH-583.
--
Resolution: Fixed
> Problem with authentication on Hadoop 0.23
> --
>
> Key: GIRAPH-583
> URL: https://issues.apache.org/jira/browse/GIRAPH-583
> Project: Giraph
> Issue Type: Bug
>Reporter: Gianmarco De Francisci Morales
>
> Hi,
> I am trying to run the RWR code on trunk and Hadoop 0.23 with Kerberos
> authentication, but I get this exception:
> {code}
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS)
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 13/03/23 17:32:36 WARN ipc.Client: Exception encountered while connecting to
> the server : javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS) cause:java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS) cause:java.io.IOException:
> Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]; Host Details : local host is:
> "gwta3005.tan.ygrid.yahoo.com/98.138.127.244"; destination host is:
> ""tiberiumtan-nn1.tan.ygrid.yahoo.com":8020;
> Exception in thread "main" java.io.IOException: Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)]; Host Details : local host is:
> "gwta3005.tan.ygrid.yahoo.com/98.138.127.244"; destination host is:
> ""tiberiumtan-nn1.tan.ygrid.yahoo.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:738)
> at org.apache.hadoop.ipc.Client.call(Client.java:1092)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195)
> at $Proxy6.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67)
> at $Proxy6.getDelegationToken(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:603)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:786)
> at
> org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:466)
> at
> org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:444)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:122)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:101)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:81)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
> at
> org.apache.giraph.io.formats.TextVertexOutputFormat.checkOutputSpecs(TextVertexOutputFormat.java:55)
> at
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:50)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:417)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:332)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1221)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> at o
[jira] [Commented] (GIRAPH-583) Problem with authentication on Hadoop 0.23
[
https://issues.apache.org/jira/browse/GIRAPH-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13612934#comment-13612934
]
Eugene Koontz commented on GIRAPH-583:
--
Hi Gianmarco,
Ok - going to close this. Please open a new JIRA and I'll try to investigate
the error you're seeing with Zookeeper.
-Eugene
> Problem with authentication on Hadoop 0.23
> --
>
> Key: GIRAPH-583
> URL: https://issues.apache.org/jira/browse/GIRAPH-583
> Project: Giraph
> Issue Type: Bug
>Reporter: Gianmarco De Francisci Morales
>
> Hi,
> I am trying to run the RWR code on trunk and Hadoop 0.23 with Kerberos
> authentication, but I get this exception:
> {code}
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS)
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 13/03/23 17:32:36 WARN ipc.Client: Exception encountered while connecting to
> the server : javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS) cause:java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS) cause:java.io.IOException:
> Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]; Host Details : local host is:
> "gwta3005.tan.ygrid.yahoo.com/98.138.127.244"; destination host is:
> ""tiberiumtan-nn1.tan.ygrid.yahoo.com":8020;
> Exception in thread "main" java.io.IOException: Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)]; Host Details : local host is:
> "gwta3005.tan.ygrid.yahoo.com/98.138.127.244"; destination host is:
> ""tiberiumtan-nn1.tan.ygrid.yahoo.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:738)
> at org.apache.hadoop.ipc.Client.call(Client.java:1092)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195)
> at $Proxy6.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67)
> at $Proxy6.getDelegationToken(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:603)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:786)
> at
> org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:466)
> at
> org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:444)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:122)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:101)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:81)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
> at
> org.apache.giraph.io.formats.TextVertexOutputFormat.checkOutputSpecs(TextVertexOutputFormat.java:55)
> at
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:50)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:417)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:332)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.j
[jira] [Commented] (GIRAPH-583) Problem with authentication on Hadoop 0.23
[
https://issues.apache.org/jira/browse/GIRAPH-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13612168#comment-13612168
]
Eugene Koontz commented on GIRAPH-583:
--
Interesting, thanks Gianmarco and Eli. I'm sure we'll get this ironed out. :)
> Problem with authentication on Hadoop 0.23
> --
>
> Key: GIRAPH-583
> URL: https://issues.apache.org/jira/browse/GIRAPH-583
> Project: Giraph
> Issue Type: Bug
>Reporter: Gianmarco De Francisci Morales
>
> Hi,
> I am trying to run the RWR code on trunk and Hadoop 0.23 with Kerberos
> authentication, but I get this exception:
> {code}
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS)
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 13/03/23 17:32:36 WARN ipc.Client: Exception encountered while connecting to
> the server : javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS) cause:java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS) cause:java.io.IOException:
> Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]; Host Details : local host is:
> "gwta3005.tan.ygrid.yahoo.com/98.138.127.244"; destination host is:
> ""tiberiumtan-nn1.tan.ygrid.yahoo.com":8020;
> Exception in thread "main" java.io.IOException: Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)]; Host Details : local host is:
> "gwta3005.tan.ygrid.yahoo.com/98.138.127.244"; destination host is:
> ""tiberiumtan-nn1.tan.ygrid.yahoo.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:738)
> at org.apache.hadoop.ipc.Client.call(Client.java:1092)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195)
> at $Proxy6.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67)
> at $Proxy6.getDelegationToken(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:603)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:786)
> at
> org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:466)
> at
> org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:444)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:122)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:101)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:81)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
> at
> org.apache.giraph.io.formats.TextVertexOutputFormat.checkOutputSpecs(TextVertexOutputFormat.java:55)
> at
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:50)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:417)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:332)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInf
[jira] [Commented] (GIRAPH-583) Problem with authentication on Hadoop 0.23
[
https://issues.apache.org/jira/browse/GIRAPH-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13612158#comment-13612158
]
Eugene Koontz commented on GIRAPH-583:
--
Gianmarco, are you able to run other non-Giraph jobs (e.g. Pi or Terasort)
successfully with authentication?
> Problem with authentication on Hadoop 0.23
> --
>
> Key: GIRAPH-583
> URL: https://issues.apache.org/jira/browse/GIRAPH-583
> Project: Giraph
> Issue Type: Bug
>Reporter: Gianmarco De Francisci Morales
>
> Hi,
> I am trying to run the RWR code on trunk and Hadoop 0.23 with Kerberos
> authentication, but I get this exception:
> {code}
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS)
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 13/03/23 17:32:36 WARN ipc.Client: Exception encountered while connecting to
> the server : javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS) cause:java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS) cause:java.io.IOException:
> Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]; Host Details : local host is:
> "gwta3005.tan.ygrid.yahoo.com/98.138.127.244"; destination host is:
> ""tiberiumtan-nn1.tan.ygrid.yahoo.com":8020;
> Exception in thread "main" java.io.IOException: Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)]; Host Details : local host is:
> "gwta3005.tan.ygrid.yahoo.com/98.138.127.244"; destination host is:
> ""tiberiumtan-nn1.tan.ygrid.yahoo.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:738)
> at org.apache.hadoop.ipc.Client.call(Client.java:1092)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195)
> at $Proxy6.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67)
> at $Proxy6.getDelegationToken(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:603)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:786)
> at
> org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:466)
> at
> org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:444)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:122)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:101)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:81)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
> at
> org.apache.giraph.io.formats.TextVertexOutputFormat.checkOutputSpecs(TextVertexOutputFormat.java:55)
> at
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:50)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:417)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:332)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apa
[jira] [Commented] (GIRAPH-583) Problem with authentication on Hadoop 0.23
[
https://issues.apache.org/jira/browse/GIRAPH-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13611977#comment-13611977
]
Eugene Koontz commented on GIRAPH-583:
--
Hi Gianmarco,
Good to see that you are trying out the security-enabled aspects of the code.
What is the RWR code?
It seems like the problem is happening at job-setup time. The job is failing to
authenticate with the Namenode, it looks like.
-Eugene
> Problem with authentication on Hadoop 0.23
> --
>
> Key: GIRAPH-583
> URL: https://issues.apache.org/jira/browse/GIRAPH-583
> Project: Giraph
> Issue Type: Bug
>Reporter: Gianmarco De Francisci Morales
>
> Hi,
> I am trying to run the RWR code on trunk and Hadoop 0.23 with Kerberos
> authentication, but I get this exception:
> {code}
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS)
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 13/03/23 17:32:36 WARN ipc.Client: Exception encountered while connecting to
> the server : javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS) cause:java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS) cause:java.io.IOException:
> Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]; Host Details : local host is:
> "gwta3005.tan.ygrid.yahoo.com/98.138.127.244"; destination host is:
> ""tiberiumtan-nn1.tan.ygrid.yahoo.com":8020;
> Exception in thread "main" java.io.IOException: Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)]; Host Details : local host is:
> "gwta3005.tan.ygrid.yahoo.com/98.138.127.244"; destination host is:
> ""tiberiumtan-nn1.tan.ygrid.yahoo.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:738)
> at org.apache.hadoop.ipc.Client.call(Client.java:1092)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195)
> at $Proxy6.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67)
> at $Proxy6.getDelegationToken(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:603)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:786)
> at
> org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:466)
> at
> org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:444)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:122)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:101)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:81)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
> at
> org.apache.giraph.io.formats.TextVertexOutputFormat.checkOutputSpecs(TextVertexOutputFormat.java:55)
> at
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:50)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:417)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:332)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> a
[jira] [Updated] (GIRAPH-583) Problem with authentication on Hadoop 0.23
[
https://issues.apache.org/jira/browse/GIRAPH-583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-583:
-
Summary: Problem with authentication on Hadoop 0.23 (was: Problem with
authehtication on Hadoop 0.23)
> Problem with authentication on Hadoop 0.23
> --
>
> Key: GIRAPH-583
> URL: https://issues.apache.org/jira/browse/GIRAPH-583
> Project: Giraph
> Issue Type: Bug
>Reporter: Gianmarco De Francisci Morales
>
> Hi,
> I am trying to run the RWR code on trunk and Hadoop 0.23 with Kerberos
> authentication, but I get this exception:
> {code}
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS)
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 13/03/23 17:32:36 WARN ipc.Client: Exception encountered while connecting to
> the server : javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS) cause:java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 13/03/23 17:32:36 ERROR security.UserGroupInformation:
> PriviledgedActionException as:gdfm (auth:KERBEROS) cause:java.io.IOException:
> Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]; Host Details : local host is:
> "gwta3005.tan.ygrid.yahoo.com/98.138.127.244"; destination host is:
> ""tiberiumtan-nn1.tan.ygrid.yahoo.com":8020;
> Exception in thread "main" java.io.IOException: Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)]; Host Details : local host is:
> "gwta3005.tan.ygrid.yahoo.com/98.138.127.244"; destination host is:
> ""tiberiumtan-nn1.tan.ygrid.yahoo.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:738)
> at org.apache.hadoop.ipc.Client.call(Client.java:1092)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195)
> at $Proxy6.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67)
> at $Proxy6.getDelegationToken(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:603)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:786)
> at
> org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:466)
> at
> org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:444)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:122)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:101)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:81)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
> at
> org.apache.giraph.io.formats.TextVertexOutputFormat.checkOutputSpecs(TextVertexOutputFormat.java:55)
> at
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:50)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:417)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:332)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java
[jira] [Commented] (GIRAPH-13) Port Giraph to YARN
[
https://issues.apache.org/jira/browse/GIRAPH-13?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602572#comment-13602572
]
Eugene Koontz commented on GIRAPH-13:
-
Hi Eli,
Sorry to come in so late. Very impressive! I'm going to try your patch with
Hadoop 2.0.3-alpha today and I appreciate your offer to help with
configuration. {{mvn -Phadoop_yarn clean install}} succeeds with for me with
your latest patch so I'm off to a good start. :)
-Eugene
> Port Giraph to YARN
> ---
>
> Key: GIRAPH-13
> URL: https://issues.apache.org/jira/browse/GIRAPH-13
> Project: Giraph
> Issue Type: New Feature
>Reporter: Jakob Homan
>Assignee: Eli Reisman
> Attachments: GIRAPH-13-1.patch, GIRAPH-13-2.patch, GIRAPH-13-3.patch,
> GIRAPH-13-4.patch, GIRAPH-13-5.patch, GIRAPH-13-6.patch, GIRAPH-13-7.patch,
> GIRAPH-13-8.patch, GIRAPH-13-9.patch, GIRAPH-13-9-r1.patch,
> GIRAPH-13-9-r2.patch, GIRAPH-13-9-r3.patch
>
>
> Now that YARN (aka MR2 aka MAPREDUCE-279) has been merged into the Hadoop
> trunk, we should think about what it would take to separate out the graph
> processing bits of Giraph from the MR1-specific code so as to take advantage
> of the less-MR centric aspects of YARN, while still supporting both over the
> medium term.
> Review Board link (ready for review now): https://reviews.apache.org/r/9811/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-559) Giraph build breaks on trunk over Hive IO jar location issue?
[
https://issues.apache.org/jira/browse/GIRAPH-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602528#comment-13602528
]
Eugene Koontz commented on GIRAPH-559:
--
Nitay, Thanks, that fixes it - (your latest patch that sets snapshots/enabled
to false for the sonatype repo). Now {{mvn clean install}} passes for the
default profile.
> Giraph build breaks on trunk over Hive IO jar location issue?
> -
>
> Key: GIRAPH-559
> URL: https://issues.apache.org/jira/browse/GIRAPH-559
> Project: Giraph
> Issue Type: Bug
>Reporter: Eli Reisman
>Assignee: Nitay Joffe
> Attachments: GIRAPH-559-2.patch, GIRAPH-559.patch, GIRAPH-559.patch
>
>
> Hi Folks. When I build trunk after pulling the repo today I get this action:
> {code}
> [INFO] >>> findbugs-maven-plugin:2.5.1:check (default) @ giraph-core >>>
> [INFO]
> [INFO] --- findbugs-maven-plugin:2.5.1:findbugs (findbugs) @ giraph-core ---
> [INFO] Fork Value is true
> [INFO] Done FindBugs Analysis
> [INFO]
> [INFO] <<< findbugs-maven-plugin:2.5.1:check (default) @ giraph-core <<<
> [INFO]
> [INFO] --- findbugs-maven-plugin:2.5.1:check (default) @ giraph-core ---
> [INFO] BugInstance size is 0
> [INFO] Error size is 0
> [INFO] No errors/warnings found
> [INFO]
>
> [INFO]
>
> [INFO] Building Apache Giraph Hive I/O 0.2-SNAPSHOT
> [INFO]
>
> [WARNING] The POM for
> com.facebook.giraph.hive:hive-io-experimental:jar:0.2-SNAPSHOT is missing, no
> dependency information available
> Downloading:
> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/avro/avro/1.7.1/avro-1.7.1.jar
> Downloading:
> https://repository.apache.org/content/repositories/releases/org/apache/avro/avro/1.7.1/avro-1.7.1.jar
> Downloaded:
> https://repository.apache.org/content/repositories/releases/org/apache/avro/avro/1.7.1/avro-1.7.1.jar
> (291 KB at 1149.7 KB/sec)
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Giraph Parent .. SUCCESS [0.432s]
> [INFO] Apache Giraph Core SUCCESS [1:37.413s]
> [INFO] Apache Giraph Hive I/O FAILURE [1.082s]
> [INFO] Apache Giraph Examples SKIPPED
> [INFO] Apache Giraph Accumulo I/O SKIPPED
> [INFO] Apache Giraph HBase I/O ... SKIPPED
> [INFO] Apache Giraph HCatalog I/O SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 1:39.263s
> [INFO] Finished at: Mon Mar 11 16:38:32 PDT 2013
> [INFO] Final Memory: 35M/110M
> [INFO]
>
> [ERROR] Failed to execute goal on project giraph-hive: Could not resolve
> dependencies for project org.apache.giraph:giraph-hive:jar:0.2-SNAPSHOT:
> Failure to find
> com.facebook.giraph.hive:hive-io-experimental:jar:0.2-SNAPSHOT in
> https://repository.apache.org/content/repositories/snapshots was cached in
> the local repository, resolution will not be reattempted until the update
> interval of apache snapshots has elapsed or updates are forced -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :giraph-hive
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (GIRAPH-559) Giraph build breaks on trunk over Hive IO jar location issue?
[
https://issues.apache.org/jira/browse/GIRAPH-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz reopened GIRAPH-559:
--
Hi Nitay,
Thanks for looking into this, but the jar seems to have disappeared again - I'm
getting the same breakage as Eli reported when opening this JIRA, and I don't
see the jar in
https://oss.sonatype.org/content/repositories/snapshots/com/facebook/giraph/hive/hive-io-experimental/
.
> Giraph build breaks on trunk over Hive IO jar location issue?
> -
>
> Key: GIRAPH-559
> URL: https://issues.apache.org/jira/browse/GIRAPH-559
> Project: Giraph
> Issue Type: Bug
>Reporter: Eli Reisman
>Assignee: Nitay Joffe
> Attachments: GIRAPH-559-2.patch, GIRAPH-559.patch
>
>
> Hi Folks. When I build trunk after pulling the repo today I get this action:
> {code}
> [INFO] >>> findbugs-maven-plugin:2.5.1:check (default) @ giraph-core >>>
> [INFO]
> [INFO] --- findbugs-maven-plugin:2.5.1:findbugs (findbugs) @ giraph-core ---
> [INFO] Fork Value is true
> [INFO] Done FindBugs Analysis
> [INFO]
> [INFO] <<< findbugs-maven-plugin:2.5.1:check (default) @ giraph-core <<<
> [INFO]
> [INFO] --- findbugs-maven-plugin:2.5.1:check (default) @ giraph-core ---
> [INFO] BugInstance size is 0
> [INFO] Error size is 0
> [INFO] No errors/warnings found
> [INFO]
>
> [INFO]
>
> [INFO] Building Apache Giraph Hive I/O 0.2-SNAPSHOT
> [INFO]
>
> [WARNING] The POM for
> com.facebook.giraph.hive:hive-io-experimental:jar:0.2-SNAPSHOT is missing, no
> dependency information available
> Downloading:
> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/avro/avro/1.7.1/avro-1.7.1.jar
> Downloading:
> https://repository.apache.org/content/repositories/releases/org/apache/avro/avro/1.7.1/avro-1.7.1.jar
> Downloaded:
> https://repository.apache.org/content/repositories/releases/org/apache/avro/avro/1.7.1/avro-1.7.1.jar
> (291 KB at 1149.7 KB/sec)
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Giraph Parent .. SUCCESS [0.432s]
> [INFO] Apache Giraph Core SUCCESS [1:37.413s]
> [INFO] Apache Giraph Hive I/O FAILURE [1.082s]
> [INFO] Apache Giraph Examples SKIPPED
> [INFO] Apache Giraph Accumulo I/O SKIPPED
> [INFO] Apache Giraph HBase I/O ... SKIPPED
> [INFO] Apache Giraph HCatalog I/O SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 1:39.263s
> [INFO] Finished at: Mon Mar 11 16:38:32 PDT 2013
> [INFO] Final Memory: 35M/110M
> [INFO]
>
> [ERROR] Failed to execute goal on project giraph-hive: Could not resolve
> dependencies for project org.apache.giraph:giraph-hive:jar:0.2-SNAPSHOT:
> Failure to find
> com.facebook.giraph.hive:hive-io-experimental:jar:0.2-SNAPSHOT in
> https://repository.apache.org/content/repositories/snapshots was cached in
> the local repository, resolution will not be reattempted until the update
> interval of apache snapshots has elapsed or updates are forced -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :giraph-hive
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-518) Set up Maven to support Hadoop-2.0.3-alpha release
[
https://issues.apache.org/jira/browse/GIRAPH-518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580051#comment-13580051
]
Eugene Koontz commented on GIRAPH-518:
--
+1. Patch looks good. Passes {{mvn -Phadoop_2.0.3 clean install}} for me.
> Set up Maven to support Hadoop-2.0.3-alpha release
> --
>
> Key: GIRAPH-518
> URL: https://issues.apache.org/jira/browse/GIRAPH-518
> Project: Giraph
> Issue Type: Bug
>Affects Versions: 0.2.0
>Reporter: Eli Reisman
> Attachments: GIRAPH-518-1.patch
>
>
> This will support the new profile:
> {code}mvn -Phadoop_2.0.3 clean package{code}
> for the new Hadoop-2.0.3-alpha release. We have the technology.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (GIRAPH-158) Support YARN (next generation MapReduce)
[ https://issues.apache.org/jira/browse/GIRAPH-158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz resolved GIRAPH-158. -- Resolution: Duplicate Closing as a duplicate of GIRAPH-13 so that we concentrate on Eli's work over there! > Support YARN (next generation MapReduce) > > > Key: GIRAPH-158 > URL: https://issues.apache.org/jira/browse/GIRAPH-158 > Project: Giraph > Issue Type: New Feature >Reporter: Eugene Koontz > Labels: features, hadoop > Attachments: GIRAPH-158.patch > > > YARN is a re-architecturing of the Hadoop MapReduce framework, described here: > http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/YARN.html > It would be good to offer support within Giraph for this framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-502) In PageRankBenchmark, remove unneeded handling of -t 2
[
https://issues.apache.org/jira/browse/GIRAPH-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-502:
-
Description:
PagerankBenchmark accepts, among other options, -t. For the usage message, it's
treated as:
t=0 (no combiner)
t=1 (DoubleSumCombiner (default))
However the code mentions a t=2, but this ends up being treated the same as t=1:
{code}
if (!cmd.hasOption('t') ||
(Integer.parseInt(cmd.getOptionValue('t')) == 2)) {
configuration.setVertexCombinerClass(
DoubleSumCombiner.class);
} else if (Integer.parseInt(cmd.getOptionValue('t')) == 1) {
configuration.setVertexCombinerClass(
DoubleSumCombiner.class);
}
{code}
We should make the code usage conform to the usage message and remove the
extraneous -t=2 handling.
was:
PagerankBenchmark accepts, among other options, -t. For the usage message, it's
treated as:
t=0 (no combiner)
t=1 (DoubleSumCombiner (default))
However the code mentions a t=2, but this ends up being treated the same as t=0:
{code}
if (!cmd.hasOption('t') ||
(Integer.parseInt(cmd.getOptionValue('t')) == 2)) {
configuration.setVertexCombinerClass(
DoubleSumCombiner.class);
} else if (Integer.parseInt(cmd.getOptionValue('t')) == 1) {
configuration.setVertexCombinerClass(
DoubleSumCombiner.class);
}
{code}
We should make the code usage conform to the usage message and remove the
extraneous -t=2 handling.
> In PageRankBenchmark, remove unneeded handling of -t 2
> --
>
> Key: GIRAPH-502
> URL: https://issues.apache.org/jira/browse/GIRAPH-502
> Project: Giraph
> Issue Type: Improvement
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
>Priority: Minor
> Attachments: GIRAPH-502.patch
>
>
> PagerankBenchmark accepts, among other options, -t. For the usage message,
> it's treated as:
> t=0 (no combiner)
> t=1 (DoubleSumCombiner (default))
> However the code mentions a t=2, but this ends up being treated the same as
> t=1:
> {code}
> if (!cmd.hasOption('t') ||
> (Integer.parseInt(cmd.getOptionValue('t')) == 2)) {
> configuration.setVertexCombinerClass(
> DoubleSumCombiner.class);
> } else if (Integer.parseInt(cmd.getOptionValue('t')) == 1) {
> configuration.setVertexCombinerClass(
> DoubleSumCombiner.class);
> }
> {code}
> We should make the code usage conform to the usage message and remove the
> extraneous -t=2 handling.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-502) In PageRankBenchmark, remove unneeded handling of -t 2
[
https://issues.apache.org/jira/browse/GIRAPH-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-502:
-
Attachment: GIRAPH-502.patch
> In PageRankBenchmark, remove unneeded handling of -t 2
> --
>
> Key: GIRAPH-502
> URL: https://issues.apache.org/jira/browse/GIRAPH-502
> Project: Giraph
> Issue Type: Improvement
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
>Priority: Minor
> Attachments: GIRAPH-502.patch
>
>
> PagerankBenchmark accepts, among other options, -t. For the usage message,
> it's treated as:
> t=0 (no combiner)
> t=1 (DoubleSumCombiner (default))
> However the code mentions a t=2, but this ends up being treated the same as
> t=0:
> {code}
> if (!cmd.hasOption('t') ||
> (Integer.parseInt(cmd.getOptionValue('t')) == 2)) {
> configuration.setVertexCombinerClass(
> DoubleSumCombiner.class);
> } else if (Integer.parseInt(cmd.getOptionValue('t')) == 1) {
> configuration.setVertexCombinerClass(
> DoubleSumCombiner.class);
> }
> {code}
> We should make the code usage conform to the usage message and remove the
> extraneous -t=2 handling.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-502) In PageRankBenchmark, remove unneeded handling of -t 2
[
https://issues.apache.org/jira/browse/GIRAPH-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-502:
-
Summary: In PageRankBenchmark, remove unneeded handling of -t 2 (was: In
PageRankBenchmark, remove unused handling of -t 2)
> In PageRankBenchmark, remove unneeded handling of -t 2
> --
>
> Key: GIRAPH-502
> URL: https://issues.apache.org/jira/browse/GIRAPH-502
> Project: Giraph
> Issue Type: Improvement
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
>Priority: Minor
>
> PagerankBenchmark accepts, among other options, -t. For the usage message,
> it's treated as:
> t=0 (no combiner)
> t=1 (DoubleSumCombiner (default))
> However the code mentions a t=2, but this ends up being treated the same as
> t=0:
> {code}
> if (!cmd.hasOption('t') ||
> (Integer.parseInt(cmd.getOptionValue('t')) == 2)) {
> configuration.setVertexCombinerClass(
> DoubleSumCombiner.class);
> } else if (Integer.parseInt(cmd.getOptionValue('t')) == 1) {
> configuration.setVertexCombinerClass(
> DoubleSumCombiner.class);
> }
> {code}
> We should make the code usage conform to the usage message and remove the
> extraneous -t=2 handling.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-502) In PageRankBenchmark, remove unused handling of -t 2
Eugene Koontz created GIRAPH-502:
Summary: In PageRankBenchmark, remove unused handling of -t 2
Key: GIRAPH-502
URL: https://issues.apache.org/jira/browse/GIRAPH-502
Project: Giraph
Issue Type: Improvement
Reporter: Eugene Koontz
Assignee: Eugene Koontz
Priority: Minor
PagerankBenchmark accepts, among other options, -t. For the usage message, it's
treated as:
t=0 (no combiner)
t=1 (DoubleSumCombiner (default))
However the code mentions a t=2, but this ends up being treated the same as t=0:
{code}
if (!cmd.hasOption('t') ||
(Integer.parseInt(cmd.getOptionValue('t')) == 2)) {
configuration.setVertexCombinerClass(
DoubleSumCombiner.class);
} else if (Integer.parseInt(cmd.getOptionValue('t')) == 1) {
configuration.setVertexCombinerClass(
DoubleSumCombiner.class);
}
{code}
We should make the code usage conform to the usage message and remove the
extraneous -t=2 handling.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-418) Create maven profile for CDH 4.1.2
[
https://issues.apache.org/jira/browse/GIRAPH-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13561204#comment-13561204
]
Eugene Koontz commented on GIRAPH-418:
--
Hi Nitay,
Ok, thanks for that. It seems that {{install}} depends on {{verify}} somehow? I
guess it must be an implicit dependency of maven's build process; I don't see
{{install}} in either the top-level {{pom.xml}} or {{giraph-core/pom.xml}}.
-Eugene
> Create maven profile for CDH 4.1.2
> --
>
> Key: GIRAPH-418
> URL: https://issues.apache.org/jira/browse/GIRAPH-418
> Project: Giraph
> Issue Type: Improvement
>Reporter: Eugene Koontz
> Attachments: GIRAPH-418.patch, GIRAPH-418.patch, GIRAPH-418.patch,
> pom.xml
>
>
> Mirko writes:
> bq. Hello all,
> bq. I created a now pom.xml file, which allows compiling Giraph with the new
> CDH 4.1.2 repository.
> bq. I attached it to this email.
> bq. But there are still some problems. I can not start the examples, because
> I do not have demo data and I do not jet know, what formats have to be used.
> Has anybody some test data files to share for a public tutorial?
> bq. Currently I prepare a tutorial on working with Giraph, but until now I
> had to find out, that the wiki content is quite outdated. The
> ShortestPathExample looks nice, but without some hints I can not start. Did I
> miss some documentation? If not, I will create it, but therefor I would like
> to ask for some help.
> bq. My plan is, to create some demo data files and some Demo-Apps comparable
> to the benchmark classes but based on real data. I have some correlation
> obtained from wikipedia user data, but until now I can not analuse that data
> with Giraph.
> bq. I get the following error ( at the end of this mail ) if I want to start
> the PageRankBenchmark.
> bq. Has anybody an idea, what I can do to solve this problem?
> bq. Thanks a lot!
> bq. Best wishes
> bq. Mirko
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-418) Create maven profile for CDH 4.1.2
[
https://issues.apache.org/jira/browse/GIRAPH-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560904#comment-13560904
]
Eugene Koontz commented on GIRAPH-418:
--
Hm, I thought I did run {{mvn -Phadoop_cdh4.1.2 clean verify}} at the top level
in the gist given in my last comment; am I misunderstanding something? Also I
see that subsequent builds after build #618 are passing; e.g. #619:
https://builds.apache.org/job/Giraph-trunk-Commit/619/
-Eugene
> Create maven profile for CDH 4.1.2
> --
>
> Key: GIRAPH-418
> URL: https://issues.apache.org/jira/browse/GIRAPH-418
> Project: Giraph
> Issue Type: Improvement
>Reporter: Eugene Koontz
> Attachments: GIRAPH-418.patch, GIRAPH-418.patch, GIRAPH-418.patch,
> pom.xml
>
>
> Mirko writes:
> bq. Hello all,
> bq. I created a now pom.xml file, which allows compiling Giraph with the new
> CDH 4.1.2 repository.
> bq. I attached it to this email.
> bq. But there are still some problems. I can not start the examples, because
> I do not have demo data and I do not jet know, what formats have to be used.
> Has anybody some test data files to share for a public tutorial?
> bq. Currently I prepare a tutorial on working with Giraph, but until now I
> had to find out, that the wiki content is quite outdated. The
> ShortestPathExample looks nice, but without some hints I can not start. Did I
> miss some documentation? If not, I will create it, but therefor I would like
> to ask for some help.
> bq. My plan is, to create some demo data files and some Demo-Apps comparable
> to the benchmark classes but based on real data. I have some correlation
> obtained from wikipedia user data, but until now I can not analuse that data
> with Giraph.
> bq. I get the following error ( at the end of this mail ) if I want to start
> the PageRankBenchmark.
> bq. Has anybody an idea, what I can do to solve this problem?
> bq. Thanks a lot!
> bq. Best wishes
> bq. Mirko
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-418) Create maven profile for CDH 4.1.2
[ https://issues.apache.org/jira/browse/GIRAPH-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560875#comment-13560875 ] Eugene Koontz commented on GIRAPH-418: -- Hi Claudio, That's strange. I ran "mvn -Phadoop_cdh4.1.2 clean verify" before committing and it passed: https://gist.github.com/4611075 -Eugene > Create maven profile for CDH 4.1.2 > -- > > Key: GIRAPH-418 > URL: https://issues.apache.org/jira/browse/GIRAPH-418 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz > Attachments: GIRAPH-418.patch, GIRAPH-418.patch, GIRAPH-418.patch, > pom.xml > > > Mirko writes: > bq. Hello all, > bq. I created a now pom.xml file, which allows compiling Giraph with the new > CDH 4.1.2 repository. > bq. I attached it to this email. > bq. But there are still some problems. I can not start the examples, because > I do not have demo data and I do not jet know, what formats have to be used. > Has anybody some test data files to share for a public tutorial? > bq. Currently I prepare a tutorial on working with Giraph, but until now I > had to find out, that the wiki content is quite outdated. The > ShortestPathExample looks nice, but without some hints I can not start. Did I > miss some documentation? If not, I will create it, but therefor I would like > to ask for some help. > bq. My plan is, to create some demo data files and some Demo-Apps comparable > to the benchmark classes but based on real data. I have some correlation > obtained from wikipedia user data, but until now I can not analuse that data > with Giraph. > bq. I get the following error ( at the end of this mail ) if I want to start > the PageRankBenchmark. > bq. Has anybody an idea, what I can do to solve this problem? > bq. Thanks a lot! > bq. Best wishes > bq. Mirko -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-418) Create maven profile for CDH 4.1.2
[
https://issues.apache.org/jira/browse/GIRAPH-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560817#comment-13560817
]
Eugene Koontz commented on GIRAPH-418:
--
Committed; thanks for the review, Nitay. Removed the unnecessary
{{activeByDefault=false}}.
> Create maven profile for CDH 4.1.2
> --
>
> Key: GIRAPH-418
> URL: https://issues.apache.org/jira/browse/GIRAPH-418
> Project: Giraph
> Issue Type: Improvement
>Reporter: Eugene Koontz
> Attachments: GIRAPH-418.patch, GIRAPH-418.patch, GIRAPH-418.patch,
> pom.xml
>
>
> Mirko writes:
> bq. Hello all,
> bq. I created a now pom.xml file, which allows compiling Giraph with the new
> CDH 4.1.2 repository.
> bq. I attached it to this email.
> bq. But there are still some problems. I can not start the examples, because
> I do not have demo data and I do not jet know, what formats have to be used.
> Has anybody some test data files to share for a public tutorial?
> bq. Currently I prepare a tutorial on working with Giraph, but until now I
> had to find out, that the wiki content is quite outdated. The
> ShortestPathExample looks nice, but without some hints I can not start. Did I
> miss some documentation? If not, I will create it, but therefor I would like
> to ask for some help.
> bq. My plan is, to create some demo data files and some Demo-Apps comparable
> to the benchmark classes but based on real data. I have some correlation
> obtained from wikipedia user data, but until now I can not analuse that data
> with Giraph.
> bq. I get the following error ( at the end of this mail ) if I want to start
> the PageRankBenchmark.
> bq. Has anybody an idea, what I can do to solve this problem?
> bq. Thanks a lot!
> bq. Best wishes
> bq. Mirko
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-418) Create maven profile for CDH 4.1.2
[ https://issues.apache.org/jira/browse/GIRAPH-418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-418: - Summary: Create maven profile for CDH 4.1.2 (was: Create maven profile for CDH 4.1.1) > Create maven profile for CDH 4.1.2 > -- > > Key: GIRAPH-418 > URL: https://issues.apache.org/jira/browse/GIRAPH-418 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz > Attachments: GIRAPH-418.patch, GIRAPH-418.patch, GIRAPH-418.patch, > pom.xml > > > Mirko writes: > bq. Hello all, > bq. I created a now pom.xml file, which allows compiling Giraph with the new > CDH 4.1.2 repository. > bq. I attached it to this email. > bq. But there are still some problems. I can not start the examples, because > I do not have demo data and I do not jet know, what formats have to be used. > Has anybody some test data files to share for a public tutorial? > bq. Currently I prepare a tutorial on working with Giraph, but until now I > had to find out, that the wiki content is quite outdated. The > ShortestPathExample looks nice, but without some hints I can not start. Did I > miss some documentation? If not, I will create it, but therefor I would like > to ask for some help. > bq. My plan is, to create some demo data files and some Demo-Apps comparable > to the benchmark classes but based on real data. I have some correlation > obtained from wikipedia user data, but until now I can not analuse that data > with Giraph. > bq. I get the following error ( at the end of this mail ) if I want to start > the PageRankBenchmark. > bq. Has anybody an idea, what I can do to solve this problem? > bq. Thanks a lot! > bq. Best wishes > bq. Mirko -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-485) Hadoop Shims, smart detect version on job start
[ https://issues.apache.org/jira/browse/GIRAPH-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558128#comment-13558128 ] Eugene Koontz commented on GIRAPH-485: -- Sounds interesting; let's look into it. > Hadoop Shims, smart detect version on job start > --- > > Key: GIRAPH-485 > URL: https://issues.apache.org/jira/browse/GIRAPH-485 > Project: Giraph > Issue Type: Wish >Reporter: Nitay Joffe > > We should be able to (in theory at least, haven't looked deeply into it yet) > use hive style shims to get rid of our munge whatnots and compile against all > of the hadoops at once. This would mean having a common interface and a > module for each hadoop we support. Then, when the user launches a giraph job > we can detect the hadoop we're running in and load the appropriate module. > This will require some exploration and research to figure out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-418) Create maven profile for CDH 4.1.1
[
https://issues.apache.org/jira/browse/GIRAPH-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558126#comment-13558126
]
Eugene Koontz commented on GIRAPH-418:
--
Thanks for the reminder, Nitay.
Attached new patch against trunk@bfc6473685.
With this patch, {{{mvn -Phadoop_cdh4.1.2 clean test}}} fails due to:
{{{
Running org.apache.giraph.examples.TryMultiIpcBindingPortsTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.242 sec <<<
FAILURE!
}}}
> Create maven profile for CDH 4.1.1
> --
>
> Key: GIRAPH-418
> URL: https://issues.apache.org/jira/browse/GIRAPH-418
> Project: Giraph
> Issue Type: Improvement
>Reporter: Eugene Koontz
> Attachments: GIRAPH-418.patch, GIRAPH-418.patch, GIRAPH-418.patch,
> pom.xml
>
>
> Mirko writes:
> bq. Hello all,
> bq. I created a now pom.xml file, which allows compiling Giraph with the new
> CDH 4.1.2 repository.
> bq. I attached it to this email.
> bq. But there are still some problems. I can not start the examples, because
> I do not have demo data and I do not jet know, what formats have to be used.
> Has anybody some test data files to share for a public tutorial?
> bq. Currently I prepare a tutorial on working with Giraph, but until now I
> had to find out, that the wiki content is quite outdated. The
> ShortestPathExample looks nice, but without some hints I can not start. Did I
> miss some documentation? If not, I will create it, but therefor I would like
> to ask for some help.
> bq. My plan is, to create some demo data files and some Demo-Apps comparable
> to the benchmark classes but based on real data. I have some correlation
> obtained from wikipedia user data, but until now I can not analuse that data
> with Giraph.
> bq. I get the following error ( at the end of this mail ) if I want to start
> the PageRankBenchmark.
> bq. Has anybody an idea, what I can do to solve this problem?
> bq. Thanks a lot!
> bq. Best wishes
> bq. Mirko
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-418) Create maven profile for CDH 4.1.1
[ https://issues.apache.org/jira/browse/GIRAPH-418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-418: - Attachment: GIRAPH-418.patch > Create maven profile for CDH 4.1.1 > -- > > Key: GIRAPH-418 > URL: https://issues.apache.org/jira/browse/GIRAPH-418 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz > Attachments: GIRAPH-418.patch, GIRAPH-418.patch, GIRAPH-418.patch, > pom.xml > > > Mirko writes: > bq. Hello all, > bq. I created a now pom.xml file, which allows compiling Giraph with the new > CDH 4.1.2 repository. > bq. I attached it to this email. > bq. But there are still some problems. I can not start the examples, because > I do not have demo data and I do not jet know, what formats have to be used. > Has anybody some test data files to share for a public tutorial? > bq. Currently I prepare a tutorial on working with Giraph, but until now I > had to find out, that the wiki content is quite outdated. The > ShortestPathExample looks nice, but without some hints I can not start. Did I > miss some documentation? If not, I will create it, but therefor I would like > to ask for some help. > bq. My plan is, to create some demo data files and some Demo-Apps comparable > to the benchmark classes but based on real data. I have some correlation > obtained from wikipedia user data, but until now I can not analuse that data > with Giraph. > bq. I get the following error ( at the end of this mail ) if I want to start > the PageRankBenchmark. > bq. Has anybody an idea, what I can do to solve this problem? > bq. Thanks a lot! > bq. Best wishes > bq. Mirko -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-454) Fix typo in munge comments in TestBspBasic
[
https://issues.apache.org/jira/browse/GIRAPH-454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13539064#comment-13539064
]
Eugene Koontz commented on GIRAPH-454:
--
I followed the directions in the link that Nitay posted and set up a ~/.netrc
per those directions. Push seems to have worked - just pushed the commit to fix
this JIRA to https://git-wip-us.apache.org/repos/asf/giraph.git. Thanks Nitay
and thanks Eli for rebasing the patch!
> Fix typo in munge comments in TestBspBasic
> --
>
> Key: GIRAPH-454
> URL: https://issues.apache.org/jira/browse/GIRAPH-454
> Project: Giraph
> Issue Type: Bug
> Components: test
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Attachments: GIRAPH-454-2.patch, GIRAPH-454.patch
>
>
> Simply missing some comment delimiters in the imports part of the test.
> Without this change, {{TestBspBasic}} can't compile, but with this change,
> Hadoop 2.0.2 profile build:
> {{mvn -Phadoop_2.0.2 clean verify}}
> succeeds.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-454) Fix typo in munge comments in TestBspBasic
[
https://issues.apache.org/jira/browse/GIRAPH-454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-454:
-
Summary: Fix typo in munge comments in TestBspBasic (was: typo in munge
comments in TestBspBasic)
> Fix typo in munge comments in TestBspBasic
> --
>
> Key: GIRAPH-454
> URL: https://issues.apache.org/jira/browse/GIRAPH-454
> Project: Giraph
> Issue Type: Bug
> Components: test
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Attachments: GIRAPH-454-2.patch, GIRAPH-454.patch
>
>
> Simply missing some comment delimiters in the imports part of the test.
> Without this change, {{TestBspBasic}} can't compile, but with this change,
> Hadoop 2.0.2 profile build:
> {{mvn -Phadoop_2.0.2 clean verify}}
> succeeds.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-454) typo in munge comments in TestBspBasic
[
https://issues.apache.org/jira/browse/GIRAPH-454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13534369#comment-13534369
]
Eugene Koontz commented on GIRAPH-454:
--
Thanks for the review Alessandro. Will commit today, after I learn how to use
our shiny new git-enabled commit ability :)
> typo in munge comments in TestBspBasic
> --
>
> Key: GIRAPH-454
> URL: https://issues.apache.org/jira/browse/GIRAPH-454
> Project: Giraph
> Issue Type: Bug
> Components: test
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Attachments: GIRAPH-454.patch
>
>
> Simply missing some comment delimiters in the imports part of the test.
> Without this change, {{TestBspBasic}} can't compile, but with this change,
> Hadoop 2.0.2 profile build:
> {{mvn -Phadoop_2.0.2 clean verify}}
> succeeds.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-418) Create maven profile for CDH 4.1.1
[ https://issues.apache.org/jira/browse/GIRAPH-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13533138#comment-13533138 ] Eugene Koontz commented on GIRAPH-418: -- Just noticed that nothing's been done with this JIRA lately. Going to revisit it next week. > Create maven profile for CDH 4.1.1 > -- > > Key: GIRAPH-418 > URL: https://issues.apache.org/jira/browse/GIRAPH-418 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz > Attachments: GIRAPH-418.patch, GIRAPH-418.patch, pom.xml > > > Mirko writes: > bq. Hello all, > bq. I created a now pom.xml file, which allows compiling Giraph with the new > CDH 4.1.2 repository. > bq. I attached it to this email. > bq. But there are still some problems. I can not start the examples, because > I do not have demo data and I do not jet know, what formats have to be used. > Has anybody some test data files to share for a public tutorial? > bq. Currently I prepare a tutorial on working with Giraph, but until now I > had to find out, that the wiki content is quite outdated. The > ShortestPathExample looks nice, but without some hints I can not start. Did I > miss some documentation? If not, I will create it, but therefor I would like > to ask for some help. > bq. My plan is, to create some demo data files and some Demo-Apps comparable > to the benchmark classes but based on real data. I have some correlation > obtained from wikipedia user data, but until now I can not analuse that data > with Giraph. > bq. I get the following error ( at the end of this mail ) if I want to start > the PageRankBenchmark. > bq. Has anybody an idea, what I can do to solve this problem? > bq. Thanks a lot! > bq. Best wishes > bq. Mirko -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-454) typo in munge comments in TestBspBasic
[
https://issues.apache.org/jira/browse/GIRAPH-454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-454:
-
Attachment: GIRAPH-454.patch
> typo in munge comments in TestBspBasic
> --
>
> Key: GIRAPH-454
> URL: https://issues.apache.org/jira/browse/GIRAPH-454
> Project: Giraph
> Issue Type: Bug
> Components: test
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Attachments: GIRAPH-454.patch
>
>
> Simply missing some comment delimiters in the imports part of the test.
> Without this change, {{TestBspBasic}} can't compile, but with this change,
> Hadoop 2.0.2 profile build:
> {{mvn -Phadoop_2.0.2 clean verify}}
> succeeds.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-454) typo in munge comments in TestBspBasic
Eugene Koontz created GIRAPH-454:
Summary: typo in munge comments in TestBspBasic
Key: GIRAPH-454
URL: https://issues.apache.org/jira/browse/GIRAPH-454
Project: Giraph
Issue Type: Bug
Components: test
Reporter: Eugene Koontz
Assignee: Eugene Koontz
Simply missing some comment delimiters in the imports part of the test. Without
this change, {{TestBspBasic}} can't compile, but with this change, Hadoop 2.0.2
profile build:
{{mvn -Phadoop_2.0.2 clean verify}}
succeeds.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-430) Switch to pure-git?
[ https://issues.apache.org/jira/browse/GIRAPH-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499340#comment-13499340 ] Eugene Koontz commented on GIRAPH-430: -- +1. Happy to see this. Thanks Nitay for bringing it to our attention. > Switch to pure-git? > --- > > Key: GIRAPH-430 > URL: https://issues.apache.org/jira/browse/GIRAPH-430 > Project: Giraph > Issue Type: Improvement >Reporter: Nitay Joffe >Assignee: Nitay Joffe >Priority: Minor > > Apache is working on pure-git support. See https://git-wip-us.apache.org/ > From what I can tell incoming new projects are being setup as pure-git, and > this service has been running fairly stable for a while now. > > What do folks think about switching to this? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-418) Create maven profile for CDH 4.1.1
[ https://issues.apache.org/jira/browse/GIRAPH-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496656#comment-13496656 ] Eugene Koontz commented on GIRAPH-418: -- Mirko writes: bq. Hello, I worked on the changes related to all of your comments. Here is the patch to add a new profile, as well as the complete pom file. Next thing for me is now to create some examples, by looking on BigTop and to write documentation on this. Result will be a tutorial. Best wishes Mirko Attaching patch derived from Mirko's attachment from his Nov 13, 2012 mail. > Create maven profile for CDH 4.1.1 > -- > > Key: GIRAPH-418 > URL: https://issues.apache.org/jira/browse/GIRAPH-418 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz > Attachments: GIRAPH-418.patch, GIRAPH-418.patch, pom.xml > > > Mirko writes: > bq. Hello all, > bq. I created a now pom.xml file, which allows compiling Giraph with the new > CDH 4.1.2 repository. > bq. I attached it to this email. > bq. But there are still some problems. I can not start the examples, because > I do not have demo data and I do not jet know, what formats have to be used. > Has anybody some test data files to share for a public tutorial? > bq. Currently I prepare a tutorial on working with Giraph, but until now I > had to find out, that the wiki content is quite outdated. The > ShortestPathExample looks nice, but without some hints I can not start. Did I > miss some documentation? If not, I will create it, but therefor I would like > to ask for some help. > bq. My plan is, to create some demo data files and some Demo-Apps comparable > to the benchmark classes but based on real data. I have some correlation > obtained from wikipedia user data, but until now I can not analuse that data > with Giraph. > bq. I get the following error ( at the end of this mail ) if I want to start > the PageRankBenchmark. > bq. Has anybody an idea, what I can do to solve this problem? > bq. Thanks a lot! > bq. Best wishes > bq. Mirko -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-418) Create maven profile for CDH 4.1.1
[
https://issues.apache.org/jira/browse/GIRAPH-418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-418:
-
Attachment: GIRAPH-418.patch
>From Mirko's Nov 13 mail:
{quote}
Hello,
I worked on the changes related to all of your comments.
Here is the patch to add a new profile, as well as the complete pom file.
Next thing for me is now to create some examples, by looking on
BigTop and to write documentation on this. Result will be a tutorial.
Best wishes
Mirko
{quote}
> Create maven profile for CDH 4.1.1
> --
>
> Key: GIRAPH-418
> URL: https://issues.apache.org/jira/browse/GIRAPH-418
> Project: Giraph
> Issue Type: Improvement
>Reporter: Eugene Koontz
> Attachments: GIRAPH-418.patch, GIRAPH-418.patch, pom.xml
>
>
> Mirko writes:
> bq. Hello all,
> bq. I created a now pom.xml file, which allows compiling Giraph with the new
> CDH 4.1.2 repository.
> bq. I attached it to this email.
> bq. But there are still some problems. I can not start the examples, because
> I do not have demo data and I do not jet know, what formats have to be used.
> Has anybody some test data files to share for a public tutorial?
> bq. Currently I prepare a tutorial on working with Giraph, but until now I
> had to find out, that the wiki content is quite outdated. The
> ShortestPathExample looks nice, but without some hints I can not start. Did I
> miss some documentation? If not, I will create it, but therefor I would like
> to ask for some help.
> bq. My plan is, to create some demo data files and some Demo-Apps comparable
> to the benchmark classes but based on real data. I have some correlation
> obtained from wikipedia user data, but until now I can not analuse that data
> with Giraph.
> bq. I get the following error ( at the end of this mail ) if I want to start
> the PageRankBenchmark.
> bq. Has anybody an idea, what I can do to solve this problem?
> bq. Thanks a lot!
> bq. Best wishes
> bq. Mirko
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-392) Can't build hadoop 0.23+ profiles
[
https://issues.apache.org/jira/browse/GIRAPH-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495584#comment-13495584
]
Eugene Koontz commented on GIRAPH-392:
--
Hi Roman, thanks for your comment in GIRAPH-418. Pasting it in here too.
bq. Bigtop has integrated running Giraph tests against Hadoop 2.X. You can take
a look at how we're doing it wrt. testdata, etc:
https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=blob;f=bigtop-tests/test-artifacts/giraph/src/main/groovy/org/apache/bigtop/itest/giraphsmoke/TestGiraphSmoke.groovy;h=dd82c2f01c9c403847ad15c59975d768f86f23e9;hb=HEAD
> Can't build hadoop 0.23+ profiles
> -
>
> Key: GIRAPH-392
> URL: https://issues.apache.org/jira/browse/GIRAPH-392
> Project: Giraph
> Issue Type: Bug
> Components: build
>Affects Versions: 0.2.0
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Blocker
> Fix For: 0.2.0
>
>
> {noformat}
> $ mvn -Phadoop_2.0.2 -DskipTests clean install
> ..
> [WARNING] The POM for org.apache.hadoop:hadoop-core:jar:0.20.1 is missing, no
> dependency information available
> [WARNING] The POM for org.apache.hadoop:hadoop-test:jar:2.0.2-alpha is
> missing, no dependency information available
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Giraph Parent .. SUCCESS [0.446s]
> [INFO] Apache Giraph . SUCCESS [1:00.793s]
> [INFO] Apache Giraph Formats . FAILURE [0.509s]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 1:01.960s
> [INFO] Finished at: Tue Oct 30 09:59:01 PDT 2012
> [INFO] Final Memory: 69M/719M
> [INFO]
>
> [ERROR] Failed to execute goal on project giraph-formats-contrib: Could not
> resolve dependencies for project
> org.apache.giraph:giraph-formats-contrib:jar:0.2-SNAPSHOT: Failure to find
> org.apache.hadoop:hadoop-test:jar:2.0.2-alpha in
> http://repo1.maven.org/maven2 was cached in the local repository, resolution
> will not be reattempted until the update interval of central has elapsed or
> updates are forced -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :giraph-formats-contrib
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-418) Create maven profile for CDH 4.1.1
[ https://issues.apache.org/jira/browse/GIRAPH-418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-418: - Attachment: GIRAPH-418.patch > Create maven profile for CDH 4.1.1 > -- > > Key: GIRAPH-418 > URL: https://issues.apache.org/jira/browse/GIRAPH-418 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz > Attachments: GIRAPH-418.patch, pom.xml > > > Mirko writes: > bq. Hello all, > bq. I created a now pom.xml file, which allows compiling Giraph with the new > CDH 4.1.2 repository. > bq. I attached it to this email. > bq. But there are still some problems. I can not start the examples, because > I do not have demo data and I do not jet know, what formats have to be used. > Has anybody some test data files to share for a public tutorial? > bq. Currently I prepare a tutorial on working with Giraph, but until now I > had to find out, that the wiki content is quite outdated. The > ShortestPathExample looks nice, but without some hints I can not start. Did I > miss some documentation? If not, I will create it, but therefor I would like > to ask for some help. > bq. My plan is, to create some demo data files and some Demo-Apps comparable > to the benchmark classes but based on real data. I have some correlation > obtained from wikipedia user data, but until now I can not analuse that data > with Giraph. > bq. I get the following error ( at the end of this mail ) if I want to start > the PageRankBenchmark. > bq. Has anybody an idea, what I can do to solve this problem? > bq. Thanks a lot! > bq. Best wishes > bq. Mirko -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-418) Create maven profile for CDH 4.1.1
[ https://issues.apache.org/jira/browse/GIRAPH-418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-418: - Description: Mirko writes: bq. Hello all, bq. I created a now pom.xml file, which allows compiling Giraph with the new CDH 4.1.2 repository. bq. I attached it to this email. bq. But there are still some problems. I can not start the examples, because I do not have demo data and I do not jet know, what formats have to be used. Has anybody some test data files to share for a public tutorial? bq. Currently I prepare a tutorial on working with Giraph, but until now I had to find out, that the wiki content is quite outdated. The ShortestPathExample looks nice, but without some hints I can not start. Did I miss some documentation? If not, I will create it, but therefor I would like to ask for some help. bq. My plan is, to create some demo data files and some Demo-Apps comparable to the benchmark classes but based on real data. I have some correlation obtained from wikipedia user data, but until now I can not analuse that data with Giraph. bq. I get the following error ( at the end of this mail ) if I want to start the PageRankBenchmark. bq. Has anybody an idea, what I can do to solve this problem? bq. Thanks a lot! bq. Best wishes bq. Mirko > Create maven profile for CDH 4.1.1 > -- > > Key: GIRAPH-418 > URL: https://issues.apache.org/jira/browse/GIRAPH-418 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz > Attachments: pom.xml > > > Mirko writes: > bq. Hello all, > bq. I created a now pom.xml file, which allows compiling Giraph with the new > CDH 4.1.2 repository. > bq. I attached it to this email. > bq. But there are still some problems. I can not start the examples, because > I do not have demo data and I do not jet know, what formats have to be used. > Has anybody some test data files to share for a public tutorial? > bq. Currently I prepare a tutorial on working with Giraph, but until now I > had to find out, that the wiki content is quite outdated. The > ShortestPathExample looks nice, but without some hints I can not start. Did I > miss some documentation? If not, I will create it, but therefor I would like > to ask for some help. > bq. My plan is, to create some demo data files and some Demo-Apps comparable > to the benchmark classes but based on real data. I have some correlation > obtained from wikipedia user data, but until now I can not analuse that data > with Giraph. > bq. I get the following error ( at the end of this mail ) if I want to start > the PageRankBenchmark. > bq. Has anybody an idea, what I can do to solve this problem? > bq. Thanks a lot! > bq. Best wishes > bq. Mirko -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-418) Create maven profile for CDH 4.1.1
[ https://issues.apache.org/jira/browse/GIRAPH-418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-418: - Attachment: pom.xml >From mirko.kae...@cloudera.com > Create maven profile for CDH 4.1.1 > -- > > Key: GIRAPH-418 > URL: https://issues.apache.org/jira/browse/GIRAPH-418 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz > Attachments: pom.xml > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-418) Create maven profile for CDH 4.1.1
Eugene Koontz created GIRAPH-418: Summary: Create maven profile for CDH 4.1.1 Key: GIRAPH-418 URL: https://issues.apache.org/jira/browse/GIRAPH-418 Project: Giraph Issue Type: Improvement Reporter: Eugene Koontz Attachments: pom.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-376) Facebook Hadoop profile broken
[ https://issues.apache.org/jira/browse/GIRAPH-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483582#comment-13483582 ] Eugene Koontz commented on GIRAPH-376: -- Thanks Nitay, it's a good idea to keep them sorted for human readability -Eugene > Facebook Hadoop profile broken > -- > > Key: GIRAPH-376 > URL: https://issues.apache.org/jira/browse/GIRAPH-376 > Project: Giraph > Issue Type: Bug >Reporter: Nitay Joffe >Assignee: Nitay Joffe >Priority: Minor > Attachments: GIRAPH-376-2.patch, GIRAPH-376-3.patch, GIRAPH-376.patch > > > Building against Facebook's Hadoop's jar is broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-376) Facebook Hadoop profile broken
[ https://issues.apache.org/jira/browse/GIRAPH-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483252#comment-13483252 ] Eugene Koontz commented on GIRAPH-376: -- Hi Nitay, I haven't had a chance to test with the Facebook profile. I have worked on it to get the Hadoop 23 and Hadoop 2.0.1 profile working. Not there yet, but no fault of your patch. I'm going to continue with that over on GIRAPH-383. So your patch here is fine as far as I can see; +1. One question I had, though: is the lexicographic sorting just for human readability? Or is there a significance to fixing profiles? Thanks, Eugene > Facebook Hadoop profile broken > -- > > Key: GIRAPH-376 > URL: https://issues.apache.org/jira/browse/GIRAPH-376 > Project: Giraph > Issue Type: Bug >Reporter: Nitay Joffe >Assignee: Nitay Joffe >Priority: Minor > Attachments: GIRAPH-376-2.patch, GIRAPH-376-3.patch, GIRAPH-376.patch > > > Building against Facebook's Hadoop's jar is broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-376) Facebook Hadoop profile broken
[ https://issues.apache.org/jira/browse/GIRAPH-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480781#comment-13480781 ] Eugene Koontz commented on GIRAPH-376: -- Nitay, this looks encouraging! I'm having similar problems with the Hadoop 2.0.1 profile. (including the LoggerUtils.java thing you fixed). I'll take a look at this today and test with 2.0.1 and update your patch if need be. Thanks a lot for looking into this. -Eugene > Facebook Hadoop profile broken > -- > > Key: GIRAPH-376 > URL: https://issues.apache.org/jira/browse/GIRAPH-376 > Project: Giraph > Issue Type: Bug >Reporter: Nitay Joffe >Assignee: Nitay Joffe >Priority: Trivial > Attachments: GIRAPH-376-2.patch, GIRAPH-376-3.patch, GIRAPH-376.patch > > > Building against Facebook's Hadoop's jar is broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-372) Write worker addresses to Zookeeper; move addresses and resolution to NettyClient
[
https://issues.apache.org/jira/browse/GIRAPH-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477967#comment-13477967
]
Eugene Koontz commented on GIRAPH-372:
--
Hi Maja, The master doesn't have a taskId, that's why there's that {{null}} as
the second parameter in the code above that you show. Maybe this is going too
far afield for this particular JIRA. Let me just make sure your patch works
with SASL authentication before I comment further.
> Write worker addresses to Zookeeper; move addresses and resolution to
> NettyClient
> -
>
> Key: GIRAPH-372
> URL: https://issues.apache.org/jira/browse/GIRAPH-372
> Project: Giraph
> Issue Type: Improvement
>Reporter: Maja Kabiljo
>Assignee: Maja Kabiljo
> Attachments: GIRAPH-372.patch
>
>
> In preparation for GIRAPH-273, we need to have addresses of all workers
> available, so I write them to Zookeeper along with the master address.
> Since address resoultion is needed in both NettyWorkerClient and
> NettyMasterClient, I moved that to NettyClient, and also added a map
> taskId->address in there. Now NettyMasterClient and NettyWorkerClient don't
> need to take care of InetSocketAddresses, just task ids which they want to
> send messages to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-372) Write worker addresses to Zookeeper; move addresses and resolution to NettyClient
[ https://issues.apache.org/jira/browse/GIRAPH-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477660#comment-13477660 ] Eugene Koontz commented on GIRAPH-372: -- Hi Maja, This seems like a good idea. I wonder if GIRAPH-372 could lead to a solution to the workaround that mentioned in GIRAPH-362? To summarize, we need a way for workers to both talk to the master (needed for GIRAPH-211) and support accounting of requests (which requires an ID of the master or some kind of handle for clients to use to account for requests to the master). We tried to fix this with GIRAPH-360, but for now, we don't provide for accounting of SASL messages from the client to the server. Thanks, Eugene > Write worker addresses to Zookeeper; move addresses and resolution to > NettyClient > - > > Key: GIRAPH-372 > URL: https://issues.apache.org/jira/browse/GIRAPH-372 > Project: Giraph > Issue Type: Improvement >Reporter: Maja Kabiljo >Assignee: Maja Kabiljo > Attachments: GIRAPH-372.patch > > > In preparation for GIRAPH-273, we need to have addresses of all workers > available, so I write them to Zookeeper along with the master address. > Since address resoultion is needed in both NettyWorkerClient and > NettyMasterClient, I moved that to NettyClient, and also added a map > taskId->address in there. Now NettyMasterClient and NettyWorkerClient don't > need to take care of InetSocketAddresses, just task ids which they want to > send messages to. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (GIRAPH-366) TestGraphPartitioner should use getTempPath() everywhere
[ https://issues.apache.org/jira/browse/GIRAPH-366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz closed GIRAPH-366. > TestGraphPartitioner should use getTempPath() everywhere > > > Key: GIRAPH-366 > URL: https://issues.apache.org/jira/browse/GIRAPH-366 > Project: Giraph > Issue Type: Bug >Reporter: Eugene Koontz >Assignee: Eugene Koontz >Priority: Minor > Attachments: GIRAPH-366.patch > > > Easy, one-line fix. Noticed when getting a failure when test tried to create > /tmp/testHashRangePartitioner, but this file was owned by another user. Fix > is simply to use getTempPath() like in other places in test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-366) TestGraphPartitioner should use getTempPath() everywhere
[ https://issues.apache.org/jira/browse/GIRAPH-366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475362#comment-13475362 ] Eugene Koontz commented on GIRAPH-366: -- Committed; thanks Avery for the review! > TestGraphPartitioner should use getTempPath() everywhere > > > Key: GIRAPH-366 > URL: https://issues.apache.org/jira/browse/GIRAPH-366 > Project: Giraph > Issue Type: Bug >Reporter: Eugene Koontz >Assignee: Eugene Koontz >Priority: Minor > Attachments: GIRAPH-366.patch > > > Easy, one-line fix. Noticed when getting a failure when test tried to create > /tmp/testHashRangePartitioner, but this file was owned by another user. Fix > is simply to use getTempPath() like in other places in test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-366) TestGraphPartitioner should use getTempPath() everywhere
[ https://issues.apache.org/jira/browse/GIRAPH-366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-366: - Summary: TestGraphPartitioner should use getTempPath() everywhere (was: TestGraphPartitioner should use getTempPath() rather than new Path() to create outputPath) > TestGraphPartitioner should use getTempPath() everywhere > > > Key: GIRAPH-366 > URL: https://issues.apache.org/jira/browse/GIRAPH-366 > Project: Giraph > Issue Type: Bug >Reporter: Eugene Koontz >Assignee: Eugene Koontz >Priority: Minor > Attachments: GIRAPH-366.patch > > > Easy, one-line fix. Noticed when getting a failure when test tried to create > /tmp/testHashRangePartitioner, but this file was owned by another user. Fix > is simply to use getTempPath() like in other places in test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-366) TestGraphPartitioner should use getTempPath() rather than new Path() to create outputPath
[ https://issues.apache.org/jira/browse/GIRAPH-366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-366: - Attachment: GIRAPH-366.patch > TestGraphPartitioner should use getTempPath() rather than new Path() to > create outputPath > - > > Key: GIRAPH-366 > URL: https://issues.apache.org/jira/browse/GIRAPH-366 > Project: Giraph > Issue Type: Bug >Reporter: Eugene Koontz >Assignee: Eugene Koontz >Priority: Minor > Attachments: GIRAPH-366.patch > > > Easy, one-line fix. Noticed when getting a failure when test tried to create > /tmp/testHashRangePartitioner, but this file was owned by another user. Fix > is simply to use getTempPath() like in other places in test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-366) TestGraphPartitioner should use getTempPath() rather than new Path() to create outputPath
Eugene Koontz created GIRAPH-366: Summary: TestGraphPartitioner should use getTempPath() rather than new Path() to create outputPath Key: GIRAPH-366 URL: https://issues.apache.org/jira/browse/GIRAPH-366 Project: Giraph Issue Type: Bug Reporter: Eugene Koontz Assignee: Eugene Koontz Priority: Minor Easy, one-line fix. Noticed when getting a failure when test tried to create /tmp/testHashRangePartitioner, but this file was owned by another user. Fix is simply to use getTempPath() like in other places in test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-200) remove hadoop RPC and keep just netty
[ https://issues.apache.org/jira/browse/GIRAPH-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474104#comment-13474104 ] Eugene Koontz commented on GIRAPH-200: -- Hi Claudio, Unfortunately not yet, but at least we can now get rid of one munge flag: HADOOP_NON_INTERVERSIONED_RPC. I am attaching a new patch which is the same as Allesandro's most recent, but with this flag removed from pom.xml, because it's not used in the source code at all anymore with his patch. > remove hadoop RPC and keep just netty > - > > Key: GIRAPH-200 > URL: https://issues.apache.org/jira/browse/GIRAPH-200 > Project: Giraph > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Claudio Martella >Assignee: Alessandro Presta > Fix For: 0.2.0 > > Attachments: GIRAPH-200.patch, GIRAPH-200.patch, GIRAPH-200.patch, > GIRAPH-200.patch, GIRAPH-200.patch, GIRAPH-200.patch > > > Given the netty communication behaves faster than rpc, we should just keep > netty-based and drop the older one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-200) remove hadoop RPC and keep just netty
[ https://issues.apache.org/jira/browse/GIRAPH-200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-200: - Attachment: GIRAPH-200.patch This patch removes HADOOP_NON_INTERVERSIONED_RPC from where it occurs in pom.xml. > remove hadoop RPC and keep just netty > - > > Key: GIRAPH-200 > URL: https://issues.apache.org/jira/browse/GIRAPH-200 > Project: Giraph > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Claudio Martella >Assignee: Alessandro Presta > Fix For: 0.2.0 > > Attachments: GIRAPH-200.patch, GIRAPH-200.patch, GIRAPH-200.patch, > GIRAPH-200.patch, GIRAPH-200.patch, GIRAPH-200.patch > > > Given the netty communication behaves faster than rpc, we should just keep > netty-based and drop the older one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-200) remove hadoop RPC and keep just netty
[
https://issues.apache.org/jira/browse/GIRAPH-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473804#comment-13473804
]
Eugene Koontz commented on GIRAPH-200:
--
+1, tried with:
{code}
$HADOOP_RUNTIME/bin/hadoop jar
./target/giraph-0.2-SNAPSHOT-for-hadoop-3.0.0-SNAPSHOT-jar-with-dependencies.jar
\
org.apache.giraph.benchmark.PageRankBenchmark -Dgiraph.useNetty=true \
-Dgiraph.logLevel=debug \
-Dgiraph.authenticate=true -e 100 -s 100 -v -V 100 -w 6
{code}
worked with expected log output showing that SASL authentication is functioning
as expected.
> remove hadoop RPC and keep just netty
> -
>
> Key: GIRAPH-200
> URL: https://issues.apache.org/jira/browse/GIRAPH-200
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Claudio Martella
>Assignee: Alessandro Presta
> Fix For: 0.2.0
>
> Attachments: GIRAPH-200.patch, GIRAPH-200.patch, GIRAPH-200.patch,
> GIRAPH-200.patch, GIRAPH-200.patch
>
>
> Given the netty communication behaves faster than rpc, we should just keep
> netty-based and drop the older one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-168) Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than HADOOP_FACEBOOK) and remove usage of HADOOP
[
https://issues.apache.org/jira/browse/GIRAPH-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473795#comment-13473795
]
Eugene Koontz commented on GIRAPH-168:
--
Interesting tabular comparison of Hadoop API differences:
http://dbeech.github.com/hadoop-api-evolution.html
> Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than
> HADOOP_FACEBOOK) and remove usage of HADOOP
> -
>
> Key: GIRAPH-168
> URL: https://issues.apache.org/jira/browse/GIRAPH-168
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Attachments: GIRAPH-168.patch, GIRAPH-168.patch, GIRAPH-168.patch,
> GIRAPH-168.patch, GIRAPH-168.patch, GIRAPH-168.patch
>
>
> This JIRA relates to the mail thread here:
> http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser
> Currently we check for the munge flags HADOOP, HADOOP_FACEBOOK and
> HADOOP_NON_SECURE when using munge in a few places. Hopefully we can
> eliminate usage of munge in the future, but until then, we can mitigate the
> complexity by consolidating the number of flags checked. This JIRA renames
> HADOOP_FACEBOOK to HADOOP_SECURE, and removes usages of HADOOP, to handle the
> same conditional compilation requirements. It also makes it easier to add
> more maven profiles so that we can easily increase our hadoop version
> coverage.
> This patch modifies the existing hadoop_facebook profile to use the new
> HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.
> It also adds a new hadoop maven profile, hadoop_trunk, which also sets
> HADOOP_SECURE.
> Finally, it adds a default profile, hadoop_0.20.203. This is needed so that
> we can specify its dependencies separately from hadoop_trunk, because the
> hadoop dependencies have changed between trunk and 0.205.0 - the former
> requires hadoop-common, hadoop-mapreduce-client-core, and
> hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.
> With this patch, the following passes:
> {code}
> mvn clean verify && mvn -Phadoop_trunk clean verify && mvn -Phadoop_0.20.203
> clean verify
> {code}
> Current problems:
> * I left in place the usage of HADOOP_NON_SECURE, but note that the profile
> that uses this is hadoop_non_secure, which fails to compile on trunk:
> https://issues.apache.org/jira/browse/GIRAPH-167 .
> * I couldn't get -Phadoop_facebook to work; does this work outside of
> Facebook?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-363) Fix hadoop_0.23 profile broken by GIRAPH-211
[ https://issues.apache.org/jira/browse/GIRAPH-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-363: - Attachment: GIRAPH-363.patch > Fix hadoop_0.23 profile broken by GIRAPH-211 > > > Key: GIRAPH-363 > URL: https://issues.apache.org/jira/browse/GIRAPH-363 > Project: Giraph > Issue Type: Bug >Reporter: Eugene Koontz >Assignee: Eugene Koontz > Attachments: GIRAPH-363.patch > > > I neglected to test hadoop_0.23 when testing GIRAPH-211. Fortunately, Jenkins > caught the regression: > https://builds.apache.org/job/giraph-trunk-hadoop-0.23/92/ > Fix is pretty easy; a patch will be up soon pending testing other profiles. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-363) Fix hadoop_0.23 profile broken by GIRAPH-211
Eugene Koontz created GIRAPH-363: Summary: Fix hadoop_0.23 profile broken by GIRAPH-211 Key: GIRAPH-363 URL: https://issues.apache.org/jira/browse/GIRAPH-363 Project: Giraph Issue Type: Bug Reporter: Eugene Koontz Assignee: Eugene Koontz I neglected to test hadoop_0.23 when testing GIRAPH-211. Fortunately, Jenkins caught the regression: https://builds.apache.org/job/giraph-trunk-hadoop-0.23/92/ Fix is pretty easy; a patch will be up soon pending testing other profiles. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-211) Add secure authentication to Netty IPC
[ https://issues.apache.org/jira/browse/GIRAPH-211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-211: - Attachment: GIRAPH-211.patch Same as last patch except adds what I mistakenly left out of last patch: src/main/java/org/apache/giraph/comm/netty/handler/AuthorizeServerHandler.java src/main/java/org/apache/giraph/comm/netty/handler/ResponseEncoder.java > Add secure authentication to Netty IPC > -- > > Key: GIRAPH-211 > URL: https://issues.apache.org/jira/browse/GIRAPH-211 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz >Assignee: Eugene Koontz > Fix For: 0.2.0 > > Attachments: GIRAPH-211.2.patch, GIRAPH-211.3.patch, > GIRAPH-211.4.patch, GIRAPH-211.patch, GIRAPH-211.patch, GIRAPH-211.patch, > GIRAPH-211.patch, GIRAPH-211.patch, GIRAPH-211.patch, GIRAPH-211.patch, > GIRAPH-211.patch, GIRAPH-211.patch, GIRAPH-211.patch, GIRAPH-211.patch, > GIRAPH-211-proposal.txt > > > Gianmarco De Francisci Morales asked on the user list: > bq. I am getting the exception in the subject when running my giraph program > bq. on a cluster with Kerberos authentication. > This leads to the idea of having Kerberos authentication supported within > GIRAPH. Hopefully it would use our fast GIRAPH-37 IPC, but could also > interoperate with Hadoop security. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-211) Add secure authentication to Netty IPC
[
https://issues.apache.org/jira/browse/GIRAPH-211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-211:
-
Attachment: GIRAPH-211.patch
Avery, modified your patch to fix one problem: a NPE in
NettyClient.sendWritableRequest() when the supplied destWorkerId param is null.
This null param happens when requests are made to the master when
sendWritableRequest() is called by NettyClient.authenticateOnChannel().
With Avery's fix for GIRAPH-360, we associate each request with the taskId of
the worker that is making the request, and use a pair as the
key for tracking requests in the clientRequestIdRequestInfoMap. However, the
Giraph master is not associated with a taskId, and so its taskid is null.
It wouldn't be a problem that the master has a null task id for normal Giraph
requests, but for authentication, workers need to authenticate with the master
just as they must do with the other workers.
This patch works around the problem by simply not registering authentication
requests (requests with type=SASL_TOKEN_MESSAGE_REQUEST) with the
clientRequestIdRequestInfoMap. In other words, it's another workaround like the
"-2" presence in my previous patches. I know this is not ideal, but I wanted to
keep the momentum going for this JIRA.
Rather than this workaround, I thought a possible solution would be a separate
client-side pipeline component that handles request tracking. We'd move the
clientRequestIdRequestInfoMap into this component, which would handles the
tracking functionality that currently is spread across
NettyClient.sendWritableRequest(), .waitAllRequests(), and
ResponseClientHandler.messageReceived().
Or, maybe I'm missing something obvious and easy :)
Tested this patch successfully with:
{code}
mvn -Phadoop_non_secure clean verify && mvn -Phadoop_0.20.203 clean verify &&
mvn -Phadoop_1.0 clean verify && mvn -Phadoop0.23 clean verify && mvn
-Phadoop_2.0.0 clean verify && mvn -Phadoop_2.0.1 clean verify && mvn
-Phadoop_2.0.2 clean verify
{code}
> Add secure authentication to Netty IPC
> --
>
> Key: GIRAPH-211
> URL: https://issues.apache.org/jira/browse/GIRAPH-211
> Project: Giraph
> Issue Type: Improvement
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Fix For: 0.2.0
>
> Attachments: GIRAPH-211.2.patch, GIRAPH-211.3.patch,
> GIRAPH-211.4.patch, GIRAPH-211.patch, GIRAPH-211.patch, GIRAPH-211.patch,
> GIRAPH-211.patch, GIRAPH-211.patch, GIRAPH-211.patch, GIRAPH-211.patch,
> GIRAPH-211.patch, GIRAPH-211.patch, GIRAPH-211.patch, GIRAPH-211-proposal.txt
>
>
> Gianmarco De Francisci Morales asked on the user list:
> bq. I am getting the exception in the subject when running my giraph program
> bq. on a cluster with Kerberos authentication.
> This leads to the idea of having Kerberos authentication supported within
> GIRAPH. Hopefully it would use our fast GIRAPH-37 IPC, but could also
> interoperate with Hadoop security.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-360) Keep track of the task id in ChannelRotater to send requests without knowing the worker id upfront
[
https://issues.apache.org/jira/browse/GIRAPH-360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471801#comment-13471801
]
Eugene Koontz commented on GIRAPH-360:
--
Committed here:
{code}
commit c62902469b36efa8d080dbc4ed5fb470c95d9096
Author: Eugene Joseph Koontz
Date: Mon Oct 8 19:22:32 2012 +
Keep track of the task id in ChannelRotater to send requests without
knowing the worker id upfront (aching v
git-svn-id: https://svn.apache.org/repos/asf/giraph/trunk@1395733
13f79535-47bb-0310-9956-ffa450edef68
{code}
I regret that I forgot to put {{GIRAPH-360}} in the commit message.
> Keep track of the task id in ChannelRotater to send requests without knowing
> the worker id upfront
> --
>
> Key: GIRAPH-360
> URL: https://issues.apache.org/jira/browse/GIRAPH-360
> Project: Giraph
> Issue Type: Sub-task
>Reporter: Avery Ching
>Assignee: Avery Ching
> Fix For: 0.2.0
>
> Attachments: GIRAPH-360.patch
>
>
> Prep for GIRAPH-211
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-360) Keep track of the task id in ChannelRotater to send requests without knowing the worker id upfront
[ https://issues.apache.org/jira/browse/GIRAPH-360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471770#comment-13471770 ] Eugene Koontz commented on GIRAPH-360: -- +1, will commit soon. > Keep track of the task id in ChannelRotater to send requests without knowing > the worker id upfront > -- > > Key: GIRAPH-360 > URL: https://issues.apache.org/jira/browse/GIRAPH-360 > Project: Giraph > Issue Type: Sub-task >Reporter: Avery Ching >Assignee: Avery Ching > Fix For: 0.2.0 > > Attachments: GIRAPH-360.patch > > > Prep for GIRAPH-211 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-211) Add secure authentication to Netty IPC
[ https://issues.apache.org/jira/browse/GIRAPH-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471134#comment-13471134 ] Eugene Koontz commented on GIRAPH-211: -- Wow, thanks so much Avery. I'm going to try out your patch today. Will also try with hadoop-2.0.2. -Eugene > Add secure authentication to Netty IPC > -- > > Key: GIRAPH-211 > URL: https://issues.apache.org/jira/browse/GIRAPH-211 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz >Assignee: Eugene Koontz > Fix For: 0.2.0 > > Attachments: GIRAPH-211.2.patch, GIRAPH-211.3.patch, > GIRAPH-211.patch, GIRAPH-211.patch, GIRAPH-211.patch, GIRAPH-211.patch, > GIRAPH-211.patch, GIRAPH-211.patch, GIRAPH-211.patch, GIRAPH-211.patch, > GIRAPH-211.patch, GIRAPH-211-proposal.txt > > > Gianmarco De Francisci Morales asked on the user list: > bq. I am getting the exception in the subject when running my giraph program > bq. on a cluster with Kerberos authentication. > This leads to the idea of having Kerberos authentication supported within > GIRAPH. Hopefully it would use our fast GIRAPH-37 IPC, but could also > interoperate with Hadoop security. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-346) Top Level POM
[
https://issues.apache.org/jira/browse/GIRAPH-346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470891#comment-13470891
]
Eugene Koontz commented on GIRAPH-346:
--
I don't think this patch needs to be blocked by GIRAPH-211. As big as this
restructuring is, I can rewrite the GIRAPH-211 patch to match it.
However, the current patch here fails when I try to apply it to trunk. One
problem I noticed is that the munge flag {{HADOOP_NON_SASL_RPC}} is used - it
should no longer be used; it was removed by GIRAPH-212 which is now committed
to trunk.
> Top Level POM
> -
>
> Key: GIRAPH-346
> URL: https://issues.apache.org/jira/browse/GIRAPH-346
> Project: Giraph
> Issue Type: Improvement
>Reporter: Nitay Joffe
>Assignee: Nitay Joffe
>Priority: Minor
> Attachments: GIRAPH-346.patch, readme.patch
>
>
> Most Maven projects that have multiple modules (as we do with main / formats)
> have a top level pom.xml. This allows you avoid redundancy and also to build
> everything from top level all at once. I'm using a lot of the formats stuff,
> especially playing with hive / hcatalog and it's a pain having to constantly
> compile everything twice.
> So I propose we move all the main code under a subdir, say giraph/ or main/.
> Then add a pom.xml that both the giraph/ and the formats/ subdirs can point
> to.
> Note we will still build completely separate jars. If people want to just
> build giraph-main without formats they can do so directly from the
> subdirectory.
> Let me know if you guys +1 the idea and I'll whip it up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
