[jira] [Created] (GIRAPH-109) GiraphRunner should provide support for combiners
GiraphRunner should provide support for combiners - Key: GIRAPH-109 URL: https://issues.apache.org/jira/browse/GIRAPH-109 Project: Giraph Issue Type: Improvement Affects Versions: 0.70.0 Reporter: Sebastian Schelter Currently there's no way to tell GiraphRunner that you want to use a Combiner. A simple option should be added, similar to the way in- and outputformats are specified. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-110) Add guide to setup the enviroment for running the unit tests in a pseudo-distributed hadoop instance
[ https://issues.apache.org/jira/browse/GIRAPH-110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173470#comment-13173470 ] Hudson commented on GIRAPH-110: --- Integrated in Giraph-trunk-Commit #55 (See [https://builds.apache.org/job/Giraph-trunk-Commit/55/]) GIRAPH-110: Add guide to setup the enviroment for running the unittests in a pseudo-distributed hadoop instance. (ssc via aching) aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1221461 Files : * /incubator/giraph/trunk/CHANGELOG * /incubator/giraph/trunk/README Add guide to setup the enviroment for running the unit tests in a pseudo-distributed hadoop instance Key: GIRAPH-110 URL: https://issues.apache.org/jira/browse/GIRAPH-110 Project: Giraph Issue Type: Improvement Affects Versions: 0.70.0 Reporter: Sebastian Schelter Assignee: Sebastian Schelter Priority: Minor Fix For: 0.70.0 Attachments: GIRAPH-110.2.patch, GIRAPH-110.patch Giraph should provide a small guide for setting up the local environment to run the unit tests in a pseudo-distributed hadoop instance as there are some non-obvious hurdles to take. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-111) Refactor I/O to be independent of Map/Reduce
Refactor I/O to be independent of Map/Reduce Key: GIRAPH-111 URL: https://issues.apache.org/jira/browse/GIRAPH-111 Project: Giraph Issue Type: Improvement Components: graph Reporter: Ed Kohlwey The I/O mechanisms should probably be abstracted entirely from Map/Reduce in order to support making Giraph an independent framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-108) Refactor code to run independently of Map/Reduce
[ https://issues.apache.org/jira/browse/GIRAPH-108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173497#comment-13173497 ] Ed Kohlwey commented on GIRAPH-108: --- {quote} is probably not going to fly. I'd rather not introduce anything more hacky than is strictly necessary. I'll try to take a look at this code by the end of the week. {quote} I agree. My philosophy in this patch is to balance incremental change against dropping a patch bomb. I thought about tearing up the Input and Output format classes but realized that it would probably make the patch gigantic (and thus less likely to be accepted). I went ahead and started a discussion on GIRAPH-111 for this. Refactor code to run independently of Map/Reduce Key: GIRAPH-108 URL: https://issues.apache.org/jira/browse/GIRAPH-108 Project: Giraph Issue Type: Improvement Components: graph Reporter: Ed Kohlwey Attachments: GIRAPH-108 It would be nice for Giraph to be refactored such that the code could eventually be run outside of map/reduce. This will allow people to write drivers that can run in the cool new resource manager frameworks like Mesos and YARN, and eventually let the application's code base evolve to be independent of map/reduce. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-111) Refactor I/O to be independent of Map/Reduce
[ https://issues.apache.org/jira/browse/GIRAPH-111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173512#comment-13173512 ] Avery Ching commented on GIRAPH-111: I'm not clear on why this is necessary. Couldn't we simply call the I/O methods as Hadoop would when we're not using Hadoop? Am I missing something? Refactor I/O to be independent of Map/Reduce Key: GIRAPH-111 URL: https://issues.apache.org/jira/browse/GIRAPH-111 Project: Giraph Issue Type: Improvement Components: graph Reporter: Ed Kohlwey The I/O mechanisms should probably be abstracted entirely from Map/Reduce in order to support making Giraph an independent framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-111) Refactor I/O to be independent of Map/Reduce
[ https://issues.apache.org/jira/browse/GIRAPH-111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173537#comment-13173537 ] Jakob Homan commented on GIRAPH-111: bq. I'm not clear on why this is necessary. I agree. Hadoop's file formats, etc. are designed to be exceedingly forgiving and flexible as to the underlying storage mechanism. Can you point to where they're lacking for Mesos' case? bq. We could also copy out the relevant Hadoop I/O classes (InputFormat, OutputFormat, etc) into Giraph, rename their packages, and begin reworking them in an appropriate way to better suit Giraph. -1. Therein lies madness. Refactor I/O to be independent of Map/Reduce Key: GIRAPH-111 URL: https://issues.apache.org/jira/browse/GIRAPH-111 Project: Giraph Issue Type: Improvement Components: graph Reporter: Ed Kohlwey The I/O mechanisms should probably be abstracted entirely from Map/Reduce in order to support making Giraph an independent framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-110) Add guide to setup the enviroment for running the unit tests in a pseudo-distributed hadoop instance
[ https://issues.apache.org/jira/browse/GIRAPH-110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173631#comment-13173631 ] Jakob Homan commented on GIRAPH-110: Sorry to be late on this one, but I'd been meaning to ask if we should retire most of the README content in favor of the site documentation? The content between the two was originally duplicated and is starting to drift... Add guide to setup the enviroment for running the unit tests in a pseudo-distributed hadoop instance Key: GIRAPH-110 URL: https://issues.apache.org/jira/browse/GIRAPH-110 Project: Giraph Issue Type: Improvement Affects Versions: 0.70.0 Reporter: Sebastian Schelter Assignee: Sebastian Schelter Priority: Minor Fix For: 0.70.0 Attachments: GIRAPH-110.2.patch, GIRAPH-110.patch Giraph should provide a small guide for setting up the local environment to run the unit tests in a pseudo-distributed hadoop instance as there are some non-obvious hurdles to take. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-113) Change cast to Vertex used in prepareSuperstep() to BasicVertex
Change cast to Vertex used in prepareSuperstep() to BasicVertex --- Key: GIRAPH-113 URL: https://issues.apache.org/jira/browse/GIRAPH-113 Project: Giraph Issue Type: Bug Reporter: Yuanyuan Tian Priority: Minor Hi, I decided to use LongDoubleFloatDoubleVertex in a graph algorithm because it uses more compact and efficient mahout collections. However I run into an error when running the algorithm: java.lang.ClassCastException: org.apache.giraph.graph.LongDoubleFloatDoubleVertex cannot be cast to org.apache.giraph.graph.Vertex at org.apache.giraph.comm.BasicRPCCommunications.prepareSuperstep(BasicRPCCommunications.java:1016) at org.apache.giraph.graph.BspServiceWorker.startSuperstep(BspServiceWorker.java:843) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:569) at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:728) ... 7 more Basically, the problem is that in BasicRPCCommunications.prepareSuperStep(), the LongDoubleFloatDoubleVertex are cast to Vertex in the following code fragment. But LongDoubleFloatDoubleVertex inherits from BasicVertex instead of Vertex. if (vertex != null) { ((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex); partition.putVertex((VertexI, V, E, M) vertex); } else if (originalVertex != null) { partition.removeVertex(originalVertex.getVertexId()); } I did a simple change: cast LongDoubleFloatDoubleVertex to BasicVertex. The problem went away, and the algorithm finished without any error. But I am not sure this change has any implication to other parts of the code. So, I hope to get some comments from the Giraph developers. if (vertex != null) { ((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex); partition.putVertex((BasicVertexI, V, E, M) vertex); } else if (originalVertex != null) { partition.removeVertex(originalVertex.getVertexId()); } -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (GIRAPH-113) Change cast to Vertex used in prepareSuperstep() to BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avery Ching reassigned GIRAPH-113: -- Assignee: Avery Ching Change cast to Vertex used in prepareSuperstep() to BasicVertex --- Key: GIRAPH-113 URL: https://issues.apache.org/jira/browse/GIRAPH-113 Project: Giraph Issue Type: Bug Reporter: Yuanyuan Tian Assignee: Avery Ching Priority: Minor Hi, I decided to use LongDoubleFloatDoubleVertex in a graph algorithm because it uses more compact and efficient mahout collections. However I run into an error when running the algorithm: java.lang.ClassCastException: org.apache.giraph.graph.LongDoubleFloatDoubleVertex cannot be cast to org.apache.giraph.graph.Vertex at org.apache.giraph.comm.BasicRPCCommunications.prepareSuperstep(BasicRPCCommunications.java:1016) at org.apache.giraph.graph.BspServiceWorker.startSuperstep(BspServiceWorker.java:843) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:569) at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:728) ... 7 more Basically, the problem is that in BasicRPCCommunications.prepareSuperStep(), the LongDoubleFloatDoubleVertex are cast to Vertex in the following code fragment. But LongDoubleFloatDoubleVertex inherits from BasicVertex instead of Vertex. if (vertex != null) { ((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex); partition.putVertex((VertexI, V, E, M) vertex); } else if (originalVertex != null) { partition.removeVertex(originalVertex.getVertexId()); } I did a simple change: cast LongDoubleFloatDoubleVertex to BasicVertex. The problem went away, and the algorithm finished without any error. But I am not sure this change has any implication to other parts of the code. So, I hope to get some comments from the Giraph developers. if (vertex != null) { ((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex); partition.putVertex((BasicVertexI, V, E, M) vertex); } else if (originalVertex != null) { partition.removeVertex(originalVertex.getVertexId()); } -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-113) Change cast to Vertex used in prepareSuperstep() to BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avery Ching updated GIRAPH-113: --- Attachment: GIRAPH-113.patch Change cast to Vertex used in prepareSuperstep() to BasicVertex --- Key: GIRAPH-113 URL: https://issues.apache.org/jira/browse/GIRAPH-113 Project: Giraph Issue Type: Bug Reporter: Yuanyuan Tian Assignee: Avery Ching Priority: Minor Attachments: GIRAPH-113.patch Hi, I decided to use LongDoubleFloatDoubleVertex in a graph algorithm because it uses more compact and efficient mahout collections. However I run into an error when running the algorithm: java.lang.ClassCastException: org.apache.giraph.graph.LongDoubleFloatDoubleVertex cannot be cast to org.apache.giraph.graph.Vertex at org.apache.giraph.comm.BasicRPCCommunications.prepareSuperstep(BasicRPCCommunications.java:1016) at org.apache.giraph.graph.BspServiceWorker.startSuperstep(BspServiceWorker.java:843) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:569) at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:728) ... 7 more Basically, the problem is that in BasicRPCCommunications.prepareSuperStep(), the LongDoubleFloatDoubleVertex are cast to Vertex in the following code fragment. But LongDoubleFloatDoubleVertex inherits from BasicVertex instead of Vertex. if (vertex != null) { ((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex); partition.putVertex((VertexI, V, E, M) vertex); } else if (originalVertex != null) { partition.removeVertex(originalVertex.getVertexId()); } I did a simple change: cast LongDoubleFloatDoubleVertex to BasicVertex. The problem went away, and the algorithm finished without any error. But I am not sure this change has any implication to other parts of the code. So, I hope to get some comments from the Giraph developers. if (vertex != null) { ((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex); partition.putVertex((BasicVertexI, V, E, M) vertex); } else if (originalVertex != null) { partition.removeVertex(originalVertex.getVertexId()); } -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-113) Change cast to Vertex used in prepareSuperstep() to BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173921#comment-13173921 ] Jakob Homan commented on GIRAPH-113: +1 (grumbling about GIRAPH-83) Change cast to Vertex used in prepareSuperstep() to BasicVertex --- Key: GIRAPH-113 URL: https://issues.apache.org/jira/browse/GIRAPH-113 Project: Giraph Issue Type: Bug Reporter: Yuanyuan Tian Assignee: Avery Ching Priority: Minor Attachments: GIRAPH-113.patch Hi, I decided to use LongDoubleFloatDoubleVertex in a graph algorithm because it uses more compact and efficient mahout collections. However I run into an error when running the algorithm: java.lang.ClassCastException: org.apache.giraph.graph.LongDoubleFloatDoubleVertex cannot be cast to org.apache.giraph.graph.Vertex at org.apache.giraph.comm.BasicRPCCommunications.prepareSuperstep(BasicRPCCommunications.java:1016) at org.apache.giraph.graph.BspServiceWorker.startSuperstep(BspServiceWorker.java:843) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:569) at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:728) ... 7 more Basically, the problem is that in BasicRPCCommunications.prepareSuperStep(), the LongDoubleFloatDoubleVertex are cast to Vertex in the following code fragment. But LongDoubleFloatDoubleVertex inherits from BasicVertex instead of Vertex. if (vertex != null) { ((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex); partition.putVertex((VertexI, V, E, M) vertex); } else if (originalVertex != null) { partition.removeVertex(originalVertex.getVertexId()); } I did a simple change: cast LongDoubleFloatDoubleVertex to BasicVertex. The problem went away, and the algorithm finished without any error. But I am not sure this change has any implication to other parts of the code. So, I hope to get some comments from the Giraph developers. if (vertex != null) { ((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex); partition.putVertex((BasicVertexI, V, E, M) vertex); } else if (originalVertex != null) { partition.removeVertex(originalVertex.getVertexId()); } -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-113) Change cast to Vertex used in prepareSuperstep() to BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173924#comment-13173924 ] Hudson commented on GIRAPH-113: --- Integrated in Giraph-trunk-Commit #56 (See [https://builds.apache.org/job/Giraph-trunk-Commit/56/]) GIRAPH-113: Change cast to Vertex used in prepareSuperstep() to BasicVertex. (humming80 via aching) aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1221634 Files : * /incubator/giraph/trunk/CHANGELOG * /incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java Change cast to Vertex used in prepareSuperstep() to BasicVertex --- Key: GIRAPH-113 URL: https://issues.apache.org/jira/browse/GIRAPH-113 Project: Giraph Issue Type: Bug Reporter: Yuanyuan Tian Assignee: Avery Ching Priority: Minor Attachments: GIRAPH-113.patch Hi, I decided to use LongDoubleFloatDoubleVertex in a graph algorithm because it uses more compact and efficient mahout collections. However I run into an error when running the algorithm: java.lang.ClassCastException: org.apache.giraph.graph.LongDoubleFloatDoubleVertex cannot be cast to org.apache.giraph.graph.Vertex at org.apache.giraph.comm.BasicRPCCommunications.prepareSuperstep(BasicRPCCommunications.java:1016) at org.apache.giraph.graph.BspServiceWorker.startSuperstep(BspServiceWorker.java:843) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:569) at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:728) ... 7 more Basically, the problem is that in BasicRPCCommunications.prepareSuperStep(), the LongDoubleFloatDoubleVertex are cast to Vertex in the following code fragment. But LongDoubleFloatDoubleVertex inherits from BasicVertex instead of Vertex. if (vertex != null) { ((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex); partition.putVertex((VertexI, V, E, M) vertex); } else if (originalVertex != null) { partition.removeVertex(originalVertex.getVertexId()); } I did a simple change: cast LongDoubleFloatDoubleVertex to BasicVertex. The problem went away, and the algorithm finished without any error. But I am not sure this change has any implication to other parts of the code. So, I hope to get some comments from the Giraph developers. if (vertex != null) { ((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex); partition.putVertex((BasicVertexI, V, E, M) vertex); } else if (originalVertex != null) { partition.removeVertex(originalVertex.getVertexId()); } -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: GIRAPH-112: Use elements() properly in LongDoubleFloatDoubleVertex
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3287/ --- Review request for giraph. Summary --- As pointed out by YuanYua, the array returned by elements() cannot have its length used since the array contains all the elements currently stored in the mahout collections, even including invalid elements between size and capacity. Whenever possible I converted elements() into forEach(), forEachKey(), forEachPair(). Used size() in other cases. Fixed some formatting violations as well in LongDoubleFloatDoubleVertex.java. This addresses bug GIRAPH-112. https://issues.apache.org/jira/browse/GIRAPH-112 Diffs - http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleVertex.java 1221634 Diff: https://reviews.apache.org/r/3287/diff Testing --- Local unittests and MR unittests. Thanks, Avery
[jira] [Commented] (GIRAPH-108) Refactor code to run independently of Map/Reduce
[ https://issues.apache.org/jira/browse/GIRAPH-108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173931#comment-13173931 ] Avery Ching commented on GIRAPH-108: Actually, I'll let Jakob take a first crack at looking at this since he's got some expertise in the area. Refactor code to run independently of Map/Reduce Key: GIRAPH-108 URL: https://issues.apache.org/jira/browse/GIRAPH-108 Project: Giraph Issue Type: Improvement Components: graph Reporter: Ed Kohlwey Attachments: GIRAPH-108 It would be nice for Giraph to be refactored such that the code could eventually be run outside of map/reduce. This will allow people to write drivers that can run in the cool new resource manager frameworks like Mesos and YARN, and eventually let the application's code base evolve to be independent of map/reduce. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-112) Use elements() properly in LongDoubleFloatDoubleVertex
[ https://issues.apache.org/jira/browse/GIRAPH-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173932#comment-13173932 ] jirapos...@reviews.apache.org commented on GIRAPH-112: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3287/ --- Review request for giraph. Summary --- As pointed out by YuanYua, the array returned by elements() cannot have its length used since the array contains all the elements currently stored in the mahout collections, even including invalid elements between size and capacity. Whenever possible I converted elements() into forEach(), forEachKey(), forEachPair(). Used size() in other cases. Fixed some formatting violations as well in LongDoubleFloatDoubleVertex.java. This addresses bug GIRAPH-112. https://issues.apache.org/jira/browse/GIRAPH-112 Diffs - http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleVertex.java 1221634 Diff: https://reviews.apache.org/r/3287/diff Testing --- Local unittests and MR unittests. Thanks, Avery Use elements() properly in LongDoubleFloatDoubleVertex -- Key: GIRAPH-112 URL: https://issues.apache.org/jira/browse/GIRAPH-112 Project: Giraph Issue Type: Bug Components: graph Affects Versions: 0.70.0 Environment: Any Reporter: Yuanyuan Tian Fix For: 0.70.0 Attachments: GIRAPH-112.patch Original Estimate: 5m Remaining Estimate: 5m I found a bug in LongDoubleFloatDoubleVertex.write(DataOutput out) when running a small graph algorithm. The symptom is that a vertex read from a different worker becomes junk after the RPC communication. And the source of the problem is the writing of the messages in LongDoubleFloatDoubleVertex.write(DataOutput out): for(double msg : messageList.elements()) { out.writeDouble(msg); } Here messageList.elements() will returns all the elements currently stored in the mahout DoubleArrayList, even including invalid elements between size and capacity. Therefore, the write() function will write a bunch of invalid messages, which will cause error when reading them back in readfields(). The following is a simple solution: double[] elements=messageList.elements(); for(int i=0; imessageList.size(); i++) { out.writeDouble(elements[i]); } -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: GIRAPH-112: Use elements() properly in LongDoubleFloatDoubleVertex
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3287/#review4036 --- Ship it! I ran into the same issue yesterday and the solution presented here is correct. For reasons of efficiency, list.elements() returns the internal underlying array for the list, which might be bigger than the number of elements stored in the list. Therefore you should only iterate until list.size() or use the foreachKey() callback. - Sebastian On 2011-12-21 07:50:20, Avery Ching wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3287/ --- (Updated 2011-12-21 07:50:20) Review request for giraph. Summary --- As pointed out by YuanYua, the array returned by elements() cannot have its length used since the array contains all the elements currently stored in the mahout collections, even including invalid elements between size and capacity. Whenever possible I converted elements() into forEach(), forEachKey(), forEachPair(). Used size() in other cases. Fixed some formatting violations as well in LongDoubleFloatDoubleVertex.java. This addresses bug GIRAPH-112. https://issues.apache.org/jira/browse/GIRAPH-112 Diffs - http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleVertex.java 1221634 Diff: https://reviews.apache.org/r/3287/diff Testing --- Local unittests and MR unittests. Thanks, Avery