[
https://issues.apache.org/jira/browse/GIRAPH-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033356#comment-16033356
]
ASF GitHub Bot commented on GIRAPH-1148:
----------------------------------------
Github user dlogothetis commented on a diff in the pull request:
https://github.com/apache/giraph/pull/39#discussion_r119680240
--- Diff:
giraph-block-app/src/main/java/org/apache/giraph/block_app/library/Pieces.java
---
@@ -320,6 +325,91 @@ public String toString() {
}
/**
+ * Like reduceAndBroadcast, but uses array of handles for reducers and
+ * broadcasts, to make it feasible and performant when values are large.
+ * Each supplied value to reduce will be reduced in the handle defined by
+ * handleHashSupplier%numHandles
+ *
+ * @param <S> Single value type, objects passed on workers
+ * @param <R> Reduced value type
+ * @param <I> Vertex id type
+ * @param <V> Vertex value type
+ * @param <E> Edge value type
+ */
+ public static
+ <S, R extends Writable, I extends WritableComparable, V extends Writable,
+ E extends Writable>
+ Piece<I, V, E, NoMessage, Object> reduceAndBroadcastWithArrayOfHandles(
+ final String name,
+ final int numHandles,
+ final ReduceOperation<S, R> reduceOp,
+ final SupplierFromVertex<I, V, E, Long> handleHashSupplier,
+ final SupplierFromVertex<I, V, E, S> valueSupplier,
+ final ConsumerWithVertex<I, V, E, R> reducedValueConsumer) {
+ return new Piece<I, V, E, NoMessage, Object>() {
+ protected ArrayOfHandles.ArrayOfReducers<S, R> reducers;
+ protected BroadcastArrayHandle<R> broadcasts;
+
+ private int getHandleIndex(Vertex<I, V, E> vertex) {
+ return (int) Math.abs(handleHashSupplier.get(vertex) % numHandles);
+ }
+
+ @Override
+ public void registerReducers(
+ final CreateReducersApi reduceApi, Object executionStage) {
+ reducers = new ArrayOfHandles.ArrayOfReducers<>(
+ numHandles,
+ new Supplier<ReducerHandle<S, R>>() {
+ @Override
+ public ReducerHandle<S, R> get() {
+ return reduceApi.createLocalReducer(reduceOp);
--- End diff --
This means that the same ReduceOperation instance is going to be shared
across the different handles. Not entirely sure how the (de)serialization will
work here.
> Connected components - make calculate sizes work with large number of
> components
> --------------------------------------------------------------------------------
>
> Key: GIRAPH-1148
> URL: https://issues.apache.org/jira/browse/GIRAPH-1148
> Project: Giraph
> Issue Type: Improvement
> Reporter: Maja Kabiljo
> Assignee: Maja Kabiljo
>
> Currently if we have a graph with large number of connected components,
> calculating connected components sizes fails because reducer becomes too
> large. Use array of handles instead.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
