Hello everybody I was wondering if it was possible to extend the concept of aggregator from a “global” to a “local-only” perspective.
Normally, aggregators DO cause network traffic because of the cycle: Workers -> Aggregator Owner-> MasterAggregator -> AggregatorOwner -> Workers What if I’d like to fetch and aggregate values as I would normally do with aggregators but without causing this traffic? Let’s assume this situation: 1 - Define a custom partitioning class and let it partition the graph. This is the partition used to assign vertices to workers. 2 - in the computation class, every time che compute method is called on a vertex, the data needed for computation is stored inside the vertex neighbours but also in non-neighbouring vertices (think about Force Directed layout algorithm for example; to compute the forces, is necessary the distance between neighbouring and not-neighbouring vertices, applying different kind of forces). — Given that the compute class is computing on vertex X a - I pick information from X neighbours as I would normally do (iterating its edges or the incoming messages) b - When it comes to non-neighbouring vertices I would like to use data from X worker only. The first thing I tried to understand before asking this question was: does this make any sense? I am probably wrong, but this actually does. If I partition my graph to maximize locality, what I am actually trying to do is to reduce the network traffic as much as possibile. My doubt is that if I use aggregators to achieve the result the network traffic would be heavy, probably losing the advantages of the initial partitioning. What if I could access and modify an aggregator-like local data structure in the same fashion (i.e. “getAggregatedValue”) but without broadcasting it (assuming that I do not need the aggregator to be accessible to every worker)? Or could it be possibile to manually assign partition owners in order to minimise network traffic (if I need to aggregate all values from vertices in partition 3 and 3 only, I assign the partition 3 aggregator owner to partition 3 worker)? I hope in your comprehension and I hope I somehow caught your attention, even if for a brief moment. Ask me if something is not clear ;) Cheers! ~~~~~~~~~~~~~~~~~~~ Ing. Alessio Arleo Dottorando in Ingegneria Industriale e dell’Informazione Dottore Magistrale in Ingegneria Informatica e dell’Automazione Dottore in Ingegneria Informatica ed Elettronica Linkedin: it.linkedin.com/in/IngArleo <http://it.linkedin.com/in/IngArleo> Skype: Ing. Alessio Arleo Tel: +39 075 5853920 Cell: +39 349 0575782 ~~~~~~~~~~~~~~~~~~~