Re: How to measure communication between nodes in Spark Standalone Cluster?

[Yifan LI]

I am not sure there is a direct way(an api in graphx, etc) to measure the 
number of transferred vertex values among nodes during computation.

It might depend on:
- the operations in your application, e.g. only communicate with its immediate 
neighbours for each vertex.
- the partition strategy you chose, wrt the vertices replication factor
- the distribution of partitions on cluster
...


Best,
Yifan LI
LIP6, UPMC, Paris





> On 17 Nov 2014, at 11:59, Hlib Mykhailenko  wrote:
> 
> Hello,
> 
> I use Spark Standalone Cluster and I want to measure somehow internode 
> communication. 
> As I understood, Graphx transfers only vertices values. Am I right?  
> 
> But I do not want to get number of bytes which were transferred between any 
> two nodes.
> So is  there way to measure how many values of vertices were transferred 
> among nodes?
> 
> Thanks!
> 
> --
> Cordialement,
> Hlib Mykhailenko
> Doctorant à INRIA Sophia-Antipolis Méditerranée 
> 
> 2004 Route des Lucioles BP93
> 06902 SOPHIA ANTIPOLIS cedex
> 

Re: How to measure communication between nodes in Spark Standalone Cluster?

[Akhil Das]

You can use Ganglia to see the overall data transfer across the
cluster/nodes. I don't think there's a direct way to get the vertices being
transferred.

Thanks
Best Regards

On Mon, Nov 17, 2014 at 4:29 PM, Hlib Mykhailenko  wrote:

> Hello,
>
> I use Spark Standalone Cluster and I want to measure somehow internode
> communication.
> As I understood, Graphx transfers only vertices values. Am I right?
>
> But I do not want to get number of bytes which were transferred between
> any two nodes.
> So is  there way to measure how many values of vertices were transferred
> among nodes?
>
> Thanks!
>
> --
> Cordialement,
> *Hlib Mykhailenko*
> Doctorant à INRIA Sophia-Antipolis Méditerranée
> 
> 2004 Route des Lucioles BP93
> 06902 SOPHIA ANTIPOLIS cedex
>
>

How to measure communication between nodes in Spark Standalone Cluster?

[Hlib Mykhailenko]

Hello, 

I use Spark Standalone Cluster and I want to measure somehow internode 
communication. 
As I understood, Graphx transfers only vertices values. Am I right? 

But I do not want to get number of bytes which were transferred between any two 
nodes. 
So is there way to measure how many values of vertices were transferred among 
nodes? 

Thanks! 

-- 
Cordialement, 
Hlib Mykhailenko 
Doctorant à INRIA Sophia-Antipolis Méditerranée 
2004 Route des Lucioles BP93 
06902 SOPHIA ANTIPOLIS cedex