Hi,
I am trying to do a the connected component example. I have modified
it, the Writables a bit to fit my example. Some of the vertex indexes are
negative (long hashed of strings), I tried with a sample example with
around 10 vertices and it works properly. But when I load the whole set
around 1.5B vertices , all of them do not get loaded and the final output
misses a lot of the pairs. Any idea what might be the reason ? is it
required to have the vertex indexes as positive only, I tried with -ve with
a sample set and it works, any other place I should look into or put debug
statements ?
I am running with 118 workers.
Cluster Config :
Running Map TasksRunning Reduce TasksTotal SubmissionsNodesOccupied Map
SlotsOccupied Reduce SlotsReserved Map SlotsReserved Reduce SlotsMap Task
CapacityReduce Task CapacityAvg. Tasks/NodeBlacklisted NodesExcluded Nodes 0
0387 <http://had24.rsk.admobius.com:50030/machines.jsp?type=active>000016884
36.000 <http://had24.rsk.admobius.com:50030/machines.jsp?type=blacklisted>0
--
Best Regards,
Jyotirmoy Sundi
Data Engineer,
Admobius
San Francisco, CA 94158
