Hi, Our Hadoop Accelerator has its own shuffle algorithm. When jobs request arrives, we assign mappers and reducers to the most appropriate nodes in terms of data locality and available resources.
Shuffle itself adds K-V pairs of local reducer to sorted collection right away. K-V pairs of remote reducers are packed into batches and sent to them, then added to sorted collection. Internally we share job state between nodes with help of cache, this is why you may see occasional transactions. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignite-as-Hadoop-s-Map-Reduce-engine-how-it-works-tp10551p10592.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
