Hi all, I wanted to give a quick status update on the MAPREDUCE-2841 feature branch which was started earlier this summer.
Since establishing the branch, we've accomplished a number of key items: - Imported the initial code from github - Made integrations with other projects more pluggable using a ServiceLoader mechanism - Integrated the project build into our normal maven structure and release tarballs - Fixed several unit test failures that occurred in different developer environments - Fixed a couple of licensing issues with code - Various code cleanup for style, etc. - Fixed all of the compiler warnings I've also spent some time doing comparative microbenchmarks of the mapside sort between this implementation, the existing MR implementation, and Facebook's implementation (available on github). I haven't posted those yet, but I hope to do so later this week. The summary is that the native implementation compares quite favorably to the other options. This week, I'm hoping to close out a couple remaining items: - A few small code cleanup items I have pending locally (eg improving javadocs, etc) - Some basic cluster verification and performance tests - Post benchmark results mentioned above After these items are done, I think we are ready to merge this feature to trunk. After it is in trunk, we can continue to make improvements, add documentation, etc. Because the patch is structured as completely new code (no changes to existing functionality, manually enabled on a per-job basis), this should be a "zero-risk" merge. I'm anticipating calling this merge vote next week, and would appreciate if other committers can plan to be ready to vote at that time. That is to say, if you anticipate wanting to review the branch merge, this is your "heads up" that the vote is coming. Please start your reviews this week. Thanks -Todd
