The Map/Reduce SGD patch includes a very nice trick which I did not know about. Here is an example:
https://reviews.apache.org/r/3072/diff/2/?file=63195#file63195line36 It uses DummyRecordWriter to ship key/value pairs from mapper to reducer. On Mon, Dec 12, 2011 at 12:43 PM, Isabel Drost <[email protected]> wrote: > On 08.12.2011 Grant Ingersoll wrote: >> great, b/c these really are mainstream tests. I suspect most of our >> overhead is simply due to running map reduce jobs. > > Is there anything to be gained by checking the code itself with mrunit (I know > it does have limitations, but if those tests really only test map and reduce > functions, it might be good enough) and moving integration tests to a "run > only > nightly" mode? > > > Isabel -- Lance Norskog [email protected]
