My late thanks to the nice advice. I have tried this, it works. However,
to produce the line number I had to rescan the files again, add new line
numbers and then resave them as new files. It took a long time because
they are very big. Are there any built in functions that could
automatically provide the current filename (if there are multiple files)
and the line numbers in Map/Reduce?
Shi
On 2010-10-20 21:16, Hieu Khac Le wrote:
How about using the line number as the key and the string at that line as value.
-------
Please excuse typos and brief nature of this email sent from my mobile device
On Oct 20, 2010, at 9:07 PM, Shi Yu<[email protected]> wrote:
Hi,
I have a problem of comparing two huge files (100G each) consist of string
sequence. It is more like the file text compare problem. I would like to find
out how many strings are different within these two files in the natural order.
Can this task be modeled as a map/reduce job? Currently I have no idea how to
control the split of map and make sure the two input threads in one map task
are pointing to the same positions in the files.
Shi
--
Postdoctoral Scholar
Institute for Genomics and Systems Biology
Department of Medicine, the University of Chicago
Knapp Center for Biomedical Discovery
900 E. 57th St. Room 10148
Chicago, IL 60637, US
Tel: 773-702-6799