I just wrote a test where fs.default.name is file:/// and mapred.job.tracker is set to local. The test ran fine, I also see mapper and reducer were invoked but what I am trying to understand is that how did this run without specifying the job tracker port and which port task tracker connected with job tracker. It's not clear from the output:
Also what's the difference between this and bringing up miniDFS cluster? INFO org.apache.hadoop.mapred.FileInputFormat [main]: Total input paths to proc ess : 1 INFO org.apache.hadoop.mapred.JobClient [main]: Running job: job_local_0001 INFO org.apache.hadoop.mapred.Task [Thread-11]: Using ResourceCalculatorPlugin : null INFO org.apache.hadoop.mapred.MapTask [Thread-11]: numReduceTasks: 1 INFO org.apache.hadoop.mapred.MapTask [Thread-11]: io.sort.mb = 100 INFO org.apache.hadoop.mapred.MapTask [Thread-11]: data buffer = 79691776/99614 720 INFO org.apache.hadoop.mapred.MapTask [Thread-11]: record buffer = 262144/32768 0 INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z ip 92127 INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z ip 1 INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z ip 92127 INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z ip 1 INFO org.apache.hadoop.mapred.MapTask [Thread-11]: Starting flush of map output INFO org.apache.hadoop.mapred.MapTask [Thread-11]: Finished spill 0 INFO org.apache.hadoop.mapred.Task [Thread-11]: Task:attempt_local_0001_m_00000 0_0 is done. And is in the process of commiting INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: file:/c:/upb/dp/manch lia-dp/depot/services/data-platform/trunk/analytics/geoinput/geo.dat:0+18 INFO org.apache.hadoop.mapred.Task [Thread-11]: Task 'attempt_local_0001_m_0000 00_0' done. INFO org.apache.hadoop.mapred.Task [Thread-11]: Using ResourceCalculatorPlugin : null INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: INFO org.apache.hadoop.mapred.Merger [Thread-11]: Merging 1 sorted segments INFO org.apache.hadoop.mapred.Merger [Thread-11]: Down to the last merge-pass, with 1 segments left of total size: 26 bytes INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: I nside reduce INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: O utside reduce INFO org.apache.hadoop.mapred.Task [Thread-11]: Task:attempt_local_0001_r_00000 0_0 is done. And is in the process of commiting INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: INFO org.apache.hadoop.mapred.Task [Thread-11]: Task attempt_local_0001_r_00000 0_0 is allowed to commit now INFO org.apache.hadoop.mapred.FileOutputCommitter [Thread-11]: Saved output of task 'attempt_local_0001_r_000000_0' to file:/c:/upb/dp/manchlia-dp/depot/servic es/data-platform/trunk/analytics/geooutput INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: reduce > reduce INFO org.apache.hadoop.mapred.Task [Thread-11]: Task 'attempt_local_0001_r_0000 00_0' done. INFO org.apache.hadoop.mapred.JobClient [main]: map 100% reduce 100% INFO org.apache.hadoop.mapred.JobClient [main]: Job complete: job_local_0001 INFO org.apache.hadoop.mapred.JobClient [main]: Counters: 15 INFO org.apache.hadoop.mapred.JobClient [main]: FileSystemCounters INFO org.apache.hadoop.mapred.JobClient [main]: FILE_BYTES_READ=458 INFO org.apache.hadoop.mapred.JobClient [main]: FILE_BYTES_WRITTEN=96110 INFO org.apache.hadoop.mapred.JobClient [main]: Map-Reduce Framework INFO org.apache.hadoop.mapred.JobClient [main]: Map input records=2 INFO org.apache.hadoop.mapred.JobClient [main]: Reduce shuffle bytes=0 INFO org.apache.hadoop.mapred.JobClient [main]: Spilled Records=4 INFO org.apache.hadoop.mapred.JobClient [main]: Map output bytes=20 INFO org.apache.hadoop.mapred.JobClient [main]: Total committed heap usage (bytes)=321527808 INFO org.apache.hadoop.mapred.JobClient [main]: Map input bytes=18 INFO org.apache.hadoop.mapred.JobClient [main]: SPLIT_RAW_BYTES=142 INFO org.apache.hadoop.mapred.JobClient [main]: Combine input records=0 INFO org.apache.hadoop.mapred.JobClient [main]: Reduce input records=2 INFO org.apache.hadoop.mapred.JobClient [main]: Reduce input groups=1 INFO org.apache.hadoop.mapred.JobClient [main]: Combine output records=0 INFO org.apache.hadoop.mapred.JobClient [main]: Reduce output records=1 INFO org.apache.hadoop.mapred.JobClient [main]: Map output records=2 INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [main]: Inside reduce INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [main]: Outsid e reduce Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.547 sec Results : Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
