See <https://builds.apache.org/job/Mahout-Examples-Cluster-Reuters/354/changes>
Changes:
[gsingers] add some helpers to AbstractJob, add a main to DictionaryVectorizer
to try and isolate some issues in testing DicVec on Hadoop for MAHOUT-1247
------------------------------------------
[...truncated 3431 lines...]
Jun 09, 2013 3:06:34 PM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0001_m_000002_0' done.
Jun 09, 2013 3:06:34 PM org.apache.hadoop.mapred.Task initialize
INFO: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@c361f9c
Jun 09, 2013 3:06:35 PM org.apache.hadoop.mapred.Task done
INFO: Task:attempt_local_0001_m_000003_0 is done. And is in the process of
commiting
Jun 09, 2013 3:06:35 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 09, 2013 3:06:35 PM org.apache.hadoop.mapred.Task commit
INFO: Task attempt_local_0001_m_000003_0 is allowed to commit now
Jun 09, 2013 3:06:35 PM
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
INFO: Saved output of task 'attempt_local_0001_m_000003_0' to
/tmp/mahout-work-jenkins/reuters-out-seqdir-sparse-fkmeans/tokenized-documents
Jun 09, 2013 3:06:35 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 09, 2013 3:06:35 PM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0001_m_000003_0' done.
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Job complete: job_local_0001
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: Counters: 12
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: File Output Format Counters
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: Bytes Written=15194111
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: FileSystemCounters
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: FILE_BYTES_READ=217843640
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: FILE_BYTES_WRITTEN=210684746
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: File Input Format Counters
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: Bytes Read=18547540
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: Map-Reduce Framework
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: Map input records=21578
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: Physical memory (bytes) snapshot=0
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: Spilled Records=0
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: CPU time spent (ms)=0
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: Total committed heap usage (bytes)=1534066688
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: Virtual memory (bytes) snapshot=0
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: Map output records=21578
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Counters log
INFO: SPLIT_RAW_BYTES=484
Jun 09, 2013 3:06:36 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Creating Term Frequency Vectors
Jun 09, 2013 3:06:36 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Creating dictionary from
/tmp/mahout-work-jenkins/reuters-out-seqdir-sparse-fkmeans/tokenized-documents
and saving at
/tmp/mahout-work-jenkins/reuters-out-seqdir-sparse-fkmeans/wordcount
Jun 09, 2013 3:06:36 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Deleting
/tmp/mahout-work-jenkins/reuters-out-seqdir-sparse-fkmeans/wordcount
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapreduce.lib.input.FileInputFormat
listStatus
INFO: Total input paths to process : 4
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Running job: job_local_0002
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.Task initialize
INFO: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3a471b7f
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: io.sort.mb = 100
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: data buffer = 79691776/99614720
Jun 09, 2013 3:06:36 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: record buffer = 262144/327680
Jun 09, 2013 3:06:37 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer collect
INFO: Spilling map output: record full = true
Jun 09, 2013 3:06:37 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
startSpill
INFO: bufstart = 0; bufend = 3827594; bufvoid = 99614720
Jun 09, 2013 3:06:37 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
startSpill
INFO: kvstart = 0; kvend = 262144; length = 327680
Jun 09, 2013 3:06:37 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: map 0% reduce 0%
Jun 09, 2013 3:06:39 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
sortAndSpill
INFO: Finished spill 0
Jun 09, 2013 3:06:39 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
INFO: Starting flush of map output
Jun 09, 2013 3:06:40 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
sortAndSpill
INFO: Finished spill 1
Jun 09, 2013 3:06:40 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Merging 2 sorted segments
Jun 09, 2013 3:06:40 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Down to the last merge-pass, with 2 segments left of total size: 876427
bytes
Jun 09, 2013 3:06:40 PM org.apache.hadoop.mapred.Task done
INFO: Task:attempt_local_0002_m_000000_0 is done. And is in the process of
commiting
Jun 09, 2013 3:06:40 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 09, 2013 3:06:40 PM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0002_m_000000_0' done.
Jun 09, 2013 3:06:40 PM org.apache.hadoop.mapred.Task initialize
INFO: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@643b8f2e
Jun 09, 2013 3:06:40 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: io.sort.mb = 100
Jun 09, 2013 3:06:40 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: data buffer = 79691776/99614720
Jun 09, 2013 3:06:40 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: record buffer = 262144/327680
Jun 09, 2013 3:06:40 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: map 100% reduce 0%
Jun 09, 2013 3:06:41 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer collect
INFO: Spilling map output: record full = true
Jun 09, 2013 3:06:41 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
startSpill
INFO: bufstart = 0; bufend = 3830779; bufvoid = 99614720
Jun 09, 2013 3:06:41 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
startSpill
INFO: kvstart = 0; kvend = 262144; length = 327680
Jun 09, 2013 3:06:42 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
sortAndSpill
INFO: Finished spill 0
Jun 09, 2013 3:06:42 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
INFO: Starting flush of map output
Jun 09, 2013 3:06:43 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
sortAndSpill
INFO: Finished spill 1
Jun 09, 2013 3:06:43 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Merging 2 sorted segments
Jun 09, 2013 3:06:43 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Down to the last merge-pass, with 2 segments left of total size: 872415
bytes
Jun 09, 2013 3:06:43 PM org.apache.hadoop.mapred.Task done
INFO: Task:attempt_local_0002_m_000001_0 is done. And is in the process of
commiting
Jun 09, 2013 3:06:43 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 09, 2013 3:06:43 PM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0002_m_000001_0' done.
Jun 09, 2013 3:06:43 PM org.apache.hadoop.mapred.Task initialize
INFO: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@41963d89
Jun 09, 2013 3:06:43 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: io.sort.mb = 100
Jun 09, 2013 3:06:43 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: data buffer = 79691776/99614720
Jun 09, 2013 3:06:43 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: record buffer = 262144/327680
Jun 09, 2013 3:06:45 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer collect
INFO: Spilling map output: record full = true
Jun 09, 2013 3:06:45 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
startSpill
INFO: bufstart = 0; bufend = 3825666; bufvoid = 99614720
Jun 09, 2013 3:06:45 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
startSpill
INFO: kvstart = 0; kvend = 262144; length = 327680
Jun 09, 2013 3:06:45 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
sortAndSpill
INFO: Finished spill 0
Jun 09, 2013 3:06:46 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
INFO: Starting flush of map output
Jun 09, 2013 3:06:46 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
sortAndSpill
INFO: Finished spill 1
Jun 09, 2013 3:06:46 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Merging 2 sorted segments
Jun 09, 2013 3:06:46 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Down to the last merge-pass, with 2 segments left of total size: 889530
bytes
Jun 09, 2013 3:06:46 PM org.apache.hadoop.mapred.Task done
INFO: Task:attempt_local_0002_m_000002_0 is done. And is in the process of
commiting
Jun 09, 2013 3:06:46 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 09, 2013 3:06:46 PM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0002_m_000002_0' done.
Jun 09, 2013 3:06:46 PM org.apache.hadoop.mapred.Task initialize
INFO: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@64704189
Jun 09, 2013 3:06:46 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: io.sort.mb = 100
Jun 09, 2013 3:06:46 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: data buffer = 79691776/99614720
Jun 09, 2013 3:06:46 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: record buffer = 262144/327680
Jun 09, 2013 3:06:46 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer collect
INFO: Spilling map output: record full = true
Jun 09, 2013 3:06:46 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
startSpill
INFO: bufstart = 0; bufend = 3825152; bufvoid = 99614720
Jun 09, 2013 3:06:46 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
startSpill
INFO: kvstart = 0; kvend = 262144; length = 327680
Jun 09, 2013 3:06:47 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
INFO: Starting flush of map output
Jun 09, 2013 3:06:47 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
sortAndSpill
INFO: Finished spill 0
Jun 09, 2013 3:06:47 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
sortAndSpill
INFO: Finished spill 1
Jun 09, 2013 3:06:47 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Merging 2 sorted segments
Jun 09, 2013 3:06:47 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Down to the last merge-pass, with 2 segments left of total size: 718536
bytes
Jun 09, 2013 3:06:47 PM org.apache.hadoop.mapred.Task done
INFO: Task:attempt_local_0002_m_000003_0 is done. And is in the process of
commiting
Jun 09, 2013 3:06:47 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 09, 2013 3:06:47 PM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0002_m_000003_0' done.
Jun 09, 2013 3:06:47 PM org.apache.hadoop.mapred.LocalJobRunner$Job run
WARNING: job_local_0002
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
output/file.out in any of the configured local directories
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at
org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:238)
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Job complete: job_local_0002
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: Counters: 15
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: FileSystemCounters
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: FILE_BYTES_READ=461815569
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: FILE_BYTES_WRITTEN=420736374
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: File Input Format Counters
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: Bytes Read=15194111
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: Map-Reduce Framework
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: Map output materialized bytes=3356916
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: Combine output records=190548
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: Map input records=21578
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: Physical memory (bytes) snapshot=0
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: Spilled Records=381096
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: Map output bytes=22501851
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: CPU time spent (ms)=0
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: Total committed heap usage (bytes)=3047030784
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: Virtual memory (bytes) snapshot=0
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: Combine input records=1540960
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: Map output records=1540960
Jun 09, 2013 3:06:48 PM org.apache.hadoop.mapred.Counters log
INFO: SPLIT_RAW_BYTES=644
Exception in thread "main" java.lang.IllegalStateException: Job failed!
at
org.apache.mahout.vectorizer.DictionaryVectorizer.startWordCounting(DictionaryVectorizer.java:368)
at
org.apache.mahout.vectorizer.DictionaryVectorizer.createTermFrequencyVectors(DictionaryVectorizer.java:179)
at
org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.run(SparseVectorsFromSequenceFiles.java:273)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at
org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.main(SparseVectorsFromSequenceFiles.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195)
Build step 'Execute shell' marked build as failure
