See
<https://builds.apache.org/job/Mahout-Examples-Cluster-Reuters-II/522/changes>
Changes:
[smarthi] MAHOUT-944: lucene2seq - more code cleanup, removed unused imports
[smarthi] MAHOUT-833: Make conversion to sequence files map-reduce - fixed
issue with not reading a directory list
[smarthi] MAHOUT-833: Make conversion to sequence files map-reduce - first
round of Code cleanup based on feedback from code review
[smarthi] MAHOUT-944:lucene2seq - removed unused import
------------------------------------------
[...truncated 5861 lines...]
INFO: Starting flush of map output
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
sortAndSpill
INFO: Finished spill 0
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Task done
INFO: Task:attempt_local_0019_m_000000_0 is done. And is in the process of
commiting
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0019_m_000000_0' done.
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Task initialize
INFO: Using ResourceCalculatorPlugin : null
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Merging 1 sorted segments
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Down to the last merge-pass, with 1 segments left of total size: 57 bytes
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Task done
INFO: Task:attempt_local_0019_r_000000_0 is done. And is in the process of
commiting
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Task commit
INFO: Task attempt_local_0019_r_000000_0 is allowed to commit now
Jun 24, 2013 6:35:28 PM
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
INFO: Saved output of task 'attempt_local_0019_r_000000_0' to
/tmp/mahout-work-hudson/reuters-lda-model/model-19
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO: reduce > reduce
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0019_r_000000_0' done.
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: map 100% reduce 100%
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Job complete: job_local_0019
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Counters: 17
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: File Output Format Counters
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Bytes Written=389
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: FileSystemCounters
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: FILE_BYTES_READ=1614631021
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: FILE_BYTES_WRITTEN=1629239107
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: File Input Format Counters
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Bytes Read=152
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Map-Reduce Framework
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Map output materialized bytes=61
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Map input records=0
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce shuffle bytes=0
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Spilled Records=40
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Map output bytes=120
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Total committed heap usage (bytes)=697171968
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: SPLIT_RAW_BYTES=119
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Combine input records=20
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce input records=20
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce input groups=20
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Combine output records=20
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce output records=20
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapred.Counters log
INFO: Map output records=20
Jun 24, 2013 6:35:28 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: About to run iteration 20 of 20
Jun 24, 2013 6:35:28 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: About to run: Iteration 20 of 20, input path:
/tmp/mahout-work-hudson/reuters-lda-model/model-19
Jun 24, 2013 6:35:28 PM org.apache.hadoop.mapreduce.lib.input.FileInputFormat
listStatus
INFO: Total input paths to process : 1
Jun 24, 2013 6:35:29 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Running job: job_local_0020
Jun 24, 2013 6:35:29 PM org.apache.hadoop.mapred.Task initialize
INFO: Using ResourceCalculatorPlugin : null
Jun 24, 2013 6:35:29 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: io.sort.mb = 100
Jun 24, 2013 6:35:29 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: data buffer = 79691776/99614720
Jun 24, 2013 6:35:29 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
INFO: record buffer = 262144/327680
Jun 24, 2013 6:35:29 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Retrieving configuration
Jun 24, 2013 6:35:29 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Initializing read model
Jun 24, 2013 6:35:29 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Initializing write model
Jun 24, 2013 6:35:29 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Initializing model trainer
Jun 24, 2013 6:35:29 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Starting training threadpool with 4 threads
Jun 24, 2013 6:35:29 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Stopping model trainer
Jun 24, 2013 6:35:29 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Initiating stopping of training threadpool
Jun 24, 2013 6:35:29 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: threadpool took: 0.483079ms
Jun 24, 2013 6:35:30 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: map 0% reduce 0%
Jun 24, 2013 6:35:30 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: readModel.stop() took 1001.57073ms
Jun 24, 2013 6:35:31 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: writeModel.stop() took 1010.060325ms
Jun 24, 2013 6:35:31 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Writing model
Jun 24, 2013 6:35:31 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
INFO: Starting flush of map output
Jun 24, 2013 6:35:31 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer
sortAndSpill
INFO: Finished spill 0
Jun 24, 2013 6:35:31 PM org.apache.hadoop.mapred.Task done
INFO: Task:attempt_local_0020_m_000000_0 is done. And is in the process of
commiting
Jun 24, 2013 6:35:31 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 24, 2013 6:35:31 PM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0020_m_000000_0' done.
Jun 24, 2013 6:35:31 PM org.apache.hadoop.mapred.Task initialize
INFO: Using ResourceCalculatorPlugin : null
Jun 24, 2013 6:35:31 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 24, 2013 6:35:31 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Merging 1 sorted segments
Jun 24, 2013 6:35:31 PM org.apache.hadoop.mapred.Merger$MergeQueue merge
INFO: Down to the last merge-pass, with 1 segments left of total size: 57 bytes
Jun 24, 2013 6:35:31 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 24, 2013 6:35:31 PM org.apache.hadoop.mapred.Task done
INFO: Task:attempt_local_0020_r_000000_0 is done. And is in the process of
commiting
Jun 24, 2013 6:35:31 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 24, 2013 6:35:31 PM org.apache.hadoop.mapred.Task commit
INFO: Task attempt_local_0020_r_000000_0 is allowed to commit now
Jun 24, 2013 6:35:31 PM
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
INFO: Saved output of task 'attempt_local_0020_r_000000_0' to
/tmp/mahout-work-hudson/reuters-lda-model/model-20
Jun 24, 2013 6:35:31 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO: reduce > reduce
Jun 24, 2013 6:35:31 PM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0020_r_000000_0' done.
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: map 100% reduce 100%
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
INFO: Job complete: job_local_0020
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Counters: 17
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: File Output Format Counters
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Bytes Written=389
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: FileSystemCounters
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: FILE_BYTES_READ=1699611635
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: FILE_BYTES_WRITTEN=1714988583
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: File Input Format Counters
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Bytes Read=152
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Map-Reduce Framework
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Map output materialized bytes=61
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Map input records=0
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce shuffle bytes=0
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Spilled Records=40
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Map output bytes=120
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Total committed heap usage (bytes)=697171968
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: SPLIT_RAW_BYTES=119
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Combine input records=20
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce input records=20
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce input groups=20
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Combine output records=20
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Reduce output records=20
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Counters log
INFO: Map output records=20
Jun 24, 2013 6:35:32 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Completed 20 iterations in 79 seconds
Jun 24, 2013 6:35:32 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: Perplexities: ()
Jun 24, 2013 6:35:32 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: About to run: Writing final topic/term distributions from
/tmp/mahout-work-hudson/reuters-lda-model/model-20 to
/tmp/mahout-work-hudson/reuters-lda
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapreduce.lib.input.FileInputFormat
listStatus
INFO: Total input paths to process : 1
Jun 24, 2013 6:35:32 PM org.slf4j.impl.JCLLoggerAdapter info
INFO: About to run: Writing final document/topic inference from
/tmp/mahout-work-hudson/reuters-out-matrix/matrix to
/tmp/mahout-work-hudson/reuters-lda-topics
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Task initialize
INFO: Using ResourceCalculatorPlugin : null
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Task done
INFO: Task:attempt_local_0021_m_000000_0 is done. And is in the process of
commiting
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Task commit
INFO: Task attempt_local_0021_m_000000_0 is allowed to commit now
Jun 24, 2013 6:35:32 PM
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
INFO: Saved output of task 'attempt_local_0021_m_000000_0' to
/tmp/mahout-work-hudson/reuters-lda
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
INFO:
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.Task sendDone
INFO: Task 'attempt_local_0021_m_000000_0' done.
Jun 24, 2013 6:35:32 PM org.apache.hadoop.mapred.JobClient$2 run
INFO: Cleaning up the staging area
file:/tmp/hadoop-hudson/mapred/staging/hudson1905787494/.staging/job_local_0022
Jun 24, 2013 6:35:32 PM org.apache.hadoop.security.UserGroupInformation doAs
SEVERE: PriviledgedActionException as:hudson
cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory
/tmp/mahout-work-hudson/reuters-lda already exists
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException:
Output directory /tmp/mahout-work-hudson/reuters-lda already exists
at
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:949)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
at
org.apache.mahout.clustering.lda.cvb.CVB0Driver.writeDocTopicInference(CVB0Driver.java:463)
at
org.apache.mahout.clustering.lda.cvb.CVB0Driver.run(CVB0Driver.java:339)
at
org.apache.mahout.clustering.lda.cvb.CVB0Driver.run(CVB0Driver.java:198)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.mahout.clustering.lda.cvb.CVB0Driver.main(CVB0Driver.java:534)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:194)
Build step 'Execute shell' marked build as failure
