Are u running vanilla Mahout 0.9 on Hadoop 2x? While that may not be the
issue here, Mahout 0.9 doesn't support Hadoop 2x yet. Its better if u
could work against the present trunk and build the code with hadoop 2
profile if that's ur target test bed.
On Sat, Jul 12, 2014 at 11:38 AM, Reinis Vicups <mah...@orbit-x.de> wrote:
> Hi,
>
> the log bellow shows an issue that started to occur just "recently" (I
> haven't ran tests with this somewhat larger dataset (320K documents) for
> some time and when I did today, I got this "all of a sudden").
> Am using mahout 0.9-cdh5.2.0-SNAPSHOT (yes its cloudera but as far as I
> can tell, that's vanilla mahout in the community edition I use).
>
> As far as I can tell, it's happening in the middle of seq2sparse and all
> three - the input, the output and the mr-job are being generated by mahout
> and there's no my code involved.
>
> It would be cool if anyone could point me to the source of this error.
>
> thanks and kind regards
> reinis.
>
> SETTINGS OF SEQ2SPARSE
> ----------------------------------------------
>
> {"--analyzerName", "com.myproj.quantify.ticket.text.TicketTextAnalyzer",
> "--chunkSize", "200",
> "--output", finalDir,
> "--input", ticketTextsOutput.toString,
> "--minSupport", "2",
> "--minDF", "2",
> "--maxDFPercent", "85",
> "--weight", "tfidf",
> "--minLLR", "50",
> "--maxNGramSize", "3",
> "--norm", "2",
> "--namedVector", "--sequentialAccessVector", "--overwrite"}
>
>
> LOG
> -----------------------------------------------------
>
> 14/07/12 16:46:16 INFO vectorizer.SparseVectorsFromSequenceFiles:
> Creating Term Frequency Vectors
> 14/07/12 16:46:16 INFO vectorizer.DictionaryVectorizer: Creating
> dictionary from /quantify/ticket/text/final/tokenized-documents and
> saving at /quantify/ticket/text/final/wordcount
> 14/07/12 16:46:16 INFO client.RMProxy: Connecting to ResourceManager at
> hadoop1
> 14/07/12 16:46:17 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/07/12 16:46:17 INFO mapreduce.JobSubmitter: number of splits:2
> 14/07/12 16:46:17 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1404888747437_0074
> 14/07/12 16:46:17 INFO impl.YarnClientImpl: Submitted application
> application_1404888747437_0074
> 14/07/12 16:46:17 INFO mapreduce.Job: The url to track the job:
> http://hadoop1:8088/proxy/application_1404888747437_0074/
> 14/07/12 16:46:17 INFO mapreduce.Job: Running job: job_1404888747437_0074
> 14/07/12 16:46:30 INFO mapreduce.Job: Job job_1404888747437_0074 running
> in uber mode : false
> 14/07/12 16:46:30 INFO mapreduce.Job: map 0% reduce 0%
> 14/07/12 16:46:41 INFO mapreduce.Job: map 6% reduce 0%
> 14/07/12 16:46:44 INFO mapreduce.Job: map 10% reduce 0%
> 14/07/12 16:46:47 INFO mapreduce.Job: map 11% reduce 0%
> 14/07/12 16:46:48 INFO mapreduce.Job: map 14% reduce 0%
> 14/07/12 16:46:50 INFO mapreduce.Job: map 15% reduce 0%
> 14/07/12 16:46:51 INFO mapreduce.Job: map 19% reduce 0%
> 14/07/12 16:46:53 INFO mapreduce.Job: map 20% reduce 0%
> 14/07/12 16:46:54 INFO mapreduce.Job: map 23% reduce 0%
> 14/07/12 16:46:57 INFO mapreduce.Job: map 26% reduce 0%
> 14/07/12 16:47:00 INFO mapreduce.Job: map 29% reduce 0%
> 14/07/12 16:47:01 INFO mapreduce.Job: Task Id :
> attempt_1404888747437_0074_m_000000_0, Status : FAILED
> Error: java.lang.IllegalStateException: java.io.IOException: Spill failed
> at org.apache.mahout.vectorizer.collocations.llr.CollocMapper$
> 1.apply(CollocMapper.java:140)
> at org.apache.mahout.vectorizer.collocations.llr.CollocMapper$
> 1.apply(CollocMapper.java:115)
> at org.apache.mahout.math.map.OpenObjectIntHashMap.forEachPair(
> OpenObjectIntHashMap.java:185)
> at org.apache.mahout.vectorizer.collocations.llr.CollocMapper.
> map(CollocMapper.java:115)
> at org.apache.mahout.vectorizer.collocations.llr.CollocMapper.
> map(CollocMapper.java:41)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.io.IOException: Spill failed
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.
> checkSpillException(MapTask.java:1535)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.
> access$300(MapTask.java:853)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$
> Buffer.write(MapTask.java:1349)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at org.apache.mahout.vectorizer.collocations.llr.GramKey.
> write(GramKey.java:91)
> at org.apache.hadoop.io.serializer.WritableSerialization$
> WritableSerializer.serialize(WritableSerialization.java:98)
> at org.apache.hadoop.io.serializer.WritableSerialization$
> WritableSerializer.serialize(WritableSerialization.java:82)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.
> collect(MapTask.java:1126)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.
> write(MapTask.java:692)
> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.
> write(TaskInputOutputContextImpl.java:89)
> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.
> write(WrappedMapper.java:112)
> at org.apache.mahout.vectorizer.collocations.llr.CollocMapper$
> 1.apply(CollocMapper.java:131)
> ... 12 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1836016430
> at java.io.ByteArrayInputStream.read(ByteArrayInputStream.
> java:144)
> at java.io.DataInputStream.readByte(DataInputStream.java:265)
> at org.apache.mahout.math.Varint.readUnsignedVarInt(Varint.
> java:159)
> at org.apache.mahout.vectorizer.collocations.llr.GramKey.
> readFields(GramKey.java:78)
> at org.apache.hadoop.io.WritableComparator.compare(
> WritableComparator.java:132)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.
> compare(MapTask.java:1245)
> at org.apache.hadoop.util.QuickSort.sortInternal(
> QuickSort.java:105)
> at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.
> sortAndSpill(MapTask.java:1575)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.
> access$900(MapTask.java:853)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$
> SpillThread.run(MapTask.java:1505)
>
