[jira] [Updated] (MAPREDUCE-5893) CBZip2InputStream is not threadsafe

2014-05-16 Thread Xiangrui Meng (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiangrui Meng updated MAPREDUCE-5893:
-

Assignee: Xiangrui Meng
Target Version/s: 2.2.1, 1.2.2
  Status: Patch Available  (was: Open)

Changed `skipCompression` to `private` from `private static`.
Added a private constructor to set `skipCompression`.

> CBZip2InputStream is not threadsafe
> ---
>
> Key: MAPREDUCE-5893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5893
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv1, mrv2
>Affects Versions: 2.2.0, 1.2.1
>Reporter: Xiangrui Meng
>Assignee: Xiangrui Meng
> Attachments: bzip2.diff
>
>
> Hadoop uses CBZip2InputStream to decode bzip2 files. However, the 
> implementation is not threadsafe. This is not a really problem for Hadoop 
> MapReduce because Hadoop runs each task in a separate JVM. But for other 
> libraries that utilize multithreading and use Hadoop's InputFormat, e.g., 
> Spark, it will cause exceptions like the following:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 6 
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.recvDecodingTables(CBZip2InputStream.java:729)
>  
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.getAndMoveToFrontDecode(CBZip2InputStream.java:795)
>  
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock(CBZip2InputStream.java:499)
>  
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.changeStateToProcessABlock(CBZip2InputStream.java:330)
>  
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.read(CBZip2InputStream.java:394)
>  
> org.apache.hadoop.io.compress.BZip2Codec$BZip2CompressionInputStream.read(BZip2Codec.java:428)
>  java.io.InputStream.read(InputStream.java:101) 
> org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:205) 
> org.apache.hadoop.util.LineReader.readLine(LineReader.java:169) 
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:176) 
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:43) 
> org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:198) 
> org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:181) 
> org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) 
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:35)
>  scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) 
> org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1000) 
> org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847) 
> org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847) 
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077)
>  
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077)
>  org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111) 
> org.apache.spark.scheduler.Task.run(Task.scala:51) 
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  java.lang.Thread.run(Thread.java:724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (MAPREDUCE-5893) CBZip2InputStream is not threadsafe

2014-05-16 Thread Xiangrui Meng (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiangrui Meng updated MAPREDUCE-5893:
-

Attachment: bzip2.diff

> CBZip2InputStream is not threadsafe
> ---
>
> Key: MAPREDUCE-5893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5893
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv1, mrv2
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Xiangrui Meng
> Attachments: bzip2.diff
>
>
> Hadoop uses CBZip2InputStream to decode bzip2 files. However, the 
> implementation is not threadsafe. This is not a really problem for Hadoop 
> MapReduce because Hadoop runs each task in a separate JVM. But for other 
> libraries that utilize multithreading and use Hadoop's InputFormat, e.g., 
> Spark, it will cause exceptions like the following:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 6 
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.recvDecodingTables(CBZip2InputStream.java:729)
>  
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.getAndMoveToFrontDecode(CBZip2InputStream.java:795)
>  
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock(CBZip2InputStream.java:499)
>  
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.changeStateToProcessABlock(CBZip2InputStream.java:330)
>  
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.read(CBZip2InputStream.java:394)
>  
> org.apache.hadoop.io.compress.BZip2Codec$BZip2CompressionInputStream.read(BZip2Codec.java:428)
>  java.io.InputStream.read(InputStream.java:101) 
> org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:205) 
> org.apache.hadoop.util.LineReader.readLine(LineReader.java:169) 
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:176) 
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:43) 
> org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:198) 
> org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:181) 
> org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) 
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:35)
>  scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) 
> org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1000) 
> org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847) 
> org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847) 
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077)
>  
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077)
>  org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111) 
> org.apache.spark.scheduler.Task.run(Task.scala:51) 
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  java.lang.Thread.run(Thread.java:724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (MAPREDUCE-5893) CBZip2InputStream is not threadsafe

2014-05-16 Thread Xiangrui Meng (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000652#comment-14000652
 ] 

Xiangrui Meng commented on MAPREDUCE-5893:
--

Checked the code in the trunk. This class has a static boolean member 
`skipDecompression`, which indicates whether it is decompressing or checking 
the next marker.

> CBZip2InputStream is not threadsafe
> ---
>
> Key: MAPREDUCE-5893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5893
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv1, mrv2
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Xiangrui Meng
>
> Hadoop uses CBZip2InputStream to decode bzip2 files. However, the 
> implementation is not threadsafe. This is not a really problem for Hadoop 
> MapReduce because Hadoop runs each task in a separate JVM. But for other 
> libraries that utilize multithreading and use Hadoop's InputFormat, e.g., 
> Spark, it will cause exceptions like the following:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 6 
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.recvDecodingTables(CBZip2InputStream.java:729)
>  
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.getAndMoveToFrontDecode(CBZip2InputStream.java:795)
>  
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock(CBZip2InputStream.java:499)
>  
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.changeStateToProcessABlock(CBZip2InputStream.java:330)
>  
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.read(CBZip2InputStream.java:394)
>  
> org.apache.hadoop.io.compress.BZip2Codec$BZip2CompressionInputStream.read(BZip2Codec.java:428)
>  java.io.InputStream.read(InputStream.java:101) 
> org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:205) 
> org.apache.hadoop.util.LineReader.readLine(LineReader.java:169) 
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:176) 
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:43) 
> org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:198) 
> org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:181) 
> org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) 
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:35)
>  scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) 
> org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1000) 
> org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847) 
> org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847) 
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077)
>  
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077)
>  org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111) 
> org.apache.spark.scheduler.Task.run(Task.scala:51) 
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  java.lang.Thread.run(Thread.java:724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (MAPREDUCE-5893) CBZip2InputStream is not threadsafe

2014-05-16 Thread Xiangrui Meng (JIRA) 
Xiangrui Meng created MAPREDUCE-5893:


 Summary: CBZip2InputStream is not threadsafe
 Key: MAPREDUCE-5893
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5893
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1, mrv2
Affects Versions: 2.2.0, 1.2.1
Reporter: Xiangrui Meng


Hadoop uses CBZip2InputStream to decode bzip2 files. However, the 
implementation is not threadsafe. This is not a really problem for Hadoop 
MapReduce because Hadoop runs each task in a separate JVM. But for other 
libraries that utilize multithreading and use Hadoop's InputFormat, e.g., 
Spark, it will cause exceptions like the following:

{code}
java.lang.ArrayIndexOutOfBoundsException: 6 
org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.recvDecodingTables(CBZip2InputStream.java:729)
 
org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.getAndMoveToFrontDecode(CBZip2InputStream.java:795)
 
org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock(CBZip2InputStream.java:499)
 
org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.changeStateToProcessABlock(CBZip2InputStream.java:330)
 
org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.read(CBZip2InputStream.java:394)
 
org.apache.hadoop.io.compress.BZip2Codec$BZip2CompressionInputStream.read(BZip2Codec.java:428)
 java.io.InputStream.read(InputStream.java:101) 
org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:205) 
org.apache.hadoop.util.LineReader.readLine(LineReader.java:169) 
org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:176) 
org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:43) 
org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:198) 
org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:181) 
org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) 
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:35) 
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) 
org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1000) 
org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847) 
org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847) 
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077) 
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077) 
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111) 
org.apache.spark.scheduler.Task.run(Task.scala:51) 
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
java.lang.Thread.run(Thread.java:724)
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)