meng xi created SPARK-21025:
-------------------------------
Summary: missing data in jsc.union
Key: SPARK-21025
URL: https://issues.apache.org/jira/browse/SPARK-21025
Project: Spark
Issue Type: Bug
Components: Java API
Affects Versions: 2.1.1, 2.1.0
Environment: Ubuntu 16.04
Reporter: meng xi
we are using an iterator of RDD for some special data processing, and then
using union to rebuild a new RDD. we found the result RDD are often empty or
missing most of the data. Here is a simplified code snippet for this bug:
SparkConf sparkConf = new
SparkConf().setAppName("Test").setMaster("local[*]");
SparkContext sparkContext = SparkContext.getOrCreate(sparkConf);
JavaSparkContext jsc = JavaSparkContext.fromSparkContext(sparkContext);
JavaRDD<String[]> src = jsc.parallelize(IntStream.range(0,
3000).mapToObj(i -> new String[10]).collect(Collectors.toList()));
Iterator<String[]> it = src.toLocalIterator();
List<JavaRDD<String[]>> rddList = new LinkedList<>();
List<String[]> resultBuffer = new LinkedList<>();
while (it.hasNext()) {
resultBuffer.add(it.next());
if (resultBuffer.size() == 1000) {
JavaRDD<String[]> rdd = jsc.parallelize(resultBuffer);
// rdd.count();
rddList.add(rdd);
resultBuffer.clear();
}
}
JavaRDD<String[]> desc = jsc.union(jsc.parallelize(resultBuffer),
rddList);
System.out.println(desc.count());
this code should duplicate the original RDD, but it just returns an empty RDD.
Please note that if I uncomment the rdd.count, it will return the correct
result.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
