data size: text file, 315G
cmd:
./spark-submit --class com.spark.test.JavaWordCountWithSave --num-executors 7
--executor-memory 60g --driver-memory 2g --executor-cores 32
--master yarn-client /home/cjs/spark-test.jar hdfs://wordcount/input
hdfs://wordcount/output
code of JavaWordCountWithSave:
```
public final class JavaWordCountWithSave {
private static final Pattern SPACE = Pattern.compile(" ");
public static void main(String[] args) throws Exception {
if (args.length < 2) {
System.err.println("Usage: JavaWordCount <file>");
System.exit(1);
}
SparkConf sparkConf = new
SparkConf().setAppName("JavaWordCountWithSave");
JavaSparkContext ctx = new JavaSparkContext(sparkConf);
JavaRDD<String> lines = ctx.textFile(args[0], 1);
JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String,
String>() {
@Override
public Iterable<String> call(String s) {
return Arrays.asList(SPACE.split(s));
}
});
JavaPairRDD<String, Integer> ones = words.mapToPair(new
PairFunction<String, String, Integer>() {
@Override
public Tuple2<String, Integer> call(String s) {
return new Tuple2<String, Integer>(s, 1);
}
});
JavaPairRDD<String, Integer> counts = ones.reduceByKey(new
Function2<Integer, Integer, Integer>() {
@Override
public Integer call(Integer i1, Integer i2) {
return i1 + i2;
}
});
counts.saveAsTextFile(args[1]);
ctx.stop();
}
}
```
log of driver---
14/11/20 14:57:46 WARN TaskSetManager: Lost task 167.0 in stage 1.0 (TID 5207,
linux-171): ExecutorLostFailure (executor lost)
14/11/20 14:57:46 WARN TaskSetManager: Lost task 41.0 in stage 1.0 (TID 5081,
linux-171): ExecutorLostFailure (executor lost)
14/11/20 14:57:46 WARN TaskSetManager: Lost task 104.0 in stage 1.0 (TID 5144,
linux-171): ExecutorLostFailure (executor lost)
14/11/20 14:57:46 WARN TaskSetManager: Lost task 62.0 in stage 1.0 (TID 5102,
linux-171): ExecutorLostFailure (executor lost)
14/11/20 14:57:46 WARN TaskSetManager: Lost task 20.0 in stage 1.0 (TID 5060,
linux-171): ExecutorLostFailure (executor lost)
14/11/20 14:57:46 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 5
14/11/20 14:57:46 INFO DAGScheduler: Executor lost: 5 (epoch 1)
14/11/20 14:57:46 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 5
14/11/20 14:57:46 INFO BlockManagerMasterActor: Trying to remove executor 5
from BlockManagerMaster.
14/11/20 14:57:46 INFO BlockManagerMaster: Removed 5 successfully in
removeExecutor
14/11/20 14:57:46 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 5
14/11/20 14:57:46 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 5
log of executor---
2014-11-20 14:57:46,879 | INFO | [connection-manager-thread] | key already
cancelled ? sun.nio.ch.SelectionKeyImpl@a6b0591 |
org.apache.spark.Logging$class.logInfo(Logging.scala:80)
java.nio.channels.CancelledKeyException
at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:379)
at
org.apache.spark.network.nio.ConnectionManager$$anon$4.run(ConnectionManager.scala:132)
2014-11-20 14:57:46,958 | INFO | [handle-read-write-executor-3] | Removing
SendingConnection to ConnectionManagerId(172.168.xxx.16,2267) |
org.apache.spark.Logging$class.logInfo(Logging.scala:59)
2014-11-20 14:57:46,963 | INFO | [handle-read-write-executor-3] | Notifying
org.apache.spark.network.nio.ConnectionManager$MessageStatus@272b8b5a |
org.apache.spark.Logging$class.logInfo(Logging.scala:59)
2014-11-20 14:57:46,963 | INFO | [handle-read-write-executor-3] | Notifying
org.apache.spark.network.nio.ConnectionManager$MessageStatus@1bc9d5cd |
org.apache.spark.Logging$class.logInfo(Logging.scala:59)
2014-11-20 14:57:47,107 | ERROR | [Connection manager future execution
context-2] | Failed to get block(s) from 172.168.xxx.16:2267 |
org.apache.spark.Logging$class.logError(Logging.scala:96)
java.io.IOException: sendMessageReliably failed without being ACK'd
at
org.apache.spark.network.nio.ConnectionManager$$anonfun$14.apply(ConnectionManager.scala:822)
at
org.apache.spark.network.nio.ConnectionManager$$anonfun$14.apply(ConnectionManager.scala:818)
at
org.apache.spark.network.nio.ConnectionManager$MessageStatus.markDone(ConnectionManager.scala:61)
at
org.apache.spark.network.nio.ConnectionManager$$anonfun$removeConnection$3.apply(ConnectionManager.scala:451)
at
org.apache.spark.network.nio.ConnectionManager$$anonfun$removeConnection$3.apply(ConnectionManager.scala:449)
at scala.collection.immutable.List.foreach(List.scala:318)
at
org.apache.spark.network.nio.ConnectionManager.removeConnection(ConnectionManager.scala:449)
at
org.apache.spark.network.nio.ConnectionManager$$anonfun$addListeners$3.apply(ConnectionManager.scala:428)
at
org.apache.spark.network.nio.ConnectionManager$$anonfun$addListeners$3.apply(ConnectionManager.scala:428)
at
org.apache.spark.network.nio.Connection.callOnCloseCallback(Connection.scala:152)
at org.apache.spark.network.nio.Connection.close(Connection.scala:124)
at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:414)
at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:192)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-11-20 14:57:47,148 | INFO | [handle-read-write-executor-3] | Notifying
org.apache.spark.network.nio.ConnectionManager$MessageStatus@5f59c2f6 |
org.apache.spark.Logging$class.logInfo(Logging.scala:59)
2014-11-20 14:57:47,149 | ERROR | [Connection manager future execution
context-4] | Failed to get block(s) from 172.168.xxx.16:2267 |
org.apache.spark.Logging$class.logError(Logging.scala:96)
java.io.IOException: sendMessageReliably failed without being ACK'd
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
