Hi
1.I am using spark streaming 1.3 for reading from a kafka queue and pushing
events to external source.
I passed in my job 20 executors but it is showing only 6 in executor tab ?
When I used highlevel streaming 1.2 - its showing 20 executors. My cluster
is 10 node yarn cluster with each node has 8 cores.
I am calling the script as :
spark-submit --class classname --num-executors 10 --executor-cores 2
--master yarn-client jarfile
2. On Streaming UI
Started at: Mon Jul 20 11:02:10 GMT+00:00 2015
Time since start: 13 minutes 28 seconds
Network receivers: 0
Batch interval: 1 second
Processed batches: 807
Waiting batches: 0
Received records: 0
Processed records: 0
Received records and processed records are always 0 . And Speed of
processing is slow compare to highlevel api.
I am procesing the stream using mapPartition.
When I used
directKafkaStream.foreachRDD(new Function<JavaPairRDD<byte[],byte[]>,
Void>() {
@Override
public Void call(JavaPairRDD<byte[], byte[]> rdd) throws Exception {
// TODO Auto-generated method stub
OffsetRange[] offsetRanges = ((HasOffsetRanges)rdd).offsetRanges();
}
}
It throws an exception
java.lang.ClassCastException: org.apache.spark.api.java.JavaPairRDD cannot
be cast to org.apache.spark.streaming.kafka.HasOffsetRanges
Thanks
Shushant
