Hi everyone, I m experiencing a problem with my spark streaming job when running it on yarn. The problem appears only when running this application in a Yarn queue along with other Tez/MR applications
This problem is in processing time, which exceeds 1 minute for batches of 1 second. Normally when running my application along with just Spark applications in Yarn queue the processing of each batch will take 0.7 sec at max (for ~200 events/sec). I checked the number of executor and it is not changing during the whole execution of the streaming job Further information about my streaming application: It reads data from Kafka and Kinesis Streams and output data to Kafka topic. batch duration is 1 second I submit the job using the following command (nothing exceptional ) : $ spark-submit --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:executors.log4j.properties" --conf spark.driver.extraClassPath=./conf --num-executors 14 --executor-memory 2g --files conf/app.conf,conf/log4j.properties,conf/executors.log4j.properties --master yarn-client --driver-memory 1G --executor-memory 4g --num-executors 20 --executor-cores 4 --class MainClass MyJar.jar Is there any explanation about this issue? Thanks in advance Amjad
