Hi, I'm running Spark Streaming with Kafka Direct Stream, batch interval is 10 seconds. After running about 72 hours, the batch processing time almost doubles. I didn't find anything wrong on JVM GC logs, but I did find that broadcast variable reading time increasing, like this:
initially: ``` 16/06/08 18:17:02 INFO broadcast.TorrentBroadcast: Reading broadcast variable 0 took 223 ms 16/06/08 18:17:08 INFO broadcast.TorrentBroadcast: Reading broadcast variable 2 took 73 ms 16/06/08 18:17:10 INFO broadcast.TorrentBroadcast: Reading broadcast variable 3 took 13 ms 16/06/08 18:17:11 INFO broadcast.TorrentBroadcast: Reading broadcast variable 4 took 9 ms 16/06/08 18:17:12 INFO broadcast.TorrentBroadcast: Reading broadcast variable 5 took 8 ms 16/06/08 18:17:14 INFO broadcast.TorrentBroadcast: Reading broadcast variable 6 took 13 ms 16/06/08 18:17:15 INFO broadcast.TorrentBroadcast: Reading broadcast variable 8 took 7 ms 16/06/08 18:17:16 INFO broadcast.TorrentBroadcast: Reading broadcast variable 10 took 8 ms 16/06/08 18:17:20 INFO broadcast.TorrentBroadcast: Reading broadcast variable 12 took 6 ms 16/06/08 18:17:20 INFO broadcast.TorrentBroadcast: Reading broadcast variable 13 took 7 ms ``` after 23 hours: ``` 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast variable 282938 took 18 ms 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast variable 282939 took 20 ms 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast variable 282940 took 20 ms 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast variable 282941 took 14 ms 16/06/09 17:23:44 INFO broadcast.TorrentBroadcast: Reading broadcast variable 282942 took 17 ms 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast variable 282943 took 18 ms 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast variable 282944 took 14 ms 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast variable 282945 took 18 ms 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast variable 282946 took 14 ms 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast variable 282947 took 18 ms ``` FYI, we're running on AWS EMR with Spark version 1.6.1, in YARN client mode. application environment follows: ``` Java Home /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre Java Version 1.7.0_101 (Oracle Corporation) Scala Version version 2.10.5 spark.app.id application_1463430559850_0056 spark.cleaner.ttl 60 spark.default.parallelism 96 spark.driver.appUIAddress http://172.16.4.168:4041 spark.driver.extraClassPath /etc/hadoop/conf:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* spark.driver.extraJavaOptions -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M -XX:OnOutOfMemoryError='kill -9 %p' spark.driver.extraLibraryPath /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native spark.driver.host 172.16.4.168 spark.driver.port 42142 spark.dynamicAllocation.enabled false spark.eventLog.compress true spark.eventLog.dir hdfs:///var/log/spark/apps spark.eventLog.enabled false spark.executor.cores 8 spark.executor.extraClassPath /etc/hadoop/conf:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p' spark.executor.extraLibraryPath /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native spark.executor.id driver spark.executor.instances 24 spark.executor.memory 6G spark.externalBlockStore.folderName spark-9ede6685-a7f9-4d65-bfb4-0920e8c5ba25 spark.history.fs.cleaner.enabled true spark.history.fs.cleaner.maxAge 2d spark.history.fs.logDirectory hdfs:///var/log/spark/apps spark.history.ui.port 18080 spark.master yarn-client spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS ip-172-16-4-168.ec2.internal spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES http://ip-172-16-4-168.ec2.internal:20888/proxy/application_1463430559850_0056 spark.scheduler.mode FAIR spark.serializer org.apache.spark.serializer.KryoSerializer spark.shuffle.service.enabled false spark.streaming.backpressure.enabled true spark.streaming.concurrentJobs 2 spark.streaming.kafka.maxRatePerPartition 312 spark.streaming.ui.retainedBatches 1000 spark.submit.deployMode client spark.ui.filters org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter spark.ui.retainedJobs 1000 spark.ui.retainedStages 1000 spark.yarn.executor.memoryOverhead 4096 spark.yarn.historyServer.address ip-172-16-4-168.ec2.internal:18080 SPARK_SUBMIT true SPARK_YARN_MODE true awt.toolkit sun.awt.X11.XToolkit file.encoding UTF-8 file.encoding.pkg sun.io file.separator / java.awt.graphicsenv sun.awt.X11GraphicsEnvironment java.awt.printerjob sun.print.PSPrinterJob java.class.version 51.0 java.endorsed.dirs /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/endorsed java.ext.dirs /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/ext:/usr/java/packages/lib/ext java.home /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre java.io.tmpdir /tmp java.library.path /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib java.runtime.name OpenJDK Runtime Environment java.runtime.version 1.7.0_101-mockbuild_2016_04_26_18_10-b00 java.specification.name Java Platform API Specification java.specification.vendor Oracle Corporation java.specification.version 1.7 java.vendor Oracle Corporation java.vendor.url http://java.oracle.com/ java.vendor.url.bug http://bugreport.sun.com/bugreport/ java.version 1.7.0_101 java.vm.info mixed mode java.vm.name OpenJDK 64-Bit Server VM java.vm.specification.name Java Virtual Machine Specification java.vm.specification.vendor Oracle Corporation java.vm.specification.version 1.7 java.vm.vendor Oracle Corporation java.vm.version 24.95-b01 line.separator log4j.configuration file:///etc/spark/conf/log4j.properties os.arch amd64 os.name Linux os.version 4.4.5-15.26.amzn1.x86_64 path.separator : sun.arch.data.model 64 sun.boot.class.path /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/resources.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/rt.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/jsse.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/jce.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/charsets.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/rhino.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/jfr.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/classes sun.boot.library.path /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/amd64 sun.cpu.endian little sun.cpu.isalist sun.io.unicode.encoding UnicodeLittle sun.java.launcher SUN_STANDARD sun.jnu.encoding UTF-8 sun.management.compiler HotSpot 64-Bit Tiered Compilers sun.nio.ch.bugLevel sun.os.patch.level unknown user.country US user.dir /home/hadoop user.home /home/hadoop user.language en user.name hadoop user.timezone UTC ``` -- John Simon --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
