Sorry, forgot to mention that I don't use broadcast variables. That's why I'm puzzled here. -- John Simon
On Thu, Jun 9, 2016 at 11:09 AM, John Simon <john.si...@tapjoy.com> wrote: > Hi, > > I'm running Spark Streaming with Kafka Direct Stream, batch interval > is 10 seconds. > After running about 72 hours, the batch processing time almost doubles. > I didn't find anything wrong on JVM GC logs, but I did find that > broadcast variable reading time increasing, like this: > > initially: > > ``` > 16/06/08 18:17:02 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 0 took 223 ms > 16/06/08 18:17:08 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 2 took 73 ms > 16/06/08 18:17:10 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 3 took 13 ms > 16/06/08 18:17:11 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 4 took 9 ms > 16/06/08 18:17:12 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 5 took 8 ms > 16/06/08 18:17:14 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 6 took 13 ms > 16/06/08 18:17:15 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 8 took 7 ms > 16/06/08 18:17:16 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 10 took 8 ms > 16/06/08 18:17:20 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 12 took 6 ms > 16/06/08 18:17:20 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 13 took 7 ms > ``` > > after 23 hours: > > ``` > 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282938 took 18 ms > 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282939 took 20 ms > 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282940 took 20 ms > 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282941 took 14 ms > 16/06/09 17:23:44 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282942 took 17 ms > 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282943 took 18 ms > 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282944 took 14 ms > 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282945 took 18 ms > 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282946 took 14 ms > 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282947 took 18 ms > ``` > > FYI, we're running on AWS EMR with Spark version 1.6.1, in YARN client mode. > > application environment follows: > > ``` > Java Home /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre > Java Version 1.7.0_101 (Oracle Corporation) > Scala Version version 2.10.5 > > spark.app.id application_1463430559850_0056 > spark.cleaner.ttl 60 > spark.default.parallelism 96 > spark.driver.appUIAddress http://172.16.4.168:4041 > spark.driver.extraClassPath > /etc/hadoop/conf:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* > spark.driver.extraJavaOptions > -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties > -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 > -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled > -XX:MaxPermSize=512M -XX:OnOutOfMemoryError='kill -9 %p' > spark.driver.extraLibraryPath > /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native > spark.driver.host 172.16.4.168 > spark.driver.port 42142 > spark.dynamicAllocation.enabled false > spark.eventLog.compress true > spark.eventLog.dir hdfs:///var/log/spark/apps > spark.eventLog.enabled false > spark.executor.cores 8 > spark.executor.extraClassPath > /etc/hadoop/conf:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* > spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC > -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 > -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p' > spark.executor.extraLibraryPath > /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native > spark.executor.id driver > spark.executor.instances 24 > spark.executor.memory 6G > spark.externalBlockStore.folderName spark-9ede6685-a7f9-4d65-bfb4-0920e8c5ba25 > spark.history.fs.cleaner.enabled true > spark.history.fs.cleaner.maxAge 2d > spark.history.fs.logDirectory hdfs:///var/log/spark/apps > spark.history.ui.port 18080 > spark.master yarn-client > spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS > ip-172-16-4-168.ec2.internal > spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES > http://ip-172-16-4-168.ec2.internal:20888/proxy/application_1463430559850_0056 > spark.scheduler.mode FAIR > spark.serializer org.apache.spark.serializer.KryoSerializer > spark.shuffle.service.enabled false > spark.streaming.backpressure.enabled true > spark.streaming.concurrentJobs 2 > spark.streaming.kafka.maxRatePerPartition 312 > spark.streaming.ui.retainedBatches 1000 > spark.submit.deployMode client > spark.ui.filters org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter > spark.ui.retainedJobs 1000 > spark.ui.retainedStages 1000 > spark.yarn.executor.memoryOverhead 4096 > spark.yarn.historyServer.address ip-172-16-4-168.ec2.internal:18080 > > SPARK_SUBMIT true > SPARK_YARN_MODE true > awt.toolkit sun.awt.X11.XToolkit > file.encoding UTF-8 > file.encoding.pkg sun.io > file.separator / > java.awt.graphicsenv sun.awt.X11GraphicsEnvironment > java.awt.printerjob sun.print.PSPrinterJob > java.class.version 51.0 > java.endorsed.dirs > /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/endorsed > java.ext.dirs > /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/ext:/usr/java/packages/lib/ext > java.home /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre > java.io.tmpdir /tmp > java.library.path > /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > java.runtime.name OpenJDK Runtime Environment > java.runtime.version 1.7.0_101-mockbuild_2016_04_26_18_10-b00 > java.specification.name Java Platform API Specification > java.specification.vendor Oracle Corporation > java.specification.version 1.7 > java.vendor Oracle Corporation > java.vendor.url http://java.oracle.com/ > java.vendor.url.bug http://bugreport.sun.com/bugreport/ > java.version 1.7.0_101 > java.vm.info mixed mode > java.vm.name OpenJDK 64-Bit Server VM > java.vm.specification.name Java Virtual Machine Specification > java.vm.specification.vendor Oracle Corporation > java.vm.specification.version 1.7 > java.vm.vendor Oracle Corporation > java.vm.version 24.95-b01 > line.separator > log4j.configuration file:///etc/spark/conf/log4j.properties > os.arch amd64 > os.name Linux > os.version 4.4.5-15.26.amzn1.x86_64 > path.separator : > sun.arch.data.model 64 > sun.boot.class.path > /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/resources.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/rt.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/jsse.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/jce.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/charsets.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/rhino.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/jfr.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/classes > sun.boot.library.path > /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/amd64 > sun.cpu.endian little > sun.cpu.isalist > sun.io.unicode.encoding UnicodeLittle > sun.java.launcher SUN_STANDARD > sun.jnu.encoding UTF-8 > sun.management.compiler HotSpot 64-Bit Tiered Compilers > sun.nio.ch.bugLevel > sun.os.patch.level unknown > user.country US > user.dir /home/hadoop > user.home /home/hadoop > user.language en > user.name hadoop > user.timezone UTC > ``` > > > -- > John Simon --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org