[ https://issues.apache.org/jira/browse/SPARK-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen resolved SPARK-4133. ------------------------------- Resolution: Fixed I'm going to close this issue for now, since I think this was caused by multiple SparkContexts and not a Spark bug, per-se. Please comment here or open a new issue if you see {{PARSING_ERROR(2)}} on newer versions of Spark without creating multiple active SparkContexts. Note that {{FAILED_TO_UNCOMPRESS(5)}} is a distinct issue, which is being addressed at SPARK-4105. That issue is still open because we've seen reports of it in newer versions. > PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0 > ---------------------------------------------------------- > > Key: SPARK-4133 > URL: https://issues.apache.org/jira/browse/SPARK-4133 > Project: Spark > Issue Type: Bug > Components: Streaming > Affects Versions: 1.1.0 > Reporter: Antonio Jesus Navarro > Attachments: spark_ex.logs > > > Snappy related problems found when trying to upgrade existing Spark Streaming > App from 1.0.2 to 1.1.0. > We can not run an existing 1.0.2 spark app if upgraded to 1.1.0 > > IOException is thrown by snappy (parsing_error(2)) > {code} > Executor task launch worker-0 DEBUG storage.BlockManager - Getting local > block broadcast_0 > Executor task launch worker-0 DEBUG storage.BlockManager - Level for block > broadcast_0 is StorageLevel(true, true, false, true, 1) > Executor task launch worker-0 DEBUG storage.BlockManager - Getting block > broadcast_0 from memory > Executor task launch worker-0 DEBUG storage.BlockManager - Getting local > block broadcast_0 > Executor task launch worker-0 DEBUG executor.Executor - Task 0's epoch is 0 > Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0 > not registered locally > Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started > reading broadcast variable 0 > sparkDriver-akka.actor.default-dispatcher-4 INFO > receiver.ReceiverSupervisorImpl - Registered receiver 0 > Executor task launch worker-0 INFO util.RecurringTimer - Started timer for > BlockGenerator at time 1414656492400 > Executor task launch worker-0 INFO receiver.BlockGenerator - Started > BlockGenerator > Thread-87 INFO receiver.BlockGenerator - Started block pushing thread > Executor task launch worker-0 INFO receiver.ReceiverSupervisorImpl - > Starting receiver > sparkDriver-akka.actor.default-dispatcher-5 INFO scheduler.ReceiverTracker - > Registered receiver for stream 0 from akka://sparkDriver > Executor task launch worker-0 INFO kafka.KafkaReceiver - Starting Kafka > Consumer Stream with group: stratioStreaming > Executor task launch worker-0 INFO kafka.KafkaReceiver - Connecting to > Zookeeper: node.stratio.com:2181 > sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] > received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 > cap=0]) from Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] > received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 > cap=0]) from Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor] > received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 > cap=0]) from Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] > handled message (8.442354 ms) > StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from > Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] > handled message (8.412421 ms) > StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from > Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor] > handled message (8.385471 ms) > StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from > Actor[akka://sparkDriver/deadLetters] > Executor task launch worker-0 INFO utils.VerifiableProperties - Verifying > properties > Executor task launch worker-0 INFO utils.VerifiableProperties - Property > group.id is overridden to stratioStreaming > Executor task launch worker-0 INFO utils.VerifiableProperties - Property > zookeeper.connect is overridden to node.stratio.com:2181 > Executor task launch worker-0 INFO utils.VerifiableProperties - Property > zookeeper.connection.timeout.ms is overridden to 10000 > Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading > broadcast variable 0 took 0.033998997 s > Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - > [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], Connecting to > zookeeper instance at node.stratio.com:2181 > Executor task launch worker-0 DEBUG zkclient.ZkConnection - Creating new > ZookKeeper instance to connect to node.stratio.com:2181. > ZkClient-EventThread-169-node.stratio.com:2181 INFO zkclient.ZkEventThread - > Starting ZkClient event thread. > Executor task launch worker-0 INFO zookeeper.ZooKeeper - Initiating client > connection, connectString=node.stratio.com:2181 sessionTimeout=6000 > watcher=org.I0Itec.zkclient.ZkClient@5b4bdc81 > Executor task launch worker-0 DEBUG zkclient.ZkClient - Awaiting connection > to Zookeeper server > Executor task launch worker-0 DEBUG zkclient.ZkClient - Waiting for keeper > state SyncConnected > Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO > zookeeper.ClientCnxn - Opening socket connection to server > node.stratio.com/172.19.0.96:2181. Will not attempt to authenticate using > SASL (unknown error) > Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO > zookeeper.ClientCnxn - Socket connection established to > node.stratio.com/172.19.0.96:2181, initiating session > Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG > zookeeper.ClientCnxn - Session establishment request sent on > node.stratio.com/172.19.0.96:2181 > Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO > zookeeper.ClientCnxn - Session establishment complete on server > node.stratio.com/172.19.0.96:2181, sessionid = 0x1496007e6710002, negotiated > timeout = 6000 > Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Received > event: WatchedEvent state:SyncConnected type:None path:null > Executor task launch worker-0-EventThread INFO zkclient.ZkClient - zookeeper > state changed (SyncConnected) > Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Leaving > process event > Executor task launch worker-0 DEBUG zkclient.ZkClient - State is SyncConnected > RecurringTimer - BlockGenerator DEBUG util.RecurringTimer - Callback for > BlockGenerator called at time 1414656492400 > Executor task launch worker-0 DEBUG utils.KafkaScheduler - Initializing task > scheduler. > Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - > [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], starting auto > committer every 60000 ms > Executor task launch worker-0 DEBUG utils.KafkaScheduler - Scheduling task > kafka-consumer-autocommit with initial delay 60000 ms and period 60000 ms. > Executor task launch worker-0 INFO kafka.KafkaReceiver - Connected to > node.stratio.com:2181 > Executor task launch worker-0 DEBUG consumer.ZookeeperConsumerConnector - > [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], entering consume > Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - > [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], begin registering > consumer stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a in ZK > Executor task launch worker-0 DEBUG storage.BlockManager - Getting local > block broadcast_0 > Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0 > not registered locally > Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started > reading broadcast variable 0 > Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading > broadcast variable 0 took 5.5676E-5 s > Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0 > in stage 0.0 (TID 0) > java.io.IOException: PARSING_ERROR(2) > at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78) > at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method) > at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545) > at > org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125) > at > org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88) > at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58) > at > org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128) > at > org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232) > at > org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > at > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0 > in stage 0.0 (TID 0) > java.io.IOException: PARSING_ERROR(2) > at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78) > at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method) > at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545) > at > org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125) > at > org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88) > at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58) > at > org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128) > at > org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232) > at > org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > at > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] > received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 > cap=2144]) from Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] > received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 > cap=2144]) from Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.TaskSchedulerImpl > - parentName: , name: TaskSet_0, runningTasks: 0 > sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.TaskSchedulerImpl > - parentName: , name: TaskSet_0, runningTasks: 0 > sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] > handled message (1.213476 ms) > StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from > Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] > handled message (1.543991 ms) > StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from > Actor[akka://sparkDriver/deadLetters] > Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in > stage 0.0 (TID 0, localhost): java.io.IOException: PARSING_ERROR(2) > org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78) > org.xerial.snappy.SnappyNative.uncompressedLength(Native Method) > org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545) > > org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125) > > org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88) > org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58) > > org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128) > > org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232) > > org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:606) > > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in > stage 0.0 (TID 0, localhost): java.io.IOException: PARSING_ERROR(2) > org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78) > org.xerial.snappy.SnappyNative.uncompressedLength(Native Method) > org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545) > > org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125) > > org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88) > org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58) > > org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128) > > org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232) > > org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:606) > > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0 > failed 1 times; aborting job > Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0 > failed 1 times; aborting job > Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet > 0.0, whose tasks have all completed, from pool > Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet > 0.0, whose tasks have all completed, from pool > sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.TaskSchedulerImpl > - Cancelling stage 0 > sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.TaskSchedulerImpl > - Cancelling stage 0 > Thread-84 INFO scheduler.DAGScheduler - Failed to run runJob at > ReceiverTracker.scala:275 > Thread-85 INFO scheduler.DAGScheduler - Failed to run runJob at > ReceiverTracker.scala:275 > sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - > Removing running stage 0 > sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - > Removing running stage 0 > sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - > After removal of stage 0, remaining stages = 0 > sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - > After removal of stage 0, remaining stages = 0 > Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG > zookeeper.ClientCnxn - Reading reply sessionid:0x1496007e6710002, packet:: > clientPath:null serverPath:null finished:false header:: 1,1 replyHeader:: > 1,25,-101 request:: > '/consumers/stratioStreaming/ids/stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b227374726174696f5f73747265616d696e675f616374696f6e223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343134363536343932343737227d,v{s{31,s{'world,'anyone}}},1 > response:: > {code} > > Only spark version changed > As far as we have checked, snappy will throw this error when dealing with > zero bytes length arrays. > We have tried: > > Changing from snappy to LZF > {code} > Executor task launch worker-0 DEBUG zkclient.ZkConnection - Creating new > ZookKeeper instance to connect to node.stratio.com:2181. > ZkClient-EventThread-166-node.stratio.com:2181 INFO zkclient.ZkEventThread - > Starting ZkClient event thread. > Executor task launch worker-0 INFO zookeeper.ZooKeeper - Initiating client > connection, connectString=node.stratio.com:2181 sessionTimeout=6000 > watcher=org.I0Itec.zkclient.ZkClient@5a4f889 > Executor task launch worker-0 DEBUG zkclient.ZkClient - Awaiting connection > to Zookeeper server > Executor task launch worker-0 DEBUG zkclient.ZkClient - Waiting for keeper > state SyncConnected > Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO > zookeeper.ClientCnxn - Opening socket connection to server > node.stratio.com/172.19.0.96:2181. Will not attempt to authenticate using > SASL (unknown error) > Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO > zookeeper.ClientCnxn - Socket connection established to > node.stratio.com/172.19.0.96:2181, initiating session > Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG > zookeeper.ClientCnxn - Session establishment request sent on > node.stratio.com/172.19.0.96:2181 > Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO > zookeeper.ClientCnxn - Session establishment complete on server > node.stratio.com/172.19.0.96:2181, sessionid = 0x1496007e6710009, negotiated > timeout = 6000 > Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Received > event: WatchedEvent state:SyncConnected type:None path:null > Executor task launch worker-0-EventThread INFO zkclient.ZkClient - zookeeper > state changed (SyncConnected) > Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Leaving > process event > Executor task launch worker-0 DEBUG zkclient.ZkClient - State is SyncConnected > ProducerSendThread- DEBUG async.ProducerSendThread - 5000 ms elapsed. Queue > time reached. Sending.. > ProducerSendThread- DEBUG async.ProducerSendThread - Handling 0 events > Executor task launch worker-0 DEBUG storage.BlockManager - Getting local > block broadcast_0 > Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0 > not registered locally > Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started > reading broadcast variable 0 > Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0 > in stage 0.0 (TID 0) > java.io.EOFException > at > java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) > at > java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) > at > java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) > at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) > at > org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57) > at > org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57) > at > org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95) > at > org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235) > at > org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > at > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading > broadcast variable 0 took 1.002E-4 s > Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0 > in stage 0.0 (TID 0) > java.io.EOFException > at > java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) > at > java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) > at > java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) > at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) > at > org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57) > at > org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57) > at > org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95) > at > org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235) > at > org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > at > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Executor task launch worker-0 DEBUG utils.KafkaScheduler - Initializing task > scheduler. > Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - > [stratioStreaming_ajn-stratio-1414658065894-94786a0e], starting auto > committer every 60000 ms > Executor task launch worker-0 DEBUG utils.KafkaScheduler - Scheduling task > kafka-consumer-autocommit with initial delay 60000 ms and period 60000 ms. > Executor task launch worker-0 INFO kafka.KafkaReceiver - Connected to > node.stratio.com:2181 > Executor task launch worker-0 DEBUG consumer.ZookeeperConsumerConnector - > [stratioStreaming_ajn-stratio-1414658065894-94786a0e], entering consume > sparkDriver-akka.actor.default-dispatcher-3 DEBUG local.LocalActor - [actor] > received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2066 > cap=2066]) from Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-3 DEBUG scheduler.TaskSchedulerImpl > - parentName: , name: TaskSet_0, runningTasks: 0 > sparkDriver-akka.actor.default-dispatcher-3 DEBUG local.LocalActor - [actor] > handled message (1.674221 ms) > StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=179 lim=2066 cap=2066]) > from Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-5 DEBUG local.LocalActor - [actor] > received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2066 > cap=2066]) from Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-5 DEBUG scheduler.TaskSchedulerImpl > - parentName: , name: TaskSet_0, runningTasks: 0 > sparkDriver-akka.actor.default-dispatcher-5 DEBUG local.LocalActor - [actor] > handled message (0.994221 ms) > StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2066 cap=2066]) from > Actor[akka://sparkDriver/deadLetters] > Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in > stage 0.0 (TID 0, localhost): java.io.EOFException: > > java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) > > java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) > java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) > java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) > > org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57) > > org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57) > > org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95) > > org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235) > > org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:606) > > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > sparkDriver-akka.actor.default-dispatcher-5 DEBUG scheduler.JobGenerator - > Got event GenerateJobs(1414658066000 ms) > sparkDriver-akka.actor.default-dispatcher-5 DEBUG streaming.DStreamGraph - > Generating jobs for time 1414658066000 ms > RecurringTimer - BlockGenerator DEBUG util.RecurringTimer - Callback for > BlockGenerator called at time 1414658066000 > Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in > stage 0.0 (TID 0, localhost): java.io.EOFException: > > java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) > > java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) > java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) > java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) > > org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57) > > org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57) > > org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95) > > org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235) > > org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:606) > > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > sparkDriver-akka.actor.default-dispatcher-5 DEBUG dstream.MappedDStream - > Time 1414658066000 ms is valid > RecurringTimer - JobGenerator DEBUG util.RecurringTimer - Callback for > JobGenerator called at time 1414658066000 > Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0 > failed 1 times; aborting job > Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0 > failed 1 times; aborting job > RecurringTimer - JobGenerator DEBUG util.RecurringTimer - Callback for > JobGenerator called at time 1414658066000 > sparkDriver-akka.actor.default-dispatcher-5 DEBUG scheduler.JobGenerator - > Got event GenerateJobs(1414658066000 ms) > RecurringTimer - JobGenerator DEBUG util.RecurringTimer - Callback for > JobGenerator called at time 1414658066000 > sparkDriver-akka.actor.default-dispatcher-3 DEBUG scheduler.JobGenerator - > Got event GenerateJobs(1414658066000 ms) > sparkDriver-akka.actor.default-dispatcher-5 DEBUG streaming.DStreamGraph - > Generating jobs for time 1414658066000 ms > sparkDriver-akka.actor.default-dispatcher-3 DEBUG streaming.DStreamGraph - > Generating jobs for time 1414658066000 ms > sparkDriver-akka.actor.default-dispatcher-5 DEBUG dstream.FilteredDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-5 DEBUG kafka.KafkaInputDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-5 INFO scheduler.ReceiverTracker - > Stream 0 received 0 blocks > Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet > 0.0, whose tasks have all completed, from pool > Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet > 0.0, whose tasks have all completed, from pool > sparkDriver-akka.actor.default-dispatcher-3 DEBUG dstream.MappedDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-5 DEBUG dstream.FilteredDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-3 DEBUG dstream.FilteredDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-5 DEBUG dstream.MapValuedDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-5 DEBUG dstream.ShuffledDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-5 DEBUG > dstream.MapPartitionedDStream - Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-5 DEBUG dstream.MappedDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-3 DEBUG kafka.KafkaInputDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-3 INFO scheduler.ReceiverTracker - > Stream 0 received 0 blocks > sparkDriver-akka.actor.default-dispatcher-5 DEBUG kafka.KafkaInputDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-5 INFO scheduler.ReceiverTracker - > Stream 0 received 0 blocks > sparkDriver-akka.actor.default-dispatcher-5 INFO scheduler.TaskSchedulerImpl > - Cancelling stage 0 > sparkDriver-akka.actor.default-dispatcher-3 INFO scheduler.TaskSchedulerImpl > - Cancelling stage 0 > Thread-85 INFO scheduler.DAGScheduler - Failed to run runJob at > ReceiverTracker.scala:275 > sparkDriver-akka.actor.default-dispatcher-3 INFO kafka.KafkaInputDStream - > Persisting RDD 1 for time 1414658066000 ms to StorageLevel(false, true, > false, false, 1) at time 1414658066000 ms > Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - > [stratioStreaming_ajn-stratio-1414658065894-94786a0e], begin registering > consumer stratioStreaming_ajn-stratio-1414658065894-94786a0e in ZK > Thread-84 INFO scheduler.DAGScheduler - Failed to run runJob at > ReceiverTracker.scala:275 > sparkDriver-akka.actor.default-dispatcher-5 INFO kafka.KafkaInputDStream - > Persisting RDD 1 for time 1414658066000 ms to StorageLevel(false, true, > false, false, 1) at time 1414658066000 ms > sparkDriver-akka.actor.default-dispatcher-5 DEBUG scheduler.DAGScheduler - > Removing running stage 0 > sparkDriver-akka.actor.default-dispatcher-3 DEBUG scheduler.DAGScheduler - > Removing running stage 0 > sparkDriver-akka.actor.default-dispatcher-5 DEBUG scheduler.DAGScheduler - > After removal of stage 0, remaining stages = 0 > sparkDriver-akka.actor.default-dispatcher-3 DEBUG scheduler.DAGScheduler - > After removal of stage 0, remaining stages = 0 > sparkDriver-akka.actor.default-dispatcher-5 DEBUG dstream.MappedDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-5 DEBUG dstream.FilteredDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-3 DEBUG streaming.DStreamGraph - > Generated 1 jobs for time 1414658066000 ms > sparkDriver-akka.actor.default-dispatcher-5 DEBUG dstream.MappedDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-5 DEBUG dstream.FilteredDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-3 INFO scheduler.JobScheduler - > Added jobs for time 1414658066000 ms > sparkDriver-akka.actor.default-dispatcher-3 DEBUG scheduler.JobGenerator - > Got event DoCheckpoint(1414658066000 ms) > sparkDriver-akka.actor.default-dispatcher-5 INFO scheduler.JobScheduler - > Starting job streaming job 1414658066000 ms.0 from job set of time > 1414658066000 ms > sparkDriver-akka.actor.default-dispatcher-5 DEBUG dstream.MappedDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-5 DEBUG dstream.FilteredDStream - > Time 1414658066000 ms is valid > sparkDriver-akka.actor.default-dispatcher-5 DEBUG dstream.MappedDStream - > Time 1414658066000 ms is valid > Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG > zookeeper.ClientCnxn - Reading reply sessionid:0x1496007e6710009, packet:: > clientPath:null serverPath:null finished:false header:: 1,1 replyHeader:: > 1,50,0 request:: > '/consumers/stratioStreaming/ids/stratioStreaming_ajn-stratio-1414658065894-94786a0e,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b227374726174696f5f73747265616d696e675f616374696f6e223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343134363538303636303136227d,v{s{31,s{'world,'anyone}}},1 > response:: > '/consumers/stratioStreaming/ids/stratioStreaming_ajn-stratio-1414658065894-94786a0e > > {code} > > Changing spark.broadcast.compress false > {code} > Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading > broadcast variable 0 took 0.240869283 s > sparkDriver-akka.actor.default-dispatcher-4 DEBUG kafka.KafkaInputDStream - > Cleared 1 RDDs that were older than 1414657342000 ms: 1414657342000 ms > sparkDriver-akka.actor.default-dispatcher-4 DEBUG kafka.KafkaInputDStream - > Cleared 1 RDDs that were older than 1414657342000 ms: 1414657342000 ms > sparkDriver-akka.actor.default-dispatcher-4 DEBUG streaming.DStreamGraph - > Cleared old metadata for time 1414657344000 ms > sparkDriver-akka.actor.default-dispatcher-13 DEBUG > storage.BlockManagerSlaveActor - removing RDD 3 > Executor task launch worker-1 DEBUG storage.BlockManager - Getting local > block broadcast_1 > sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.MappedDStream - > Time 1414657344000 ms is valid > sparkDriver-akka.actor.default-dispatcher-3 DEBUG dstream.FilteredDStream - > Time 1414657344000 ms is valid > sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.FilteredDStream - > Time 1414657344000 ms is valid > sparkDriver-akka.actor.default-dispatcher-13 INFO storage.BlockManager - > Removing RDD 3 > sparkDriver-akka.actor.default-dispatcher-2 DEBUG > storage.BlockManagerSlaveActor - [actor] handled message (134.08408 ms) > RemoveRdd(3) from Actor[akka://sparkDriver/temp/$f] > sparkDriver-akka.actor.default-dispatcher-2 DEBUG > storage.BlockManagerSlaveActor - [actor] received message RemoveRdd(2) from > Actor[akka://sparkDriver/temp/$i] > sparkDriver-akka.actor.default-dispatcher-4 DEBUG > storage.BlockManagerSlaveActor - removing RDD 2 > sparkDriver-akka.actor.default-dispatcher-4 INFO storage.BlockManager - > Removing RDD 2 > sparkDriver-akka.actor.default-dispatcher-2 DEBUG > storage.BlockManagerSlaveActor - [actor] handled message (0.050955 ms) > RemoveRdd(2) from Actor[akka://sparkDriver/temp/$i] > sparkDriver-akka.actor.default-dispatcher-2 DEBUG > storage.BlockManagerSlaveActor - [actor] received message RemoveRdd(1) from > Actor[akka://sparkDriver/temp/$j] > sparkDriver-akka.actor.default-dispatcher-5 DEBUG > storage.BlockManagerSlaveActor - removing RDD 1 > sparkDriver-akka.actor.default-dispatcher-5 INFO storage.BlockManager - > Removing RDD 1 > sparkDriver-akka.actor.default-dispatcher-2 DEBUG > storage.BlockManagerSlaveActor - [actor] handled message (0.037738 ms) > RemoveRdd(1) from Actor[akka://sparkDriver/temp/$j] > Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0 > in stage 0.0 (TID 0) > java.io.EOFException > at > java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) > at > java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) > at > java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) > at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) > at > org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57) > at > org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57) > at > org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95) > at > org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235) > at > org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > at > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > sparkDriver-akka.actor.default-dispatcher-5 DEBUG > storage.BlockManagerSlaveActor - Done removing RDD 1, response is 0 > sparkDriver-akka.actor.default-dispatcher-2 DEBUG > storage.BlockManagerSlaveActor - Done removing RDD 2, response is 0 > sparkDriver-akka.actor.default-dispatcher-3 DEBUG > storage.BlockManagerSlaveActor - Done removing RDD 3, response is 0 > Executor task launch worker-1 DEBUG storage.BlockManager - Level for block > broadcast_1 is StorageLevel(true, true, false, true, 1) > sparkDriver-akka.actor.default-dispatcher-3 DEBUG dstream.FilteredDStream - > Time 1414657344000 ms is valid > sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.MappedDStream - > Time 1414657344000 ms is valid > sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.FilteredDStream - > Time 1414657344000 ms is valid > sparkDriver-akka.actor.default-dispatcher-3 DEBUG streaming.DStreamGraph - > Generated 4 jobs for time 1414657344000 ms > sparkDriver-akka.actor.default-dispatcher-3 INFO scheduler.JobScheduler - > Added jobs for time 1414657344000 ms > sparkDriver-akka.actor.default-dispatcher-3 DEBUG scheduler.JobGenerator - > Got event DoCheckpoint(1414657344000 ms) > sparkDriver-akka.actor.default-dispatcher-5 DEBUG > storage.BlockManagerSlaveActor - Sent response: 0 to > Actor[akka://sparkDriver/temp/$j] > sparkDriver-akka.actor.default-dispatcher-2 DEBUG > storage.BlockManagerSlaveActor - Sent response: 0 to > Actor[akka://sparkDriver/temp/$i] > Executor task launch worker-1 DEBUG storage.BlockManager - Getting block > broadcast_1 from memory > sparkDriver-akka.actor.default-dispatcher-3 DEBUG > storage.BlockManagerSlaveActor - Sent response: 0 to > Actor[akka://sparkDriver/temp/$f] > Executor task launch worker-0 DEBUG storage.BlockManager - Getting local > block broadcast_0 > Executor task launch worker-0 DEBUG storage.BlockManager - Level for block > broadcast_0 is StorageLevel(true, true, false, true, 1) > Executor task launch worker-0 DEBUG storage.BlockManager - Getting block > broadcast_0 from memory > Executor task launch worker-0 DEBUG executor.Executor - Task 0's epoch is 0 > Executor task launch worker-1 DEBUG executor.Executor - Task 1's epoch is 0 > Executor task launch worker-0 DEBUG storage.BlockManager - Getting local > block broadcast_0 > Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0 > not registered locally > Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started > reading broadcast variable 0 > sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.MappedDStream - > Time 1414657344000 ms is valid > sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.FilteredDStream - > Time 1414657344000 ms is valid > Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading > broadcast variable 0 took 7.0321E-5 s > Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0 > in stage 0.0 (TID 0) > java.io.EOFException > at > java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) > at > java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) > at > java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) > at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) > at > org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57) > at > org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57) > at > org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95) > at > org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235) > at > org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > at > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > sparkDriver-akka.actor.default-dispatcher-5 DEBUG local.LocalActor - [actor] > received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2066 > cap=2066]) from Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-3 DEBUG local.LocalActor - [actor] > received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2066 > cap=2066]) from Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-5 DEBUG scheduler.TaskSchedulerImpl > - parentName: , name: TaskSet_0, runningTasks: 0 > sparkDriver-akka.actor.default-dispatcher-3 DEBUG scheduler.TaskSchedulerImpl > - parentName: , name: TaskSet_0, runningTasks: 0 > sparkDriver-akka.actor.default-dispatcher-5 DEBUG local.LocalActor - [actor] > handled message (1.681797 ms) > StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2066 cap=2066]) from > Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-3 DEBUG local.LocalActor - [actor] > handled message (0.688875 ms) > StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2066 cap=2066]) from > Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.MappedDStream - > Time 1414657344000 ms is valid > sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.FilteredDStream - > Time 1414657344000 ms is valid > sparkDriver-akka.actor.default-dispatcher-5 DEBUG spark.HeartbeatReceiver - > [actor] received message > Heartbeat(localhost,[Lscala.Tuple2;@1a94802,BlockManagerId(<driver>, > ajn-stratio.local, 53377, 0)) from Actor[akka://sparkDriver/temp/$c] > sparkDriver-akka.actor.default-dispatcher-3 INFO > receiver.ReceiverSupervisorImpl - Registered receiver 0 > Executor task launch worker-0 INFO util.RecurringTimer - Started timer for > BlockGenerator at time 1414657344800 > Executor task launch worker-0 INFO receiver.BlockGenerator - Started > BlockGenerator > Executor task launch worker-0 INFO receiver.ReceiverSupervisorImpl - > Starting receiver > Thread-87 INFO receiver.BlockGenerator - Started block pushing thread > Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in > stage 0.0 (TID 0, localhost): java.io.EOFException: > > java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) > > java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) > java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) > java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) > > org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57) > > org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57) > > org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95) > > org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235) > > org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:606) > > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in > stage 0.0 (TID 0, localhost): java.io.EOFException: > > java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) > > java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) > java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) > java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) > > org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57) > > org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57) > > org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95) > > org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235) > > org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:606) > > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > Executor task launch worker-0 INFO kafka.KafkaReceiver - Starting Kafka > Consumer Stream with group: stratioStreaming > Executor task launch worker-0 INFO kafka.KafkaReceiver - Connecting to > Zookeeper: node.stratio.com:2181 > sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.MappedDStream - > Time 1414657344000 ms is valid > sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.FilteredDStream - > Time 1414657344000 ms is valid > Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0 > failed 1 times; aborting job > Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0 > failed 1 times; aborting job > sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.ReceiverTracker - > Registered receiver for stream 0 from akka://sparkDriver > Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet > 0.0, whose tasks have all completed, from pool > Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet > 0.0, whose tasks have all completed, from pool > sparkDriver-akka.actor.default-dispatcher-3 DEBUG > storage.BlockManagerMasterActor - [actor] received message > BlockManagerHeartbeat(BlockManagerId(<driver>, ajn-stratio.local, 53377, 0)) > from Actor[akka://sparkDriver/temp/$d] > sparkDriver-akka.actor.default-dispatcher-3 DEBUG > storage.BlockManagerMasterActor - [actor] handled message (0.197908 ms) > BlockManagerHeartbeat(BlockManagerId(<driver>, ajn-stratio.local, 53377, 0)) > from Actor[akka://sparkDriver/temp/$d] > sparkDriver-akka.actor.default-dispatcher-5 DEBUG spark.HeartbeatReceiver - > [actor] handled message (169.804965 ms) > Heartbeat(localhost,[Lscala.Tuple2;@1a94802,BlockManagerId(<driver>, > ajn-stratio.local, 53377, 0)) from Actor[akka://sparkDriver/temp/$c] > {code} > > Changing from TorrentBroadcast to HTTPBroadcast ("spark.broadcast.factory", > > "org.apache.spark.broadcast.HttpBroadcastFactory"). > {code} > sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.DAGScheduler - > Missing parents: List() > sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - > submitStage(Stage 1) > sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - > missing: List() > sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.DAGScheduler - > Submitting Stage 1 (FilteredRDD[6] at filter at FilteredDStream.scala:35), > which has no missing parents > sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - > submitMissingTasks(Stage 1) > sparkDriver-akka.actor.default-dispatcher-12 DEBUG dstream.MappedDStream - > Time 1414657758000 ms is valid > sparkDriver-akka.actor.default-dispatcher-12 DEBUG dstream.FilteredDStream - > Time 1414657758000 ms is valid > sparkDriver-akka.actor.default-dispatcher-12 DEBUG dstream.MappedDStream - > Time 1414657758000 ms is valid > sparkDriver-akka.actor.default-dispatcher-12 DEBUG dstream.FilteredDStream - > Time 1414657758000 ms is valid > sparkDriver-akka.actor.default-dispatcher-12 DEBUG dstream.MappedDStream - > Time 1414657758000 ms is valid > sparkDriver-akka.actor.default-dispatcher-12 DEBUG dstream.FilteredDStream - > Time 1414657758000 ms is valid > Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG > zookeeper.ClientCnxn - Reading reply sessionid:0x1496007e6710006, packet:: > clientPath:null serverPath:null finished:false header:: 7,4 replyHeader:: > 7,41,0 request:: > '/consumers/stratioStreaming/ids/stratioStreaming_ajn-stratio-1414657757842-d7a2ca15,F > response:: > #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b227374726174696f5f73747265616d696e675f7265717565737473223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343134363537373538303535227d,s{41,41,1414657758409,1414657758409,0,0,0,92710854385008646,108,0,41} > > sparkDriver-akka.actor.default-dispatcher-12 DEBUG dstream.MappedDStream - > Time 1414657758000 ms is valid > sparkDriver-akka.actor.default-dispatcher-12 DEBUG dstream.FilteredDStream - > Time 1414657758000 ms is valid > sparkDriver-akka.actor.default-dispatcher-12 DEBUG dstream.MappedDStream - > Time 1414657758000 ms is valid > sparkDriver-akka.actor.default-dispatcher-12 DEBUG dstream.FilteredDStream - > Time 1414657758000 ms is valid > sparkDriver-akka.actor.default-dispatcher-12 DEBUG dstream.MappedDStream - > Time 1414657758000 ms is valid > sparkDriver-akka.actor.default-dispatcher-12 DEBUG dstream.FilteredDStream - > Time 1414657758000 ms is valid > qtp1571833412-35 DEBUG http.HttpParser - filled 167/167 > RecurringTimer - BlockGenerator DEBUG util.RecurringTimer - Callback for > BlockGenerator called at time 1414657758400 > sparkDriver-akka.actor.default-dispatcher-12 DEBUG dstream.MappedDStream - > Time 1414657758000 ms is valid > sparkDriver-akka.actor.default-dispatcher-12 DEBUG dstream.FilteredDStream - > Time 1414657758000 ms is valid > qtp1571833412-35 - /broadcast_0 DEBUG server.Server - REQUEST /broadcast_0 on > BlockingHttpConnection@7cbd5b8f,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-5,l=10,c=0},r=1 > sparkDriver-akka.actor.default-dispatcher-12 DEBUG streaming.DStreamGraph - > Generated 14 jobs for time 1414657758000 ms > sparkDriver-akka.actor.default-dispatcher-12 INFO scheduler.JobScheduler - > Added jobs for time 1414657758000 ms > sparkDriver-akka.actor.default-dispatcher-3 INFO scheduler.JobScheduler - > Starting job streaming job 1414657758000 ms.0 from job set of time > 1414657758000 ms > sparkDriver-akka.actor.default-dispatcher-12 DEBUG scheduler.JobGenerator - > Got event DoCheckpoint(1414657758000 ms) > qtp1571833412-35 - /broadcast_0 DEBUG server.Server - RESPONSE /broadcast_0 > 404 handled=true > pool-7-thread-1 INFO spark.SparkContext - Starting job: collect at > ActionBaseFunction.java:65 > Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG > zookeeper.ClientCnxn - Reading reply sessionid:0x1496007e6710006, packet:: > clientPath:null serverPath:null finished:false header:: 8,8 replyHeader:: > 8,41,0 request:: '/consumers/stratioStreaming/ids,T response:: > v{'stratioStreaming_ajn-stratio-1414657757842-d7a2ca15} > Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG > zookeeper.ClientCnxn - Reading reply sessionid:0x1496007e6710006, packet:: > clientPath:null serverPath:null finished:false header:: 9,4 replyHeader:: > 9,41,0 request:: > '/consumers/stratioStreaming/ids/stratioStreaming_ajn-stratio-1414657757842-d7a2ca15,F > response:: > #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b227374726174696f5f73747265616d696e675f7265717565737473223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343134363537373538303535227d,s{41,41,1414657758409,1414657758409,0,0,0,92710854385008646,108,0,41} > > pool-7-thread-1 INFO spark.SparkContext - Job finished: collect at > ActionBaseFunction.java:65, took 3.9409E-5 s > Executor task launch worker-0 DEBUG storage.BlockManager - Getting local > block broadcast_0 > Executor task launch worker-0 DEBUG storage.BlockManager - Level for block > broadcast_0 is StorageLevel(true, true, false, true, 1) > Executor task launch worker-0 DEBUG storage.BlockManager - Getting block > broadcast_0 from memory > sparkDriver-akka.actor.default-dispatcher-12 INFO scheduler.JobScheduler - > Finished job streaming job 1414657758000 ms.0 from job set of time > 1414657758000 ms > Executor task launch worker-0 INFO storage.BlockManager - Found block > broadcast_0 locally > sparkDriver-akka.actor.default-dispatcher-12 INFO scheduler.JobScheduler - > Starting job streaming job 1414657758000 ms.1 from job set of time > 1414657758000 ms > Executor task launch worker-0 DEBUG executor.Executor - Task 0's epoch is 0 > Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0 > in stage 0.0 (TID 0) > java.io.FileNotFoundException: http://172.17.42.1:34477/broadcast_0 > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1624) > at > org.apache.spark.broadcast.HttpBroadcast$.org$apache$spark$broadcast$HttpBroadcast$$read(HttpBroadcast.scala:197) > at > org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala:89) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > at > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > pool-7-thread-1 INFO spark.SparkContext - Starting job: collect at > ActionBaseFunction.java:65 > pool-7-thread-1 INFO spark.SparkContext - Job finished: collect at > ActionBaseFunction.java:65, took 3.1765E-5 s > Executor task launch worker-0 INFO util.RecurringTimer - Started timer for > BlockGenerator at time 1414657758600 > Executor task launch worker-0 INFO receiver.BlockGenerator - Started > BlockGenerator > Executor task launch worker-0 INFO receiver.ReceiverSupervisorImpl - > Starting receiver > sparkDriver-akka.actor.default-dispatcher-2 INFO storage.MemoryStore - > ensureFreeSpace(3136) called with curMem=1216, maxMem=991470551 > sparkDriver-akka.actor.default-dispatcher-12 INFO scheduler.JobScheduler - > Finished job streaming job 1414657758000 ms.1 from job set of time > 1414657758000 ms > sparkDriver-akka.actor.default-dispatcher-12 INFO scheduler.JobScheduler - > Starting job streaming job 1414657758000 ms.2 from job set of time > 1414657758000 ms > sparkDriver-akka.actor.default-dispatcher-5 DEBUG local.LocalActor - [actor] > received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=1868 > cap=1868]) from Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-2 INFO storage.MemoryStore - Block > broadcast_1 stored as values in memory (estimated size 3.1 KB, free 945.5 MB) > Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG > zookeeper.ClientCnxn - Reading reply sessionid:0x1496007e6710006, packet:: > clientPath:null serverPath:null finished:false header:: 10,8 replyHeader:: > 10,41,0 request:: '/brokers/ids,F response:: v{'7} > sparkDriver-akka.actor.default-dispatcher-2 DEBUG storage.BlockManager - Put > block broadcast_1 locally took 7 ms > sparkDriver-akka.actor.default-dispatcher-2 DEBUG storage.BlockManager - > Putting block broadcast_1 without replication took 7 ms > pool-7-thread-1 INFO spark.SparkContext - Starting job: collect at > ActionBaseFunction.java:65 > Executor task launch worker-0 INFO kafka.KafkaReceiver - Starting Kafka > Consumer Stream with group: stratioStreaming > sparkDriver-akka.actor.default-dispatcher-3 INFO > receiver.ReceiverSupervisorImpl - Registered receiver 0 > Executor task launch worker-0 INFO kafka.KafkaReceiver - Connecting to > Zookeeper: node.stratio.com:2181 > Executor task launch worker-0 INFO utils.VerifiableProperties - Verifying > properties > Executor task launch worker-0 INFO utils.VerifiableProperties - Property > group.id is overridden to stratioStreaming > Executor task launch worker-0 INFO utils.VerifiableProperties - Property > zookeeper.connect is overridden to node.stratio.com:2181 > Executor task launch worker-0 INFO utils.VerifiableProperties - Property > zookeeper.connection.timeout.ms is overridden to 10000 > Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - > [stratioStreaming_ajn-stratio-1414657758445-7b49bb3b], Connecting to > zookeeper instance at node.stratio.com:2181 > sparkDriver-akka.actor.default-dispatcher-4 INFO scheduler.ReceiverTracker - > Registered receiver for stream 0 from akka://sparkDriver > Thread-99 INFO receiver.BlockGenerator - Started block pushing thread > sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.DAGScheduler - > Submitting 2 missing tasks from Stage 1 (FilteredRDD[6] at filter at > FilteredDStream.scala:35) > sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - > New pending tasks: Set(ResultTask(1, 1), ResultTask(1, 0)) > sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.TaskSchedulerImpl > - Adding task set 1.0 with 2 tasks > sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.TaskSetManager - > Epoch for TaskSet 1.0: 0 > sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.TaskSetManager - > Valid locality levels for TaskSet 1.0: NO_PREF, ANY > Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG > zookeeper.ClientCnxn - Reading reply sessionid:0x1496007e6710006, packet:: > clientPath:null serverPath:null finished:false header:: 11,4 replyHeader:: > 11,41,0 request:: '/brokers/ids/7,F response:: > #7b226a6d785f706f7274223a393939392c2274696d657374616d70223a2231343134363535333735373234222c22686f7374223a226e6f64652e7374726174696f2e636f6d222c2276657273696f6e223a312c22706f7274223a393039327d,s{18,18,1414655375792,1414655375792,0,0,0,92710854385008640,95,0,18} > > sparkDriver-akka.actor.default-dispatcher-3 DEBUG local.LocalActor - [actor] > received message ReviveOffers from Actor[akka://sparkDriver/deadLetters] > pool-7-thread-1 INFO spark.SparkContext - Job finished: collect at > ActionBaseFunction.java:65, took 3.0385E-5 s > Executor task launch worker-0 DEBUG zkclient.ZkConnection - Creating new > ZookKeeper instance to connect to node.stratio.com:2181. > Executor task launch worker-0 INFO zookeeper.ZooKeeper - Initiating client > connection, connectString=node.stratio.com:2181 sessionTimeout=6000 > watcher=org.I0Itec.zkclient.ZkClient@2fdc4517 > ZkClient-EventThread-189-node.stratio.com:2181 INFO zkclient.ZkEventThread - > Starting ZkClient event thread. > sparkDriver-akka.actor.default-dispatcher-5 DEBUG scheduler.TaskSchedulerImpl > - parentName: , name: TaskSet_0, runningTasks: 0 > sparkDriver-akka.actor.default-dispatcher-3 DEBUG scheduler.TaskSchedulerImpl > - parentName: , name: TaskSet_0, runningTasks: 1 > sparkDriver-akka.actor.default-dispatcher-5 DEBUG local.LocalActor - [actor] > handled message (4.883443 ms) > StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=1868 cap=1868]) from > Actor[akka://sparkDriver/deadLetters] > sparkDriver-akka.actor.default-dispatcher-3 DEBUG scheduler.TaskSchedulerImpl > - parentName: , name: TaskSet_1, runningTasks: 0 > sparkDriver-akka.actor.default-dispatcher-3 INFO scheduler.TaskSetManager - > Starting task 0.0 in stage 1.0 (TID 1, localhost, PROCESS_LOCAL, 880 bytes) > Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO > zookeeper.ClientCnxn - Opening socket connection to server > node.stratio.com/172.19.0.96:2181. Will not attempt to authenticate using > SASL (unknown error) > Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO > zookeeper.ClientCnxn - Socket connection established to > node.stratio.com/172.19.0.96:2181, initiating session > Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG > zookeeper.ClientCnxn - Session establishment request sent on > node.stratio.com/172.19.0.96:2181 > Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO > zookeeper.ClientCnxn - Session establishment complete on server > node.stratio.com/172.19.0.96:2181, sessionid = 0x1496007e6710007, negotiated > timeout = 6000 > Executor task launch worker-0 DEBUG zkclient.ZkClient - Awaiting connection > to Zookeeper server > Executor task launch worker-0 DEBUG zkclient.ZkClient - Waiting for keeper > state SyncConnected > sparkDriver-akka.actor.default-dispatcher-12 INFO scheduler.JobScheduler - > Finished job streaming job 1414657758000 ms.2 from job set of time > 1414657758000 ms > Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Received > event: WatchedEvent state:SyncConnected type:None path:null > Executor task launch worker-0-EventThread INFO zkclient.ZkClient - zookeeper > state changed (SyncConnected) > sparkDriver-akka.actor.default-dispatcher-12 INFO scheduler.JobScheduler - > Starting job streaming job 1414657758000 ms.3 from job set of time > 1414657758000 ms > Executor task launch worker-0 DEBUG zkclient.ZkClient - State is SyncConnected > Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in > stage 0.0 (TID 0, localhost): java.io.FileNotFoundException: > http://172.17.42.1:34477/broadcast_0 > > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1624) > > org.apache.spark.broadcast.HttpBroadcast$.org$apache$spark$broadcast$HttpBroadcast$$read(HttpBroadcast.scala:197) > > org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala:89) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:606) > > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Leaving > process event > Executor task launch worker-0 DEBUG utils.KafkaScheduler - Initializing task > scheduler. > sparkDriver-akka.actor.default-dispatcher-3 DEBUG local.LocalActor - [actor] > handled message (7.610459 ms) ReviveOffers from > Actor[akka://sparkDriver/deadLetters] > Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - > [stratioStreaming_ajn-stratio-1414657758445-7b49bb3b], starting auto > committer every 60000 ms > Executor task launch worker-1 INFO executor.Executor - Running task 0.0 in > stage 1.0 (TID 1) > Executor task launch worker-0 DEBUG utils.KafkaScheduler - Scheduling task > kafka-consumer-autocommit with initial delay 60000 ms and period 60000 ms. > sparkDriver-akka.actor.default-dispatcher-3 DEBUG local.LocalActor - [actor] > received message StatusUpdate(1,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 > cap=0]) from Actor[akka://sparkDriver/deadLetters] > Executor task launch worker-0 INFO kafka.KafkaReceiver - Connected to > node.stratio.com:2181 > Executor task launch worker-0 DEBUG consumer.ZookeeperConsumerConnector - > [stratioStreaming_ajn-stratio-1414657758445-7b49bb3b], entering consume > sparkDriver-akka.actor.default-dispatcher-3 DEBUG local.LocalActor - [actor] > handled message (0.07141 ms) > StatusUpdate(1,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from > Actor[akka://sparkDriver/deadLetters] > {code} > but with no luck for the moment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org