[jira] [Commented] (SPARK-10485) IF expression is not correctly resolved when one of the options have NullType
[ https://issues.apache.org/jira/browse/SPARK-10485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747089#comment-14747089 ] Antonio Jesus Navarro commented on SPARK-10485: --- Now, the bug is into the checking of the DataTypes: https://github.com/apache/spark/blob/branch-1.5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala#L33 else if (trueValue.dataType != falseValue.dataType) { TypeCheckResult.TypeCheckFailure(s"differing types in '$prettyString' " + s"(${trueValue.dataType.simpleString} and ${falseValue.dataType.simpleString}).") } If the true expression or the false expression is a NullType, now the type validation fails. > IF expression is not correctly resolved when one of the options have NullType > - > > Key: SPARK-10485 > URL: https://issues.apache.org/jira/browse/SPARK-10485 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.4.1 >Reporter: Antonio Jesus Navarro > > If we have this query: > {code} > SELECT IF(column > 1, 1, NULL) FROM T1 > {code} > On Spark 1.4.1 we have this: > {code} > override lazy val resolved = childrenResolved && trueValue.dataType == > falseValue.dataType > {code} > So if one of the types is NullType, the if expression is not resolved. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-10485) IF expression is not correctly resolved when one of the options have NullType
[ https://issues.apache.org/jira/browse/SPARK-10485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747089#comment-14747089 ] Antonio Jesus Navarro edited comment on SPARK-10485 at 9/16/15 7:32 AM: Now, the bug is into the checking of the DataTypes: https://github.com/apache/spark/blob/branch-1.5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala#L33 {code} else if (trueValue.dataType != falseValue.dataType) { TypeCheckResult.TypeCheckFailure(s"differing types in '$prettyString' " + s"(${trueValue.dataType.simpleString} and ${falseValue.dataType.simpleString}).") } {code} If the true expression or the false expression is a NullType, now the type validation fails. was (Author: ajnavarro): Now, the bug is into the checking of the DataTypes: https://github.com/apache/spark/blob/branch-1.5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala#L33 else if (trueValue.dataType != falseValue.dataType) { TypeCheckResult.TypeCheckFailure(s"differing types in '$prettyString' " + s"(${trueValue.dataType.simpleString} and ${falseValue.dataType.simpleString}).") } If the true expression or the false expression is a NullType, now the type validation fails. > IF expression is not correctly resolved when one of the options have NullType > - > > Key: SPARK-10485 > URL: https://issues.apache.org/jira/browse/SPARK-10485 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.4.1 >Reporter: Antonio Jesus Navarro > > If we have this query: > {code} > SELECT IF(column > 1, 1, NULL) FROM T1 > {code} > On Spark 1.4.1 we have this: > {code} > override lazy val resolved = childrenResolved && trueValue.dataType == > falseValue.dataType > {code} > So if one of the types is NullType, the if expression is not resolved. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-10485) IF expression is not correctly resolved when one of the options have NullType
Antonio Jesus Navarro created SPARK-10485: - Summary: IF expression is not correctly resolved when one of the options have NullType Key: SPARK-10485 URL: https://issues.apache.org/jira/browse/SPARK-10485 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.4.1 Reporter: Antonio Jesus Navarro If we have this query: {code} SELECT IF(column > 1, 1, NULL) FROM T1 {code} On Spark 1.4.1 we have this: {code} override lazy val resolved = childrenResolved && trueValue.dataType == falseValue.dataType {code} So if one of the types is NullType, the if expression is not resolved. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-8658) AttributeReference equals method only compare name, exprId and dataType
Antonio Jesus Navarro created SPARK-8658: Summary: AttributeReference equals method only compare name, exprId and dataType Key: SPARK-8658 URL: https://issues.apache.org/jira/browse/SPARK-8658 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.4.0, 1.3.1, 1.3.0 Reporter: Antonio Jesus Navarro The AttributeReference equals method only accept as different objects with different name, expression id or dataType. With this behavior when I tried to do a transformExpressionsDown and try to transform qualifiers inside AttributeReferences, these objects are not replaced, because the transformer considers them equal. I propose to add to the equals method this variables: name, dataType, nullable, metadata, epxrId, qualifiers -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-4133) PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0
[ https://issues.apache.org/jira/browse/SPARK-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antonio Jesus Navarro updated SPARK-4133: - Description: Snappy related problems found when trying to upgrade existing Spark Streaming App from 1.0.2 to 1.1.0. We can not run an existing 1.0.2 spark app if upgraded to 1.1.0 IOException is thrown by snappy (parsing_error(2)) Only spark version changed As far as we have checked, snappy will throw this error when dealing with zero bytes length arrays. We have tried: Changing from snappy to LZF, Changing broadcast.compression false Changing from TorrentBroadcast to HTTPBroadcast. but with no luck for the moment. {code} Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block broadcast_0 Executor task launch worker-0 DEBUG storage.BlockManager - Level for block broadcast_0 is StorageLevel(true, true, false, true, 1) Executor task launch worker-0 DEBUG storage.BlockManager - Getting block broadcast_0 from memory Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block broadcast_0 Executor task launch worker-0 DEBUG executor.Executor - Task 0's epoch is 0 Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0 not registered locally Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started reading broadcast variable 0 sparkDriver-akka.actor.default-dispatcher-4 INFO receiver.ReceiverSupervisorImpl - Registered receiver 0 Executor task launch worker-0 INFO util.RecurringTimer - Started timer for BlockGenerator at time 1414656492400 Executor task launch worker-0 INFO receiver.BlockGenerator - Started BlockGenerator Thread-87 INFO receiver.BlockGenerator - Started block pushing thread Executor task launch worker-0 INFO receiver.ReceiverSupervisorImpl - Starting receiver sparkDriver-akka.actor.default-dispatcher-5 INFO scheduler.ReceiverTracker - Registered receiver for stream 0 from akka://sparkDriver Executor task launch worker-0 INFO kafka.KafkaReceiver - Starting Kafka Consumer Stream with group: stratioStreaming Executor task launch worker-0 INFO kafka.KafkaReceiver - Connecting to Zookeeper: node.stratio.com:2181 sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] handled message (8.442354 ms) StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] handled message (8.412421 ms) StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor] handled message (8.385471 ms) StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] Executor task launch worker-0 INFO utils.VerifiableProperties - Verifying properties Executor task launch worker-0 INFO utils.VerifiableProperties - Property group.id is overridden to stratioStreaming Executor task launch worker-0 INFO utils.VerifiableProperties - Property zookeeper.connect is overridden to node.stratio.com:2181 Executor task launch worker-0 INFO utils.VerifiableProperties - Property zookeeper.connection.timeout.ms is overridden to 1 Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading broadcast variable 0 took 0.033998997 s Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], Connecting to zookeeper instance at node.stratio.com:2181 Executor task launch worker-0 DEBUG zkclient.ZkConnection - Creating new ZookKeeper instance to connect to node.stratio.com:2181. ZkClient-EventThread-169-node.stratio.com:2181 INFO zkclient.ZkEventThread - Starting ZkClient event thread. Executor task launch worker-0 INFO zookeeper.ZooKeeper - Initiating client connection, connectString=node.stratio.com:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@5b4bdc81 Executor task launch worker-0 DEBUG zkclient.ZkClient - Awaiting connection to Zookeeper server Executor task launch worker-0 DEBUG zkclient.ZkClient - Waiting for keeper state SyncConnected Executor task launch
[jira] [Updated] (SPARK-4133) PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0
[ https://issues.apache.org/jira/browse/SPARK-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antonio Jesus Navarro updated SPARK-4133: - Description: Snappy related problems found when trying to upgrade existing Spark Streaming App from 1.0.2 to 1.1.0. We can not run an existing 1.0.2 spark app if upgraded to 1.1.0 IOException is thrown by snappy (parsing_error(2)) {code} Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block broadcast_0 Executor task launch worker-0 DEBUG storage.BlockManager - Level for block broadcast_0 is StorageLevel(true, true, false, true, 1) Executor task launch worker-0 DEBUG storage.BlockManager - Getting block broadcast_0 from memory Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block broadcast_0 Executor task launch worker-0 DEBUG executor.Executor - Task 0's epoch is 0 Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0 not registered locally Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started reading broadcast variable 0 sparkDriver-akka.actor.default-dispatcher-4 INFO receiver.ReceiverSupervisorImpl - Registered receiver 0 Executor task launch worker-0 INFO util.RecurringTimer - Started timer for BlockGenerator at time 1414656492400 Executor task launch worker-0 INFO receiver.BlockGenerator - Started BlockGenerator Thread-87 INFO receiver.BlockGenerator - Started block pushing thread Executor task launch worker-0 INFO receiver.ReceiverSupervisorImpl - Starting receiver sparkDriver-akka.actor.default-dispatcher-5 INFO scheduler.ReceiverTracker - Registered receiver for stream 0 from akka://sparkDriver Executor task launch worker-0 INFO kafka.KafkaReceiver - Starting Kafka Consumer Stream with group: stratioStreaming Executor task launch worker-0 INFO kafka.KafkaReceiver - Connecting to Zookeeper: node.stratio.com:2181 sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] handled message (8.442354 ms) StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] handled message (8.412421 ms) StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor] handled message (8.385471 ms) StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] Executor task launch worker-0 INFO utils.VerifiableProperties - Verifying properties Executor task launch worker-0 INFO utils.VerifiableProperties - Property group.id is overridden to stratioStreaming Executor task launch worker-0 INFO utils.VerifiableProperties - Property zookeeper.connect is overridden to node.stratio.com:2181 Executor task launch worker-0 INFO utils.VerifiableProperties - Property zookeeper.connection.timeout.ms is overridden to 1 Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading broadcast variable 0 took 0.033998997 s Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], Connecting to zookeeper instance at node.stratio.com:2181 Executor task launch worker-0 DEBUG zkclient.ZkConnection - Creating new ZookKeeper instance to connect to node.stratio.com:2181. ZkClient-EventThread-169-node.stratio.com:2181 INFO zkclient.ZkEventThread - Starting ZkClient event thread. Executor task launch worker-0 INFO zookeeper.ZooKeeper - Initiating client connection, connectString=node.stratio.com:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@5b4bdc81 Executor task launch worker-0 DEBUG zkclient.ZkClient - Awaiting connection to Zookeeper server Executor task launch worker-0 DEBUG zkclient.ZkClient - Waiting for keeper state SyncConnected Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO zookeeper.ClientCnxn - Opening socket connection to server node.stratio.com/172.19.0.96:2181. Will not attempt to authenticate using SASL (unknown error) Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO zookeeper.ClientCnxn - Socket connection
[jira] [Commented] (SPARK-4133) PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0
[ https://issues.apache.org/jira/browse/SPARK-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14189821#comment-14189821 ] Antonio Jesus Navarro commented on SPARK-4133: -- Hi Josh, I edited the issue to add more logs. Thanks. PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0 -- Key: SPARK-4133 URL: https://issues.apache.org/jira/browse/SPARK-4133 Project: Spark Issue Type: Bug Components: Streaming Affects Versions: 1.1.0 Reporter: Antonio Jesus Navarro Priority: Blocker Snappy related problems found when trying to upgrade existing Spark Streaming App from 1.0.2 to 1.1.0. We can not run an existing 1.0.2 spark app if upgraded to 1.1.0 IOException is thrown by snappy (parsing_error(2)) {code} Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block broadcast_0 Executor task launch worker-0 DEBUG storage.BlockManager - Level for block broadcast_0 is StorageLevel(true, true, false, true, 1) Executor task launch worker-0 DEBUG storage.BlockManager - Getting block broadcast_0 from memory Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block broadcast_0 Executor task launch worker-0 DEBUG executor.Executor - Task 0's epoch is 0 Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0 not registered locally Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started reading broadcast variable 0 sparkDriver-akka.actor.default-dispatcher-4 INFO receiver.ReceiverSupervisorImpl - Registered receiver 0 Executor task launch worker-0 INFO util.RecurringTimer - Started timer for BlockGenerator at time 1414656492400 Executor task launch worker-0 INFO receiver.BlockGenerator - Started BlockGenerator Thread-87 INFO receiver.BlockGenerator - Started block pushing thread Executor task launch worker-0 INFO receiver.ReceiverSupervisorImpl - Starting receiver sparkDriver-akka.actor.default-dispatcher-5 INFO scheduler.ReceiverTracker - Registered receiver for stream 0 from akka://sparkDriver Executor task launch worker-0 INFO kafka.KafkaReceiver - Starting Kafka Consumer Stream with group: stratioStreaming Executor task launch worker-0 INFO kafka.KafkaReceiver - Connecting to Zookeeper: node.stratio.com:2181 sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] handled message (8.442354 ms) StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] handled message (8.412421 ms) StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor] handled message (8.385471 ms) StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters] Executor task launch worker-0 INFO utils.VerifiableProperties - Verifying properties Executor task launch worker-0 INFO utils.VerifiableProperties - Property group.id is overridden to stratioStreaming Executor task launch worker-0 INFO utils.VerifiableProperties - Property zookeeper.connect is overridden to node.stratio.com:2181 Executor task launch worker-0 INFO utils.VerifiableProperties - Property zookeeper.connection.timeout.ms is overridden to 1 Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading broadcast variable 0 took 0.033998997 s Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], Connecting to zookeeper instance at node.stratio.com:2181 Executor task launch worker-0 DEBUG zkclient.ZkConnection - Creating new ZookKeeper instance to connect to node.stratio.com:2181. ZkClient-EventThread-169-node.stratio.com:2181 INFO zkclient.ZkEventThread - Starting ZkClient event thread. Executor task launch worker-0 INFO zookeeper.ZooKeeper - Initiating client connection, connectString=node.stratio.com:2181 sessionTimeout=6000
[jira] [Created] (SPARK-4133) PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0
Antonio Jesus Navarro created SPARK-4133: Summary: PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0 Key: SPARK-4133 URL: https://issues.apache.org/jira/browse/SPARK-4133 Project: Spark Issue Type: Bug Components: Streaming Affects Versions: 1.1.0 Reporter: Antonio Jesus Navarro Priority: Blocker Snappy related problems found when trying to upgrade existing Spark Streaming App from 1.0.2 to 1.1.0. We can not run an existing 1.0.2 spark app if upgraded to 1.1.0 IOException is thrown by snappy (parsing_error(2)) Only spark version changed As far as we have checked, snappy will throw this error when dealing with zero bytes length arrays. We have tried: Changing from snappy to LZF, Changing broadcast.compression false Changing from TorrentBroadcast to HTTPBroadcast. but with no luck for the moment. {code} [ERROR] 2014-10-29 11:23:26,396 [Executor task launch worker-0] org.apache.spark.executor.Executor logError - Exception in task 0.0 in stage 0.0 (TID 0) java.io.IOException: PARSING_ERROR(2) at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78) at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method) at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545) at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125) at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88) at org.xerial.snappy.SnappyInputStream.init(SnappyInputStream.java:58) at org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128) at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232) at org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4133) PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0
[ https://issues.apache.org/jira/browse/SPARK-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14188284#comment-14188284 ] Antonio Jesus Navarro commented on SPARK-4133: -- Existing Spark Streaming app can not be upgraded from 1.0.2 to 1.1.0 PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0 -- Key: SPARK-4133 URL: https://issues.apache.org/jira/browse/SPARK-4133 Project: Spark Issue Type: Bug Components: Streaming Affects Versions: 1.1.0 Reporter: Antonio Jesus Navarro Priority: Blocker Snappy related problems found when trying to upgrade existing Spark Streaming App from 1.0.2 to 1.1.0. We can not run an existing 1.0.2 spark app if upgraded to 1.1.0 IOException is thrown by snappy (parsing_error(2)) Only spark version changed As far as we have checked, snappy will throw this error when dealing with zero bytes length arrays. We have tried: Changing from snappy to LZF, Changing broadcast.compression false Changing from TorrentBroadcast to HTTPBroadcast. but with no luck for the moment. {code} [ERROR] 2014-10-29 11:23:26,396 [Executor task launch worker-0] org.apache.spark.executor.Executor logError - Exception in task 0.0 in stage 0.0 (TID 0) java.io.IOException: PARSING_ERROR(2) at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78) at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method) at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545) at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125) at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88) at org.xerial.snappy.SnappyInputStream.init(SnappyInputStream.java:58) at org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128) at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232) at org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org