[jira] [Commented] (TINKERPOP-1426) GryoSerializer should implement Java serialization interface
[ https://issues.apache.org/jira/browse/TINKERPOP-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458260#comment-15458260 ] Chen Xin Yu commented on TINKERPOP-1426: There is performance issue with the two change sets together. This JIRA is not related to Spark 2.0, even without Spark 2.0, Class GryoSerializer should also implement Java serialization interface. So I submit a separate PR on github. https://github.com/apache/tinkerpop/pull/398 > GryoSerializer should implement Java serialization interface > > > Key: TINKERPOP-1426 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1426 > Project: TinkerPop > Issue Type: Bug > Components: io >Affects Versions: 3.2.1 >Reporter: Chen Xin Yu > > There is description for Serializer in spark: > * Implementations of this trait should implement: > * > * 1. a zero-arg constructor or a constructor that accepts a > [[org.apache.spark.SparkConf]] > * as parameter. If both constructors are defined, the latter takes > precedence. > * > * 2. Java serialization interface. > Class GryoSerializer in Tinkerepop extends Serializer, but does not implement > java.io.Serializable. > It works well before Spark 2.0. But with Spark 2.0, it changed by SPARK-13926 > for Dependency,scala. > Gyro and all its fields must implement Java serialisation interface, > otherwise hundreds of test cases are failed as: > Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: > Task not serializable: java.io.NotSerializableException: > org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer > Serialization stack: > - object not serializable (class: > org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer, value: > org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer@1b12ec8e) > - field (class: org.apache.spark.ShuffleDependency, name: serializer, > type: class org.apache.spark.serializer.Serializer) > - object (class org.apache.spark.ShuffleDependency, > org.apache.spark.ShuffleDependency@7a4f876a) > - field (class: scala.Tuple2, name: _2, type: class java.lang.Object) > - object (class scala.Tuple2, (MapPartitionsRDD[1] at mapToPair at > InputFormatRDD.java:46,org.apache.spark.ShuffleDependency@7a4f876a)) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1426) GryoSerializer should implement Java serialization interface
[ https://issues.apache.org/jira/browse/TINKERPOP-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457715#comment-15457715 ] Chen Xin Yu commented on TINKERPOP-1426: Thanks Marko, I created a PR with both change sets for TINKERPOP-1389 and TINKERPOP-1426: https://github.com/apache/tinkerpop/pull/397 > GryoSerializer should implement Java serialization interface > > > Key: TINKERPOP-1426 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1426 > Project: TinkerPop > Issue Type: Bug > Components: io >Affects Versions: 3.2.1 >Reporter: Chen Xin Yu > > There is description for Serializer in spark: > * Implementations of this trait should implement: > * > * 1. a zero-arg constructor or a constructor that accepts a > [[org.apache.spark.SparkConf]] > * as parameter. If both constructors are defined, the latter takes > precedence. > * > * 2. Java serialization interface. > Class GryoSerializer in Tinkerepop extends Serializer, but does not implement > java.io.Serializable. > It works well before Spark 2.0. But with Spark 2.0, it changed by SPARK-13926 > for Dependency,scala. > Gyro and all its fields must implement Java serialisation interface, > otherwise hundreds of test cases are failed as: > Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: > Task not serializable: java.io.NotSerializableException: > org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer > Serialization stack: > - object not serializable (class: > org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer, value: > org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer@1b12ec8e) > - field (class: org.apache.spark.ShuffleDependency, name: serializer, > type: class org.apache.spark.serializer.Serializer) > - object (class org.apache.spark.ShuffleDependency, > org.apache.spark.ShuffleDependency@7a4f876a) > - field (class: scala.Tuple2, name: _2, type: class java.lang.Object) > - object (class scala.Tuple2, (MapPartitionsRDD[1] at mapToPair at > InputFormatRDD.java:46,org.apache.spark.ShuffleDependency@7a4f876a)) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1426) GryoSerializer should implement Java serialization interface
[ https://issues.apache.org/jira/browse/TINKERPOP-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455720#comment-15455720 ] Chen Xin Yu commented on TINKERPOP-1426: I have a draft fix for this JIRA, I will try to submit a pull request on github. > GryoSerializer should implement Java serialization interface > > > Key: TINKERPOP-1426 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1426 > Project: TinkerPop > Issue Type: Bug > Components: io >Affects Versions: 3.2.1 >Reporter: Chen Xin Yu > > There is description for Serializer in spark: > * Implementations of this trait should implement: > * > * 1. a zero-arg constructor or a constructor that accepts a > [[org.apache.spark.SparkConf]] > * as parameter. If both constructors are defined, the latter takes > precedence. > * > * 2. Java serialization interface. > Class GryoSerializer in Tinkerepop extends Serializer, but does not implement > java.io.Serializable. > It works well before Spark 2.0. But with Spark 2.0, it changed by SPARK-13926 > for Dependency,scala. > Gyro and all its fields must implement Java serialisation interface, > otherwise hundreds of test cases are failed as: > Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: > Task not serializable: java.io.NotSerializableException: > org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer > Serialization stack: > - object not serializable (class: > org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer, value: > org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer@1b12ec8e) > - field (class: org.apache.spark.ShuffleDependency, name: serializer, > type: class org.apache.spark.serializer.Serializer) > - object (class org.apache.spark.ShuffleDependency, > org.apache.spark.ShuffleDependency@7a4f876a) > - field (class: scala.Tuple2, name: _2, type: class java.lang.Object) > - object (class scala.Tuple2, (MapPartitionsRDD[1] at mapToPair at > InputFormatRDD.java:46,org.apache.spark.ShuffleDependency@7a4f876a)) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1389) Support Spark 2.0.0
[ https://issues.apache.org/jira/browse/TINKERPOP-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454211#comment-15454211 ] Chen Xin Yu commented on TINKERPOP-1389: The patch I submitted before is only fixed the issues for compiling. There are hundreds test cases failed as TINKERPOP-1426 > Support Spark 2.0.0 > --- > > Key: TINKERPOP-1389 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1389 > Project: TinkerPop > Issue Type: Improvement > Components: hadoop >Affects Versions: 3.2.1 >Reporter: Chen Xin Yu > Fix For: 3.3.0 > > Attachments: TINKERPOP-1389.patch > > > Spark 2.0.0 was released: > http://spark.apache.org/news/spark-2-0-0-released.html > There are lots of improvement and changes compared to 1.6.1, we should better > bump to it for TinkerPop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TINKERPOP-1426) GryoSerializer should implement Java serialization interface
[ https://issues.apache.org/jira/browse/TINKERPOP-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Xin Yu updated TINKERPOP-1426: --- Description: There is description for Serializer in spark: * Implementations of this trait should implement: * * 1. a zero-arg constructor or a constructor that accepts a [[org.apache.spark.SparkConf]] * as parameter. If both constructors are defined, the latter takes precedence. * * 2. Java serialization interface. Class GryoSerializer in Tinkerepop extends Serializer, but does not implement java.io.Serializable. It works well before Spark 2.0. But with Spark 2.0, it changed by SPARK-13926 for Dependency,scala. Gyro and all its fields must implement Java serialisation interface, otherwise hundreds of test cases are failed as: Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException: org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer Serialization stack: - object not serializable (class: org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer, value: org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer@1b12ec8e) - field (class: org.apache.spark.ShuffleDependency, name: serializer, type: class org.apache.spark.serializer.Serializer) - object (class org.apache.spark.ShuffleDependency, org.apache.spark.ShuffleDependency@7a4f876a) - field (class: scala.Tuple2, name: _2, type: class java.lang.Object) - object (class scala.Tuple2, (MapPartitionsRDD[1] at mapToPair at InputFormatRDD.java:46,org.apache.spark.ShuffleDependency@7a4f876a)) was: In spark description for Serializer, there is: * Implementations of this trait should implement: * * 1. a zero-arg constructor or a constructor that accepts a [[org.apache.spark.SparkConf]] * as parameter. If both constructors are defined, the latter takes precedence. * * 2. Java serialization interface. Tinkerepop Class GryoSerializer extends Serializer, but does not implement serialisation. It works well before Spark 2.0. But with Spark 2.0, with the change for SPARK-13926 for Dependency,scala. Gyro must implement Java serialisation interface, otherwise hundreds of test cases are failed as: Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException: org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer Serialization stack: - object not serializable (class: org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer, value: org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer@1b12ec8e) - field (class: org.apache.spark.ShuffleDependency, name: serializer, type: class org.apache.spark.serializer.Serializer) - object (class org.apache.spark.ShuffleDependency, org.apache.spark.ShuffleDependency@7a4f876a) - field (class: scala.Tuple2, name: _2, type: class java.lang.Object) - object (class scala.Tuple2, (MapPartitionsRDD[1] at mapToPair at InputFormatRDD.java:46,org.apache.spark.ShuffleDependency@7a4f876a)) > GryoSerializer should implement Java serialization interface > > > Key: TINKERPOP-1426 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1426 > Project: TinkerPop > Issue Type: Bug > Components: io >Affects Versions: 3.2.1 >Reporter: Chen Xin Yu > > There is description for Serializer in spark: > * Implementations of this trait should implement: > * > * 1. a zero-arg constructor or a constructor that accepts a > [[org.apache.spark.SparkConf]] > * as parameter. If both constructors are defined, the latter takes > precedence. > * > * 2. Java serialization interface. > Class GryoSerializer in Tinkerepop extends Serializer, but does not implement > java.io.Serializable. > It works well before Spark 2.0. But with Spark 2.0, it changed by SPARK-13926 > for Dependency,scala. > Gyro and all its fields must implement Java serialisation interface, > otherwise hundreds of test cases are failed as: > Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: > Task not serializable: java.io.NotSerializableException: > org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer > Serialization stack: > - object not serializable (class: > org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer, value: > org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer@1b12ec8e) > - field (class: org.apache.spark.ShuffleDependency, name: serializer, > type: class org.apache.spark.serializer.Serializer) > - object (class org.apache.spark.ShuffleDependency, > org.apache.spark.ShuffleDependency@7a4f876a) > -
[jira] [Created] (TINKERPOP-1426) GryoSerializer should implement Java serialization interface
Chen Xin Yu created TINKERPOP-1426: -- Summary: GryoSerializer should implement Java serialization interface Key: TINKERPOP-1426 URL: https://issues.apache.org/jira/browse/TINKERPOP-1426 Project: TinkerPop Issue Type: Bug Components: io Affects Versions: 3.2.1 Reporter: Chen Xin Yu In spark description for Serializer, there is: * Implementations of this trait should implement: * * 1. a zero-arg constructor or a constructor that accepts a [[org.apache.spark.SparkConf]] * as parameter. If both constructors are defined, the latter takes precedence. * * 2. Java serialization interface. Tinkerepop Class GryoSerializer extends Serializer, but does not implement serialisation. It works well before Spark 2.0. But with Spark 2.0, with the change for SPARK-13926 for Dependency,scala. Gyro must implement Java serialisation interface, otherwise hundreds of test cases are failed as: Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException: org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer Serialization stack: - object not serializable (class: org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer, value: org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer@1b12ec8e) - field (class: org.apache.spark.ShuffleDependency, name: serializer, type: class org.apache.spark.serializer.Serializer) - object (class org.apache.spark.ShuffleDependency, org.apache.spark.ShuffleDependency@7a4f876a) - field (class: scala.Tuple2, name: _2, type: class java.lang.Object) - object (class scala.Tuple2, (MapPartitionsRDD[1] at mapToPair at InputFormatRDD.java:46,org.apache.spark.ShuffleDependency@7a4f876a)) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1389) Support Spark 2.0.0
[ https://issues.apache.org/jira/browse/TINKERPOP-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434989#comment-15434989 ] Chen Xin Yu commented on TINKERPOP-1389: Spark 2.0 brings lots of new JIRAs including: https://issues.apache.org/jira/browse/SPARK-12588 https://issues.apache.org/jira/browse/SPARK-13594 https://issues.apache.org/jira/browse/SPARK-4819 These 3 JIRAs caused API change, and tinkerpop 3.x compiled with spark 2.0 would be failed. I fixed the compiling error and submit a patch. > Support Spark 2.0.0 > --- > > Key: TINKERPOP-1389 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1389 > Project: TinkerPop > Issue Type: Improvement > Components: hadoop >Affects Versions: 3.2.1 >Reporter: Chen Xin Yu > Fix For: 3.3.0 > > Attachments: TINKERPOP-1389.patch > > > Spark 2.0.0 was released: > http://spark.apache.org/news/spark-2-0-0-released.html > There are lots of improvement and changes compared to 1.6.1, we should better > bump to it for TinkerPop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TINKERPOP-1389) Support Spark 2.0.0
[ https://issues.apache.org/jira/browse/TINKERPOP-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Xin Yu updated TINKERPOP-1389: --- Attachment: TINKERPOP-1389.patch > Support Spark 2.0.0 > --- > > Key: TINKERPOP-1389 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1389 > Project: TinkerPop > Issue Type: Improvement > Components: hadoop >Affects Versions: 3.2.1 >Reporter: Chen Xin Yu > Fix For: 3.3.0 > > Attachments: TINKERPOP-1389.patch > > > Spark 2.0.0 was released: > http://spark.apache.org/news/spark-2-0-0-released.html > There are lots of improvement and changes compared to 1.6.1, we should better > bump to it for TinkerPop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
