[jira] [Commented] (TINKERPOP-1301) Provide Javadoc for ScriptInput/OutputFormat's
[ https://issues.apache.org/jira/browse/TINKERPOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15315071#comment-15315071 ] ASF GitHub Bot commented on TINKERPOP-1301: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-tinkerpop/pull/327 > Provide Javadoc for ScriptInput/OutputFormat's > -- > > Key: TINKERPOP-1301 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1301 > Project: TinkerPop > Issue Type: Improvement > Components: documentation >Affects Versions: 3.2.0-incubating >Reporter: Lewis John McGibbney >Priority: Minor > > Right now > [ScriptInputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptInputFormat.html] > and > [ScriptOutputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptOutputFormat.html] > are not documented. Descriptions are however present over on some old > [TitanDB > documentation|http://s3.thinkaurelius.com/docs/titan/0.5.0/script-io-format.html] > which can be used to provide Javadoc level documentation for developers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-tinkerpop pull request #327: TINKERPOP-1301 Provide Javadoc for Sc...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-tinkerpop/pull/327 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1317) IoGraphTest throws error: URI is not hierarchical
[
https://issues.apache.org/jira/browse/TINKERPOP-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314949#comment-15314949
]
ASF GitHub Bot commented on TINKERPOP-1317:
---
Github user twilmes commented on a diff in the pull request:
https://github.com/apache/incubator-tinkerpop/pull/328#discussion_r65783485
--- Diff: docs/src/dev/provider/index.asciidoc ---
@@ -505,6 +505,11 @@ off for the test suite (the Maven SureFire Plugin is
configured this way by defa
include this setting, `false`, in the SureFire
configuration if tests are failing in an
unexplainable way.
+TIP: When running the `gremlin-test` suite against your implementation,
you may need to set `build.dir` as an
+envrionment variable, depending on your project layout. Some tests require
this to find a writable directory for
--- End diff --
Tiny little typo here: "envrionment" -> "environment"
> IoGraphTest throws error: URI is not hierarchical
> -
>
> Key: TINKERPOP-1317
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1317
> Project: TinkerPop
> Issue Type: Bug
> Components: test-suite
>Affects Versions: 3.2.0-incubating, 3.1.2-incubating
>Reporter: Jason Plurad
>Assignee: Jason Plurad
>Priority: Minor
>
> Via https://groups.google.com/d/msg/gremlin-users/aap3pxZtGyU/t-eOC6ZyAAAJ
> These methods will fail for graph providers that are trying to implement the
> test suite:
> * {{IoGraphTest.shouldReadWriteModernToFileWithHelpers()}}
> * {{IoGraphTest.shouldReadWriteClassicToFileWithHelpers()}}
> Example stack trace:
> {noformat}
> shouldReadWriteModernToFileWithHelpers[gryo](org.apache.tinkerpop.gremlin.structure.io.IoGraphTest)
> Time elapsed: 0.004 sec <<< ERROR!
> java.lang.RuntimeException: Unable to computePath for class
> org.apache.tinkerpop.gremlin.structure.io.IoGraphTest
> at
> org.apache.tinkerpop.gremlin.TestHelper.computePath(TestHelper.java:117)
> at
> org.apache.tinkerpop.gremlin.TestHelper.getRootOfBuildDirectory(TestHelper.java:105)
> at
> org.apache.tinkerpop.gremlin.TestHelper.makeTestDataPath(TestHelper.java:70)
> at
> org.apache.tinkerpop.gremlin.TestHelper.generateTempFile(TestHelper.java:127)
> at
> org.apache.tinkerpop.gremlin.structure.io.IoGraphTest.shouldReadWriteModernToFileWithHelpers(IoGraphTest.java:164)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at
> org.apache.tinkerpop.gremlin.AbstractGremli
[GitHub] incubator-tinkerpop pull request #328: TINKERPOP-1317: use graph class as ro...
Github user twilmes commented on a diff in the pull request: https://github.com/apache/incubator-tinkerpop/pull/328#discussion_r65783485 --- Diff: docs/src/dev/provider/index.asciidoc --- @@ -505,6 +505,11 @@ off for the test suite (the Maven SureFire Plugin is configured this way by defa include this setting, `false`, in the SureFire configuration if tests are failing in an unexplainable way. +TIP: When running the `gremlin-test` suite against your implementation, you may need to set `build.dir` as an +envrionment variable, depending on your project layout. Some tests require this to find a writable directory for --- End diff -- Tiny little typo here: "envrionment" -> "environment" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1317) IoGraphTest throws error: URI is not hierarchical
[
https://issues.apache.org/jira/browse/TINKERPOP-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314937#comment-15314937
]
ASF GitHub Bot commented on TINKERPOP-1317:
---
Github user analytically commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/328
Genius PR.
> IoGraphTest throws error: URI is not hierarchical
> -
>
> Key: TINKERPOP-1317
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1317
> Project: TinkerPop
> Issue Type: Bug
> Components: test-suite
>Affects Versions: 3.2.0-incubating, 3.1.2-incubating
>Reporter: Jason Plurad
>Assignee: Jason Plurad
>Priority: Minor
>
> Via https://groups.google.com/d/msg/gremlin-users/aap3pxZtGyU/t-eOC6ZyAAAJ
> These methods will fail for graph providers that are trying to implement the
> test suite:
> * {{IoGraphTest.shouldReadWriteModernToFileWithHelpers()}}
> * {{IoGraphTest.shouldReadWriteClassicToFileWithHelpers()}}
> Example stack trace:
> {noformat}
> shouldReadWriteModernToFileWithHelpers[gryo](org.apache.tinkerpop.gremlin.structure.io.IoGraphTest)
> Time elapsed: 0.004 sec <<< ERROR!
> java.lang.RuntimeException: Unable to computePath for class
> org.apache.tinkerpop.gremlin.structure.io.IoGraphTest
> at
> org.apache.tinkerpop.gremlin.TestHelper.computePath(TestHelper.java:117)
> at
> org.apache.tinkerpop.gremlin.TestHelper.getRootOfBuildDirectory(TestHelper.java:105)
> at
> org.apache.tinkerpop.gremlin.TestHelper.makeTestDataPath(TestHelper.java:70)
> at
> org.apache.tinkerpop.gremlin.TestHelper.generateTempFile(TestHelper.java:127)
> at
> org.apache.tinkerpop.gremlin.structure.io.IoGraphTest.shouldReadWriteModernToFileWithHelpers(IoGraphTest.java:164)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at
> org.apache.tinkerpop.gremlin.AbstractGremlinSuite.runChild(AbstractGremlinSuite.java:212)
> at
> org.apache.tinkerpop.gremlin.AbstractGremlinSuite.runChild(AbstractGremlinSuite.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.mave
[GitHub] incubator-tinkerpop issue #328: TINKERPOP-1317: use graph class as root clas...
Github user analytically commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/328 Genius PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1317) IoGraphTest throws error: URI is not hierarchical
[
https://issues.apache.org/jira/browse/TINKERPOP-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314925#comment-15314925
]
ASF GitHub Bot commented on TINKERPOP-1317:
---
Github user dkuppitz commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/328
VOTE: +1
> IoGraphTest throws error: URI is not hierarchical
> -
>
> Key: TINKERPOP-1317
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1317
> Project: TinkerPop
> Issue Type: Bug
> Components: test-suite
>Affects Versions: 3.2.0-incubating, 3.1.2-incubating
>Reporter: Jason Plurad
>Assignee: Jason Plurad
>Priority: Minor
>
> Via https://groups.google.com/d/msg/gremlin-users/aap3pxZtGyU/t-eOC6ZyAAAJ
> These methods will fail for graph providers that are trying to implement the
> test suite:
> * {{IoGraphTest.shouldReadWriteModernToFileWithHelpers()}}
> * {{IoGraphTest.shouldReadWriteClassicToFileWithHelpers()}}
> Example stack trace:
> {noformat}
> shouldReadWriteModernToFileWithHelpers[gryo](org.apache.tinkerpop.gremlin.structure.io.IoGraphTest)
> Time elapsed: 0.004 sec <<< ERROR!
> java.lang.RuntimeException: Unable to computePath for class
> org.apache.tinkerpop.gremlin.structure.io.IoGraphTest
> at
> org.apache.tinkerpop.gremlin.TestHelper.computePath(TestHelper.java:117)
> at
> org.apache.tinkerpop.gremlin.TestHelper.getRootOfBuildDirectory(TestHelper.java:105)
> at
> org.apache.tinkerpop.gremlin.TestHelper.makeTestDataPath(TestHelper.java:70)
> at
> org.apache.tinkerpop.gremlin.TestHelper.generateTempFile(TestHelper.java:127)
> at
> org.apache.tinkerpop.gremlin.structure.io.IoGraphTest.shouldReadWriteModernToFileWithHelpers(IoGraphTest.java:164)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at
> org.apache.tinkerpop.gremlin.AbstractGremlinSuite.runChild(AbstractGremlinSuite.java:212)
> at
> org.apache.tinkerpop.gremlin.AbstractGremlinSuite.runChild(AbstractGremlinSuite.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.sure
[GitHub] incubator-tinkerpop issue #328: TINKERPOP-1317: use graph class as root clas...
Github user dkuppitz commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/328 VOTE: +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (TINKERPOP-1325) Query results with lambda step not serializable in OLAP
[
https://issues.apache.org/jira/browse/TINKERPOP-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stephen mallette updated TINKERPOP-1325:
Affects Version/s: (was: 3.2.1)
3.2.0-incubating
Component/s: (was: tinkergraph)
server
hadoop
> Query results with lambda step not serializable in OLAP
> ---
>
> Key: TINKERPOP-1325
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1325
> Project: TinkerPop
> Issue Type: Bug
> Components: hadoop, server
>Affects Versions: 3.2.0-incubating
> Environment: OSX 10.11.5
> apache-gremlin-console-3.2.1
>Reporter: Sergey Gmyzov
>
> Query with lambda step fails in OLAP mode:
> {code}
> gremlin> :remote config alias g ds232.a
> ==>g=ds232.a
> gremlin> :>
> schema.config().option('graph.traversal_sources.g.restrict_lambda').set(false);
> ==>null
> gremlin> :>
> schema.config().option('graph.traversal_sources.a.restrict_lambda').set(false);
> ==>null
> gremlin> :> graph.tx().commit();
> ==>null
> gremlin> :> g.V().has("movieId","m267").map{it.get().value("title")};
> org.apache.spark.SparkException: Task not serializable
> Display stack trace? [yN] y
> org.apache.tinkerpop.gremlin.groovy.plugin.RemoteException:
> org.apache.spark.SparkException: Task not serializable
> at
> org.apache.tinkerpop.gremlin.console.groovy.plugin.DriverRemoteAcceptor.submit(DriverRemoteAcceptor.java:156)
> at
> org.apache.tinkerpop.gremlin.console.commands.SubmitCommand.execute(SubmitCommand.groovy:41)
> at org.codehaus.groovy.tools.shell.Shell.execute(Shell.groovy:104)
> at
> org.codehaus.groovy.tools.shell.Groovysh.super$2$execute(Groovysh.groovy)
> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
> at
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
> at
> org.codehaus.groovy.tools.shell.Groovysh.executeCommand(Groovysh.groovy:259)
> at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:158)
> at
> org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
> at
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
> at
> org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:63)
> at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
> at
> org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
> at
> org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
> at
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
> at
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
> at
> org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
> at
> org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
> at
> org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
[jira] [Updated] (TINKERPOP-1325) Query results with lambda step not serializable in OLAP
[
https://issues.apache.org/jira/browse/TINKERPOP-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Gmyzov updated TINKERPOP-1325:
-
Description:
Query with lambda step fails in OLAP mode:
{code}
gremlin> :remote config alias g ds232.a
==>g=ds232.a
gremlin> :>
schema.config().option('graph.traversal_sources.g.restrict_lambda').set(false);
==>null
gremlin> :>
schema.config().option('graph.traversal_sources.a.restrict_lambda').set(false);
==>null
gremlin> :> graph.tx().commit();
==>null
gremlin> :> g.V().has("movieId","m267").map{it.get().value("title")};
org.apache.spark.SparkException: Task not serializable
Display stack trace? [yN] y
org.apache.tinkerpop.gremlin.groovy.plugin.RemoteException:
org.apache.spark.SparkException: Task not serializable
at
org.apache.tinkerpop.gremlin.console.groovy.plugin.DriverRemoteAcceptor.submit(DriverRemoteAcceptor.java:156)
at
org.apache.tinkerpop.gremlin.console.commands.SubmitCommand.execute(SubmitCommand.groovy:41)
at org.codehaus.groovy.tools.shell.Shell.execute(Shell.groovy:104)
at
org.codehaus.groovy.tools.shell.Groovysh.super$2$execute(Groovysh.groovy)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at
org.codehaus.groovy.tools.shell.Groovysh.executeCommand(Groovysh.groovy:259)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:158)
at
org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at
org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:63)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
at
org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
at
org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at
org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
at
org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
at
org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at
org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)
a
[jira] [Created] (TINKERPOP-1325) Query results with lambda step not serializable in OLAP
Sergey Gmyzov created TINKERPOP-1325:
Summary: Query results with lambda step not serializable in OLAP
Key: TINKERPOP-1325
URL: https://issues.apache.org/jira/browse/TINKERPOP-1325
Project: TinkerPop
Issue Type: Bug
Components: tinkergraph
Affects Versions: 3.2.1
Environment: OSX 10.11.5
apache-gremlin-console-3.2.1
Reporter: Sergey Gmyzov
Query with lambda step fails in OLAP mode:
{panel}
gremlin> :remote config alias g ds232.a
==>g=ds232.a
gremlin> :>
schema.config().option('graph.traversal_sources.g.restrict_lambda').set(false);
==>null
gremlin> :>
schema.config().option('graph.traversal_sources.a.restrict_lambda').set(false);
==>null
gremlin> :> graph.tx().commit();
==>null
gremlin> :> g.V().has("movieId","m267").map{it.get().value("title")};
org.apache.spark.SparkException: Task not serializable
Display stack trace? [yN] y
org.apache.tinkerpop.gremlin.groovy.plugin.RemoteException:
org.apache.spark.SparkException: Task not serializable
at
org.apache.tinkerpop.gremlin.console.groovy.plugin.DriverRemoteAcceptor.submit(DriverRemoteAcceptor.java:156)
at
org.apache.tinkerpop.gremlin.console.commands.SubmitCommand.execute(SubmitCommand.groovy:41)
at org.codehaus.groovy.tools.shell.Shell.execute(Shell.groovy:104)
at
org.codehaus.groovy.tools.shell.Groovysh.super$2$execute(Groovysh.groovy)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at
org.codehaus.groovy.tools.shell.Groovysh.executeCommand(Groovysh.groovy:259)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:158)
at
org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at
org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:63)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
at
org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
at
org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at
org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
at
org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
at
org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1212)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSu
[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo
[
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314682#comment-15314682
]
ASF GitHub Bot commented on TINKERPOP-1321:
---
Github user dalaro commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/325
Additional notes from Marko:
> okay, so as long as there is never a need for MyRegistrar, then can you
please update your PR accordingly?
> Finally, call it GryoRegistrar (not TinkerPopRegistrar) as this has to do
with Gryo IORegistry classes.
> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
> Issue Type: Improvement
> Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting
> spark.serializer=GryoSerializer. This makes GryoSerializer responsible for
> serializing not just TinkerPop types but also scala runtime types and Spark
> internals. GryoSerializer doesn't extend either of the two serializers
> provided by Spark. It effectively assumes responsibility for reimplementing
> them.
> This is problematic. It is not totally trivial to replicate the
> functionality of Spark's standard serializers. It is also not easy to
> empirically test all meaningful cases. For instance, there is a conditional
> within Spark that selects between two concrete Map implementations depending
> on whether the current RDD partition count exceeds 2k
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
> The implementation used below this threshold serializes fine on
> GryoSerializer. The implementation used above the threshold does not. Above
> the partition threshold, I've started getting
> {{org.apache.spark.SparkException: Job aborted due to stage failure:
> Exception while getting task result:
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed
> to find one of the right cookies.}} Google leads to
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64. However, just
> switching to Spark's {{KryoSerializer}} without changing anything somehow
> fixes the problem in my environment, implying that Spark has done something
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.
> These still serialize, but KryoSerializer uses its default behavior
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's
> StarGraphSerializer. TinkerPop's reliance on its own internal shaded Kryo
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just
> by tweaking the configuration. Besides spark.serializer, there is also a
> setting spark.closure.serializer for which the only supported value is
> JavaSerializer. Key TP classes that make it into the object reference graphs
> of Spark closures implement Serializable by resorting to TinkerPop's shaded
> Kryo via HadoopPools (looking at Object/VertexWritable). This leads to
> surprises with custom property data types. It doesn't matter if those types
> implement Serializable, and it doesn't matter if Spark's KryoSerializer is
> configured to accept those types. If those types are reachable from
> Object/VertexWritable, then they must be registered with TinkerPop's internal
> shaded Kryo, or else it will choke on them (unless it was explicitly
> configured to allow unregistered classes).
> I suggest the following change to give users more flexibility in their choice
> of spark.serializer, and to allow them to reuse TinkerPop's serializers if
> they choose not to use GryoSerializer: introduce lightweight interfaces that
> decouple TinkerPop's serialization logic from the exact Kryo shaded/unshaded
> package doing the work. TinkerPop's serialization logic would be written
> against interfaces that replicate a minimal subset of Kryo, and then TP's
> shaded Kryo or Spark's unshaded Kryo could be plugged in underneath without
> having to touch the source, recompile any TinkerPop code, or munge bytecode
> at runtime.
> This would not resolve all of the potential problems/complexity around
> TinkerPop serialization, but it would make it possible for users to apply the
> spark.serializer of their ch
[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...
Github user dalaro commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/325 Additional notes from Marko: > okay, so as long as there is never a need for MyRegistrar, then can you please update your PR accordingly? > Finally, call it GryoRegistrar (not TinkerPopRegistrar) as this has to do with Gryo IORegistry classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...
Github user dalaro commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/325 After discussion on Slack, I think @okram and I tentatively agreed to proceed with this PR after I do additional work to save users who have custom serializers the effort of maintaining separate `IoRegistry` and `spark.kryo.registrator` implementations with near-identical contents. I may discover other complications during the process, but I think this involves two changes: 1. I will attempt to subclass KryoSerializer so that I can access the SparkConf passed to its constructior and check for `GryoPool.CONFIG_IO_REGISTRY` (similar to what GryoSerializer does now), then apply any registrations found therein so long as each registration either: * specifies no explicit serializer (using Kryo's internal default) or * specifies a shim serializer In particular, if the registry contains an old-style TP shaded Kryo serializer that hasn't been ported to the shim, then the KryoSerializer subclass will throw an exception. This change is necessary to support custom-serialized, `IoRegistry`-listed datatypes that appear in most Spark data outside of closures (say, in the RDD itself). 2. I will replace current callsites of `HadoopPools.initialize(conf)` with some other static method that calls `HadoopPools.initialize(conf)` and then calls some roughly equivalent `initialize(conf)` method for the static KryoPool backing `KryoPoolShimService`. This change is necessary to support custom-serialized, `IoRegistry`-listed datatypes that appear in closures. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo
[
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314655#comment-15314655
]
ASF GitHub Bot commented on TINKERPOP-1321:
---
Github user dalaro commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/325
After discussion on Slack, I think @okram and I tentatively agreed to
proceed with this PR after I do additional work to save users who have custom
serializers the effort of maintaining separate `IoRegistry` and
`spark.kryo.registrator` implementations with near-identical contents.
I may discover other complications during the process, but I think this
involves two changes:
1. I will attempt to subclass KryoSerializer so that I can access the
SparkConf passed to its constructior and check for
`GryoPool.CONFIG_IO_REGISTRY` (similar to what GryoSerializer does now), then
apply any registrations found therein so long as each registration either:
* specifies no explicit serializer (using Kryo's internal default) or
* specifies a shim serializer
In particular, if the registry contains an old-style TP shaded Kryo
serializer that hasn't been ported to the shim, then the KryoSerializer
subclass will throw an exception.
This change is necessary to support custom-serialized,
`IoRegistry`-listed datatypes that appear in most Spark data outside of
closures (say, in the RDD itself).
2. I will replace current callsites of `HadoopPools.initialize(conf)` with
some other static method that calls `HadoopPools.initialize(conf)` and then
calls some roughly equivalent `initialize(conf)` method for the static KryoPool
backing `KryoPoolShimService`.
This change is necessary to support custom-serialized,
`IoRegistry`-listed datatypes that appear in closures.
> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
> Issue Type: Improvement
> Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting
> spark.serializer=GryoSerializer. This makes GryoSerializer responsible for
> serializing not just TinkerPop types but also scala runtime types and Spark
> internals. GryoSerializer doesn't extend either of the two serializers
> provided by Spark. It effectively assumes responsibility for reimplementing
> them.
> This is problematic. It is not totally trivial to replicate the
> functionality of Spark's standard serializers. It is also not easy to
> empirically test all meaningful cases. For instance, there is a conditional
> within Spark that selects between two concrete Map implementations depending
> on whether the current RDD partition count exceeds 2k
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
> The implementation used below this threshold serializes fine on
> GryoSerializer. The implementation used above the threshold does not. Above
> the partition threshold, I've started getting
> {{org.apache.spark.SparkException: Job aborted due to stage failure:
> Exception while getting task result:
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed
> to find one of the right cookies.}} Google leads to
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64. However, just
> switching to Spark's {{KryoSerializer}} without changing anything somehow
> fixes the problem in my environment, implying that Spark has done something
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.
> These still serialize, but KryoSerializer uses its default behavior
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's
> StarGraphSerializer. TinkerPop's reliance on its own internal shaded Kryo
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just
> by tweaking the configuration. Besides spark.serializer, there is also a
> setting spark.closure.serializer for which the only supported value is
> JavaSerializer. Key TP classes that make it into the object reference graphs
> of Spark closures implement Serializable by resorting to TinkerPop's shaded
> Kryo via HadoopPools (looking at Ob
[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo
[
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314517#comment-15314517
]
ASF GitHub Bot commented on TINKERPOP-1321:
---
Github user okram commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/325
@dalaro -- what about it doesn't work? Meaning, is your problem just a
"unregistered class" issue? If so, we just register it. If I have a "bad"
serializer for one of the registered classes, then we can fix it?
See the classes that I've registered in `GryoSerializer` (The last 7 are
TinkerPop specific.):
```java
builder.addCustom(Tuple2.class, new
Tuple2Serializer())
.addCustom(Tuple2[].class)
.addCustom(Tuple3.class, new
Tuple3Serializer())
.addCustom(Tuple3[].class)
.addCustom(CompactBuffer.class, new
CompactBufferSerializer())
.addCustom(CompactBuffer[].class)
.addCustom(CompressedMapStatus.class)
.addCustom(BlockManagerId.class)
.addCustom(HighlyCompressedMapStatus.class,
new ExternalizableSerializer())
.addCustom(HttpBroadcast.class)
.addCustom(PythonBroadcast.class)
.addCustom(BoxedUnit.class)
.addCustom(Class.forName("scala.reflect.ClassTag$$anon$1"), new
JavaSerializer())
.addCustom(Class.forName("scala.reflect.ManifestFactory$$anon$1"), new
JavaSerializer())
.addCustom(WrappedArray.ofRef.class, new
WrappedArraySerializer())
.addCustom(MessagePayload.class)
.addCustom(ViewIncomingPayload.class)
.addCustom(ViewOutgoingPayload.class)
.addCustom(ViewPayload.class)
.addCustom(SerializableConfiguration.class,
new JavaSerializer())
.addCustom(VertexWritable.class, new
VertexWritableSerializer())
.addCustom(ObjectWritable.class, new
ObjectWritableSerializer())
```
> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
> Issue Type: Improvement
> Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting
> spark.serializer=GryoSerializer. This makes GryoSerializer responsible for
> serializing not just TinkerPop types but also scala runtime types and Spark
> internals. GryoSerializer doesn't extend either of the two serializers
> provided by Spark. It effectively assumes responsibility for reimplementing
> them.
> This is problematic. It is not totally trivial to replicate the
> functionality of Spark's standard serializers. It is also not easy to
> empirically test all meaningful cases. For instance, there is a conditional
> within Spark that selects between two concrete Map implementations depending
> on whether the current RDD partition count exceeds 2k
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
> The implementation used below this threshold serializes fine on
> GryoSerializer. The implementation used above the threshold does not. Above
> the partition threshold, I've started getting
> {{org.apache.spark.SparkException: Job aborted due to stage failure:
> Exception while getting task result:
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed
> to find one of the right cookies.}} Google leads to
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64. However, just
> switching to Spark's {{KryoSerializer}} without changing anything somehow
> fixes the problem in my environment, implying that Spark has done something
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.
> These still serialize, but KryoSerializer uses its default behavior
> (Field
[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...
Github user okram commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/325
@dalaro -- what about it doesn't work? Meaning, is your problem just a
"unregistered class" issue? If so, we just register it. If I have a "bad"
serializer for one of the registered classes, then we can fix it?
See the classes that I've registered in `GryoSerializer` (The last 7 are
TinkerPop specific.):
```java
builder.addCustom(Tuple2.class, new
Tuple2Serializer())
.addCustom(Tuple2[].class)
.addCustom(Tuple3.class, new
Tuple3Serializer())
.addCustom(Tuple3[].class)
.addCustom(CompactBuffer.class, new
CompactBufferSerializer())
.addCustom(CompactBuffer[].class)
.addCustom(CompressedMapStatus.class)
.addCustom(BlockManagerId.class)
.addCustom(HighlyCompressedMapStatus.class,
new ExternalizableSerializer())
.addCustom(HttpBroadcast.class)
.addCustom(PythonBroadcast.class)
.addCustom(BoxedUnit.class)
.addCustom(Class.forName("scala.reflect.ClassTag$$anon$1"), new
JavaSerializer())
.addCustom(Class.forName("scala.reflect.ManifestFactory$$anon$1"), new
JavaSerializer())
.addCustom(WrappedArray.ofRef.class, new
WrappedArraySerializer())
.addCustom(MessagePayload.class)
.addCustom(ViewIncomingPayload.class)
.addCustom(ViewOutgoingPayload.class)
.addCustom(ViewPayload.class)
.addCustom(SerializableConfiguration.class,
new JavaSerializer())
.addCustom(VertexWritable.class, new
VertexWritableSerializer())
.addCustom(ObjectWritable.class, new
ObjectWritableSerializer())
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo
[
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314508#comment-15314508
]
ASF GitHub Bot commented on TINKERPOP-1321:
---
Github user dalaro commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/325
The problem is that GryoSerializer doesn't actually work. This is not the
first time it has shown behavior on Spark or scala runtime classes which
diverges from the behavior of Spark's KryoSerializer/JavaSerializer.
I realize that asking users to write custom serializers against the shim
(to be compatible with everything) in the long term or to duplicate
registrations in the short term (if they want to use Gryo in one place and
Spark graph computation in another, using the same custom types) is not ideal,
but I think the status quo is even worse.
> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
> Issue Type: Improvement
> Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting
> spark.serializer=GryoSerializer. This makes GryoSerializer responsible for
> serializing not just TinkerPop types but also scala runtime types and Spark
> internals. GryoSerializer doesn't extend either of the two serializers
> provided by Spark. It effectively assumes responsibility for reimplementing
> them.
> This is problematic. It is not totally trivial to replicate the
> functionality of Spark's standard serializers. It is also not easy to
> empirically test all meaningful cases. For instance, there is a conditional
> within Spark that selects between two concrete Map implementations depending
> on whether the current RDD partition count exceeds 2k
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
> The implementation used below this threshold serializes fine on
> GryoSerializer. The implementation used above the threshold does not. Above
> the partition threshold, I've started getting
> {{org.apache.spark.SparkException: Job aborted due to stage failure:
> Exception while getting task result:
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed
> to find one of the right cookies.}} Google leads to
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64. However, just
> switching to Spark's {{KryoSerializer}} without changing anything somehow
> fixes the problem in my environment, implying that Spark has done something
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.
> These still serialize, but KryoSerializer uses its default behavior
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's
> StarGraphSerializer. TinkerPop's reliance on its own internal shaded Kryo
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just
> by tweaking the configuration. Besides spark.serializer, there is also a
> setting spark.closure.serializer for which the only supported value is
> JavaSerializer. Key TP classes that make it into the object reference graphs
> of Spark closures implement Serializable by resorting to TinkerPop's shaded
> Kryo via HadoopPools (looking at Object/VertexWritable). This leads to
> surprises with custom property data types. It doesn't matter if those types
> implement Serializable, and it doesn't matter if Spark's KryoSerializer is
> configured to accept those types. If those types are reachable from
> Object/VertexWritable, then they must be registered with TinkerPop's internal
> shaded Kryo, or else it will choke on them (unless it was explicitly
> configured to allow unregistered classes).
> I suggest the following change to give users more flexibility in their choice
> of spark.serializer, and to allow them to reuse TinkerPop's serializers if
> they choose not to use GryoSerializer: introduce lightweight interfaces that
> decouple TinkerPop's serialization logic from the exact Kryo shaded/unshaded
> package doing the work. TinkerPop's serialization logic would be written
> against interfaces that replicate a minimal subset of Kryo, and then TP's
> shaded Kryo or Spark's unshaded Kryo could be
[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...
Github user dalaro commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/325 The problem is that GryoSerializer doesn't actually work. This is not the first time it has shown behavior on Spark or scala runtime classes which diverges from the behavior of Spark's KryoSerializer/JavaSerializer. I realize that asking users to write custom serializers against the shim (to be compatible with everything) in the long term or to duplicate registrations in the short term (if they want to use Gryo in one place and Spark graph computation in another, using the same custom types) is not ideal, but I think the status quo is even worse. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo
[
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314503#comment-15314503
]
ASF GitHub Bot commented on TINKERPOP-1321:
---
Github user okram commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/325
So yea... sorta feels like we are where we were before in a way. And yea,
all that Input, Output, Kryo stuff is what causes all the problems.
So now I'm wondering, why not just register the requisite classes with
`GryoSerializer`? Yes, we miss one every now and then, but we add it. Sooner or
later they will all be there. Moreover, if a person needs it now, they can just
register it with `GryoMapper`. Did you try just registering your "high density
map" w/ `GryoSerializer`?
I know that last paragraph sucks given all the work you did, but having two
ways of doing something and one way being Spark-specific doesn't make me happy
and this is why `GryoSerializer` came into being..
> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
> Issue Type: Improvement
> Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting
> spark.serializer=GryoSerializer. This makes GryoSerializer responsible for
> serializing not just TinkerPop types but also scala runtime types and Spark
> internals. GryoSerializer doesn't extend either of the two serializers
> provided by Spark. It effectively assumes responsibility for reimplementing
> them.
> This is problematic. It is not totally trivial to replicate the
> functionality of Spark's standard serializers. It is also not easy to
> empirically test all meaningful cases. For instance, there is a conditional
> within Spark that selects between two concrete Map implementations depending
> on whether the current RDD partition count exceeds 2k
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
> The implementation used below this threshold serializes fine on
> GryoSerializer. The implementation used above the threshold does not. Above
> the partition threshold, I've started getting
> {{org.apache.spark.SparkException: Job aborted due to stage failure:
> Exception while getting task result:
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed
> to find one of the right cookies.}} Google leads to
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64. However, just
> switching to Spark's {{KryoSerializer}} without changing anything somehow
> fixes the problem in my environment, implying that Spark has done something
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.
> These still serialize, but KryoSerializer uses its default behavior
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's
> StarGraphSerializer. TinkerPop's reliance on its own internal shaded Kryo
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just
> by tweaking the configuration. Besides spark.serializer, there is also a
> setting spark.closure.serializer for which the only supported value is
> JavaSerializer. Key TP classes that make it into the object reference graphs
> of Spark closures implement Serializable by resorting to TinkerPop's shaded
> Kryo via HadoopPools (looking at Object/VertexWritable). This leads to
> surprises with custom property data types. It doesn't matter if those types
> implement Serializable, and it doesn't matter if Spark's KryoSerializer is
> configured to accept those types. If those types are reachable from
> Object/VertexWritable, then they must be registered with TinkerPop's internal
> shaded Kryo, or else it will choke on them (unless it was explicitly
> configured to allow unregistered classes).
> I suggest the following change to give users more flexibility in their choice
> of spark.serializer, and to allow them to reuse TinkerPop's serializers if
> they choose not to use GryoSerializer: introduce lightweight interfaces that
> decouple TinkerPop's serialization logic from the exact Kryo shaded/unshaded
> package doing the work. TinkerPop's serialization
[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...
Github user okram commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/325 So yea... sorta feels like we are where we were before in a way. And yea, all that Input, Output, Kryo stuff is what causes all the problems. So now I'm wondering, why not just register the requisite classes with `GryoSerializer`? Yes, we miss one every now and then, but we add it. Sooner or later they will all be there. Moreover, if a person needs it now, they can just register it with `GryoMapper`. Did you try just registering your "high density map" w/ `GryoSerializer`? I know that last paragraph sucks given all the work you did, but having two ways of doing something and one way being Spark-specific doesn't make me happy and this is why `GryoSerializer` came into being.. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo
[
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314456#comment-15314456
]
ASF GitHub Bot commented on TINKERPOP-1321:
---
Github user dalaro commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/325
BTW, in case you're wondering why I think it's hard to create proxies,
recall that o.a.t.shaded.kryo.{Kryo,io.Input,io.Output} are all classes, not
interfaces. There are a lot of approaches that would make this work, but the
ones I can think of suck for various reasons.
> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
> Issue Type: Improvement
> Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting
> spark.serializer=GryoSerializer. This makes GryoSerializer responsible for
> serializing not just TinkerPop types but also scala runtime types and Spark
> internals. GryoSerializer doesn't extend either of the two serializers
> provided by Spark. It effectively assumes responsibility for reimplementing
> them.
> This is problematic. It is not totally trivial to replicate the
> functionality of Spark's standard serializers. It is also not easy to
> empirically test all meaningful cases. For instance, there is a conditional
> within Spark that selects between two concrete Map implementations depending
> on whether the current RDD partition count exceeds 2k
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
> The implementation used below this threshold serializes fine on
> GryoSerializer. The implementation used above the threshold does not. Above
> the partition threshold, I've started getting
> {{org.apache.spark.SparkException: Job aborted due to stage failure:
> Exception while getting task result:
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed
> to find one of the right cookies.}} Google leads to
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64. However, just
> switching to Spark's {{KryoSerializer}} without changing anything somehow
> fixes the problem in my environment, implying that Spark has done something
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.
> These still serialize, but KryoSerializer uses its default behavior
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's
> StarGraphSerializer. TinkerPop's reliance on its own internal shaded Kryo
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just
> by tweaking the configuration. Besides spark.serializer, there is also a
> setting spark.closure.serializer for which the only supported value is
> JavaSerializer. Key TP classes that make it into the object reference graphs
> of Spark closures implement Serializable by resorting to TinkerPop's shaded
> Kryo via HadoopPools (looking at Object/VertexWritable). This leads to
> surprises with custom property data types. It doesn't matter if those types
> implement Serializable, and it doesn't matter if Spark's KryoSerializer is
> configured to accept those types. If those types are reachable from
> Object/VertexWritable, then they must be registered with TinkerPop's internal
> shaded Kryo, or else it will choke on them (unless it was explicitly
> configured to allow unregistered classes).
> I suggest the following change to give users more flexibility in their choice
> of spark.serializer, and to allow them to reuse TinkerPop's serializers if
> they choose not to use GryoSerializer: introduce lightweight interfaces that
> decouple TinkerPop's serialization logic from the exact Kryo shaded/unshaded
> package doing the work. TinkerPop's serialization logic would be written
> against interfaces that replicate a minimal subset of Kryo, and then TP's
> shaded Kryo or Spark's unshaded Kryo could be plugged in underneath without
> having to touch the source, recompile any TinkerPop code, or munge bytecode
> at runtime.
> This would not resolve all of the potential problems/complexity around
> TinkerPop serialization, but it would make it possible for users to apply the
> spark.serial
[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...
Github user dalaro commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/325
BTW, in case you're wondering why I think it's hard to create proxies,
recall that o.a.t.shaded.kryo.{Kryo,io.Input,io.Output} are all classes, not
interfaces. There are a lot of approaches that would make this work, but the
ones I can think of suck for various reasons.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (TINKERPOP-1318) java.lang.NoSuchMethodError: org/hamcrest/Matcher.describeMismatch
[
https://issues.apache.org/jira/browse/TINKERPOP-1318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314449#comment-15314449
]
ASF GitHub Bot commented on TINKERPOP-1318:
---
Github user okram commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/329
VOTE +1.
> java.lang.NoSuchMethodError: org/hamcrest/Matcher.describeMismatch
> --
>
> Key: TINKERPOP-1318
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1318
> Project: TinkerPop
> Issue Type: Bug
> Components: test-suite
>Affects Versions: 3.2.0-incubating, 3.1.2-incubating
>Reporter: Jason Plurad
>Assignee: Jason Plurad
>Priority: Minor
>
> I don't recall specifically how to make this fail with {{gremlin-test}}, but
> I did run into it at one point when writing a graph implementation. This blog
> describes the issue and workaround.
> https://tedvinke.wordpress.com/2013/12/17/mixing-junit-hamcrest-and-mockito-explaining-nosuchmethoderror/
> The error trace looks like this:
> {noformat}
> java.lang.NoSuchMethodError:
> org.hamcrest.Matcher.describeMismatch(Ljava/lang/Object;Lorg/hamcrest/Description;)V
> {noformat}
> There is a dependency conflict created by an older version of {{hamcrest}}
> coming out of {{mockito-all}}. The fix is to use {{mockito-core}} instead.
> I'll submit a patch for this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-tinkerpop issue #329: TINKERPOP-1318: use mockito-core instead of ...
Github user okram commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/329 VOTE +1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo
[
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314439#comment-15314439
]
ASF GitHub Bot commented on TINKERPOP-1321:
---
Github user dalaro commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/325
To do that cleanly, users would have to write serializers against the shim
interface.
Of course, the shim is only relevant to users writing new serializers or
willing to refactor their old ones.
I see no clean way to make this work with existing serializers that users
have hypothetically already compiled against TP's shaded Kryo. To make a user
serializer written for TP's shaded Kryo link against Spark's unshaded Kryo is
technically possible with some degree of skulduggery, it's just that all of the
ways to do it that I'm aware of are unspeakably ugly.
> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
> Issue Type: Improvement
> Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting
> spark.serializer=GryoSerializer. This makes GryoSerializer responsible for
> serializing not just TinkerPop types but also scala runtime types and Spark
> internals. GryoSerializer doesn't extend either of the two serializers
> provided by Spark. It effectively assumes responsibility for reimplementing
> them.
> This is problematic. It is not totally trivial to replicate the
> functionality of Spark's standard serializers. It is also not easy to
> empirically test all meaningful cases. For instance, there is a conditional
> within Spark that selects between two concrete Map implementations depending
> on whether the current RDD partition count exceeds 2k
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
> The implementation used below this threshold serializes fine on
> GryoSerializer. The implementation used above the threshold does not. Above
> the partition threshold, I've started getting
> {{org.apache.spark.SparkException: Job aborted due to stage failure:
> Exception while getting task result:
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed
> to find one of the right cookies.}} Google leads to
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64. However, just
> switching to Spark's {{KryoSerializer}} without changing anything somehow
> fixes the problem in my environment, implying that Spark has done something
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.
> These still serialize, but KryoSerializer uses its default behavior
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's
> StarGraphSerializer. TinkerPop's reliance on its own internal shaded Kryo
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just
> by tweaking the configuration. Besides spark.serializer, there is also a
> setting spark.closure.serializer for which the only supported value is
> JavaSerializer. Key TP classes that make it into the object reference graphs
> of Spark closures implement Serializable by resorting to TinkerPop's shaded
> Kryo via HadoopPools (looking at Object/VertexWritable). This leads to
> surprises with custom property data types. It doesn't matter if those types
> implement Serializable, and it doesn't matter if Spark's KryoSerializer is
> configured to accept those types. If those types are reachable from
> Object/VertexWritable, then they must be registered with TinkerPop's internal
> shaded Kryo, or else it will choke on them (unless it was explicitly
> configured to allow unregistered classes).
> I suggest the following change to give users more flexibility in their choice
> of spark.serializer, and to allow them to reuse TinkerPop's serializers if
> they choose not to use GryoSerializer: introduce lightweight interfaces that
> decouple TinkerPop's serialization logic from the exact Kryo shaded/unshaded
> package doing the work. TinkerPop's serialization logic would be written
> against interfaces that replicate a minimal subset of Kryo, and then TP's
> shaded Kryo or Spark's unshade
[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...
Github user dalaro commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/325 To do that cleanly, users would have to write serializers against the shim interface. Of course, the shim is only relevant to users writing new serializers or willing to refactor their old ones. I see no clean way to make this work with existing serializers that users have hypothetically already compiled against TP's shaded Kryo. To make a user serializer written for TP's shaded Kryo link against Spark's unshaded Kryo is technically possible with some degree of skulduggery, it's just that all of the ways to do it that I'm aware of are unspeakably ugly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
TinkerPop /current URL
Mr Kuppitz just made a quite straightforward change to our documentation publishing system that has allowed us to offer a static URL to our latest documentation. So rather than having to explicitly specify the version as with: http://tinkerpop.apache.org/docs/3.2.0-incubating/reference/ we can instead do: http://tinkerpop.apache.org/docs/current/reference/ Note that /current will always point at the latest official version of TinkerPop which is currently 3.2.0-incubating. Thanks, Stephen
[jira] [Commented] (TINKERPOP-1318) java.lang.NoSuchMethodError: org/hamcrest/Matcher.describeMismatch
[
https://issues.apache.org/jira/browse/TINKERPOP-1318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314431#comment-15314431
]
ASF GitHub Bot commented on TINKERPOP-1318:
---
Github user pluradj commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/329
Yes, I ran integration tests with `-DincludeNeo4j`
> java.lang.NoSuchMethodError: org/hamcrest/Matcher.describeMismatch
> --
>
> Key: TINKERPOP-1318
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1318
> Project: TinkerPop
> Issue Type: Bug
> Components: test-suite
>Affects Versions: 3.2.0-incubating, 3.1.2-incubating
>Reporter: Jason Plurad
>Assignee: Jason Plurad
>Priority: Minor
>
> I don't recall specifically how to make this fail with {{gremlin-test}}, but
> I did run into it at one point when writing a graph implementation. This blog
> describes the issue and workaround.
> https://tedvinke.wordpress.com/2013/12/17/mixing-junit-hamcrest-and-mockito-explaining-nosuchmethoderror/
> The error trace looks like this:
> {noformat}
> java.lang.NoSuchMethodError:
> org.hamcrest.Matcher.describeMismatch(Ljava/lang/Object;Lorg/hamcrest/Description;)V
> {noformat}
> There is a dependency conflict created by an older version of {{hamcrest}}
> coming out of {{mockito-all}}. The fix is to use {{mockito-core}} instead.
> I'll submit a patch for this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-tinkerpop issue #329: TINKERPOP-1318: use mockito-core instead of ...
Github user pluradj commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/329 Yes, I ran integration tests with `-DincludeNeo4j` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1319) several FeatureRequirement annotations are incorrect in gremlin-test
[
https://issues.apache.org/jira/browse/TINKERPOP-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314427#comment-15314427
]
ASF GitHub Bot commented on TINKERPOP-1319:
---
GitHub user pluradj opened a pull request:
https://github.com/apache/incubator-tinkerpop/pull/330
several FeatureRequirement annotations are incorrect
https://issues.apache.org/jira/browse/TINKERPOP-1319
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pluradj/incubator-tinkerpop TINKERPOP-1319
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-tinkerpop/pull/330.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #330
commit d0878cf70778c637cafa43d09f9a134d89ee5024
Author: Jason Plurad
Date: 2016-06-03T17:06:27Z
several FeatureRequirement annotations are incorrect
> several FeatureRequirement annotations are incorrect in gremlin-test
>
>
> Key: TINKERPOP-1319
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1319
> Project: TinkerPop
> Issue Type: Bug
> Components: test-suite
>Affects Versions: 3.2.0-incubating, 3.1.2-incubating
>Reporter: Jason Plurad
>Assignee: Jason Plurad
>Priority: Minor
>
> Several {{@FeatureRequirement}} annotations are incorrect in these
> {{gremlin-test}} tests
> * EdgeTest.java
> * FeatureSupportTest.java
> * GraphTest.java
> * PropertyTest.java
> * VertexPropertyTest.java
> * VertexTest.java
> I'll submit a patch for this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-tinkerpop pull request #330: several FeatureRequirement annotation...
GitHub user pluradj opened a pull request: https://github.com/apache/incubator-tinkerpop/pull/330 several FeatureRequirement annotations are incorrect https://issues.apache.org/jira/browse/TINKERPOP-1319 You can merge this pull request into a Git repository by running: $ git pull https://github.com/pluradj/incubator-tinkerpop TINKERPOP-1319 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-tinkerpop/pull/330.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #330 commit d0878cf70778c637cafa43d09f9a134d89ee5024 Author: Jason Plurad Date: 2016-06-03T17:06:27Z several FeatureRequirement annotations are incorrect --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-tinkerpop issue #329: use mockito-core instead of mockito-all to a...
Github user okram commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/329 Do integration tests pass? ... that will be all I care about. @spmallette will have deeper thoughts on the matter as he uses Mockito. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1318) java.lang.NoSuchMethodError: org/hamcrest/Matcher.describeMismatch
[
https://issues.apache.org/jira/browse/TINKERPOP-1318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314426#comment-15314426
]
ASF GitHub Bot commented on TINKERPOP-1318:
---
GitHub user pluradj opened a pull request:
https://github.com/apache/incubator-tinkerpop/pull/329
use mockito-core instead of mockito-all to avoid hamcrest conflict
https://issues.apache.org/jira/browse/TINKERPOP-1318
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pluradj/incubator-tinkerpop TINKERPOP-1318
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-tinkerpop/pull/329.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #329
commit 27eedc478b58cd9dee571960091a3c6505f2379e
Author: Jason Plurad
Date: 2016-06-03T17:05:10Z
use mockito-core instead of mockito-all to avoid hamcrest conflict
> java.lang.NoSuchMethodError: org/hamcrest/Matcher.describeMismatch
> --
>
> Key: TINKERPOP-1318
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1318
> Project: TinkerPop
> Issue Type: Bug
> Components: test-suite
>Affects Versions: 3.2.0-incubating, 3.1.2-incubating
>Reporter: Jason Plurad
>Assignee: Jason Plurad
>Priority: Minor
>
> I don't recall specifically how to make this fail with {{gremlin-test}}, but
> I did run into it at one point when writing a graph implementation. This blog
> describes the issue and workaround.
> https://tedvinke.wordpress.com/2013/12/17/mixing-junit-hamcrest-and-mockito-explaining-nosuchmethoderror/
> The error trace looks like this:
> {noformat}
> java.lang.NoSuchMethodError:
> org.hamcrest.Matcher.describeMismatch(Ljava/lang/Object;Lorg/hamcrest/Description;)V
> {noformat}
> There is a dependency conflict created by an older version of {{hamcrest}}
> coming out of {{mockito-all}}. The fix is to use {{mockito-core}} instead.
> I'll submit a patch for this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-tinkerpop pull request #329: use mockito-core instead of mockito-a...
GitHub user pluradj opened a pull request: https://github.com/apache/incubator-tinkerpop/pull/329 use mockito-core instead of mockito-all to avoid hamcrest conflict https://issues.apache.org/jira/browse/TINKERPOP-1318 You can merge this pull request into a Git repository by running: $ git pull https://github.com/pluradj/incubator-tinkerpop TINKERPOP-1318 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-tinkerpop/pull/329.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #329 commit 27eedc478b58cd9dee571960091a3c6505f2379e Author: Jason Plurad Date: 2016-06-03T17:05:10Z use mockito-core instead of mockito-all to avoid hamcrest conflict --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1317) IoGraphTest throws error: URI is not hierarchical
[
https://issues.apache.org/jira/browse/TINKERPOP-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314424#comment-15314424
]
ASF GitHub Bot commented on TINKERPOP-1317:
---
GitHub user pluradj opened a pull request:
https://github.com/apache/incubator-tinkerpop/pull/328
use graph class as root class to find temporary directory
https://issues.apache.org/jira/browse/TINKERPOP-1317
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pluradj/incubator-tinkerpop TINKERPOP-1317
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-tinkerpop/pull/328.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #328
commit 4b9bb87cd80bf6658e40090dd2cdb7b7b8f97122
Author: Jason Plurad
Date: 2016-06-03T17:03:19Z
use graph class as root class to find temporary directory
> IoGraphTest throws error: URI is not hierarchical
> -
>
> Key: TINKERPOP-1317
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1317
> Project: TinkerPop
> Issue Type: Bug
> Components: test-suite
>Affects Versions: 3.2.0-incubating, 3.1.2-incubating
>Reporter: Jason Plurad
>Assignee: Jason Plurad
>Priority: Minor
>
> Via https://groups.google.com/d/msg/gremlin-users/aap3pxZtGyU/t-eOC6ZyAAAJ
> These methods will fail for graph providers that are trying to implement the
> test suite:
> * {{IoGraphTest.shouldReadWriteModernToFileWithHelpers()}}
> * {{IoGraphTest.shouldReadWriteClassicToFileWithHelpers()}}
> Example stack trace:
> {noformat}
> shouldReadWriteModernToFileWithHelpers[gryo](org.apache.tinkerpop.gremlin.structure.io.IoGraphTest)
> Time elapsed: 0.004 sec <<< ERROR!
> java.lang.RuntimeException: Unable to computePath for class
> org.apache.tinkerpop.gremlin.structure.io.IoGraphTest
> at
> org.apache.tinkerpop.gremlin.TestHelper.computePath(TestHelper.java:117)
> at
> org.apache.tinkerpop.gremlin.TestHelper.getRootOfBuildDirectory(TestHelper.java:105)
> at
> org.apache.tinkerpop.gremlin.TestHelper.makeTestDataPath(TestHelper.java:70)
> at
> org.apache.tinkerpop.gremlin.TestHelper.generateTempFile(TestHelper.java:127)
> at
> org.apache.tinkerpop.gremlin.structure.io.IoGraphTest.shouldReadWriteModernToFileWithHelpers(IoGraphTest.java:164)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.r
[GitHub] incubator-tinkerpop pull request #328: use graph class as root class to find...
GitHub user pluradj opened a pull request: https://github.com/apache/incubator-tinkerpop/pull/328 use graph class as root class to find temporary directory https://issues.apache.org/jira/browse/TINKERPOP-1317 You can merge this pull request into a Git repository by running: $ git pull https://github.com/pluradj/incubator-tinkerpop TINKERPOP-1317 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-tinkerpop/pull/328.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #328 commit 4b9bb87cd80bf6658e40090dd2cdb7b7b8f97122 Author: Jason Plurad Date: 2016-06-03T17:03:19Z use graph class as root class to find temporary directory --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo
[
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314387#comment-15314387
]
ASF GitHub Bot commented on TINKERPOP-1321:
---
Github user okram commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/325
@dalaro
> The problem with custom registrations is that GryoMapper allows the user
to provide a serializer written against shaded Kryo. This is not compatible
with Spark's KryoSerializer. I don't see a way to make it compatible without
putting fake org.apache.tinkerpop.shaded.kryo.* classes on the classpath, which
could get pretty ugly.
Yea, thats a problem and the reason why `GryoSerializer` exists. We don't
want to make `SparkGraphComputer` "special." Users register their serializers
with `Graph` and all OLTP and OLAP systems take it from there. If, for
`SparkGraphComputer`, they ALSO have to register them with Spark, that is no
bueno. Is there a way to create "proxies" that map between shaded and unshaded
Kryo so that registered `GryoMapper` serializers can be used as unshaded Kryo
serializers by Spark?
> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
> Issue Type: Improvement
> Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting
> spark.serializer=GryoSerializer. This makes GryoSerializer responsible for
> serializing not just TinkerPop types but also scala runtime types and Spark
> internals. GryoSerializer doesn't extend either of the two serializers
> provided by Spark. It effectively assumes responsibility for reimplementing
> them.
> This is problematic. It is not totally trivial to replicate the
> functionality of Spark's standard serializers. It is also not easy to
> empirically test all meaningful cases. For instance, there is a conditional
> within Spark that selects between two concrete Map implementations depending
> on whether the current RDD partition count exceeds 2k
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
> The implementation used below this threshold serializes fine on
> GryoSerializer. The implementation used above the threshold does not. Above
> the partition threshold, I've started getting
> {{org.apache.spark.SparkException: Job aborted due to stage failure:
> Exception while getting task result:
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed
> to find one of the right cookies.}} Google leads to
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64. However, just
> switching to Spark's {{KryoSerializer}} without changing anything somehow
> fixes the problem in my environment, implying that Spark has done something
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.
> These still serialize, but KryoSerializer uses its default behavior
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's
> StarGraphSerializer. TinkerPop's reliance on its own internal shaded Kryo
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just
> by tweaking the configuration. Besides spark.serializer, there is also a
> setting spark.closure.serializer for which the only supported value is
> JavaSerializer. Key TP classes that make it into the object reference graphs
> of Spark closures implement Serializable by resorting to TinkerPop's shaded
> Kryo via HadoopPools (looking at Object/VertexWritable). This leads to
> surprises with custom property data types. It doesn't matter if those types
> implement Serializable, and it doesn't matter if Spark's KryoSerializer is
> configured to accept those types. If those types are reachable from
> Object/VertexWritable, then they must be registered with TinkerPop's internal
> shaded Kryo, or else it will choke on them (unless it was explicitly
> configured to allow unregistered classes).
> I suggest the following change to give users more flexibility in their choice
> of spark.serializer, and to allow them to reuse TinkerPop's serializers if
> they choose not to use GryoSerializer: introduce lightweight interfaces that
> dec
[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...
Github user okram commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/325 @dalaro > The problem with custom registrations is that GryoMapper allows the user to provide a serializer written against shaded Kryo. This is not compatible with Spark's KryoSerializer. I don't see a way to make it compatible without putting fake org.apache.tinkerpop.shaded.kryo.* classes on the classpath, which could get pretty ugly. Yea, thats a problem and the reason why `GryoSerializer` exists. We don't want to make `SparkGraphComputer` "special." Users register their serializers with `Graph` and all OLTP and OLAP systems take it from there. If, for `SparkGraphComputer`, they ALSO have to register them with Spark, that is no bueno. Is there a way to create "proxies" that map between shaded and unshaded Kryo so that registered `GryoMapper` serializers can be used as unshaded Kryo serializers by Spark? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1301) Provide Javadoc for ScriptInput/OutputFormat's
[ https://issues.apache.org/jira/browse/TINKERPOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314372#comment-15314372 ] ASF GitHub Bot commented on TINKERPOP-1301: --- Github user lewismc commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/327 @spmallette LOL np > Provide Javadoc for ScriptInput/OutputFormat's > -- > > Key: TINKERPOP-1301 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1301 > Project: TinkerPop > Issue Type: Improvement > Components: documentation >Affects Versions: 3.2.0-incubating >Reporter: Lewis John McGibbney >Priority: Minor > > Right now > [ScriptInputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptInputFormat.html] > and > [ScriptOutputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptOutputFormat.html] > are not documented. Descriptions are however present over on some old > [TitanDB > documentation|http://s3.thinkaurelius.com/docs/titan/0.5.0/script-io-format.html] > which can be used to provide Javadoc level documentation for developers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-tinkerpop issue #327: TINKERPOP-1301 Provide Javadoc for ScriptInp...
Github user lewismc commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/327 @spmallette LOL np --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1301) Provide Javadoc for ScriptInput/OutputFormat's
[ https://issues.apache.org/jira/browse/TINKERPOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314371#comment-15314371 ] ASF GitHub Bot commented on TINKERPOP-1301: --- Github user pluradj commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/327 VOTE +1 > Provide Javadoc for ScriptInput/OutputFormat's > -- > > Key: TINKERPOP-1301 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1301 > Project: TinkerPop > Issue Type: Improvement > Components: documentation >Affects Versions: 3.2.0-incubating >Reporter: Lewis John McGibbney >Priority: Minor > > Right now > [ScriptInputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptInputFormat.html] > and > [ScriptOutputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptOutputFormat.html] > are not documented. Descriptions are however present over on some old > [TitanDB > documentation|http://s3.thinkaurelius.com/docs/titan/0.5.0/script-io-format.html] > which can be used to provide Javadoc level documentation for developers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-tinkerpop issue #327: TINKERPOP-1301 Provide Javadoc for ScriptInp...
Github user pluradj commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/327 VOTE +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-tinkerpop issue #327: TINKERPOP-1301 Provide Javadoc for ScriptInp...
Github user spmallette commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/327 VOTE +1 - this was a longest road to a javadoc-only PR ever - thanks for working it through with us @lewismc --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1301) Provide Javadoc for ScriptInput/OutputFormat's
[ https://issues.apache.org/jira/browse/TINKERPOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314348#comment-15314348 ] ASF GitHub Bot commented on TINKERPOP-1301: --- Github user spmallette commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/327 VOTE +1 - this was a longest road to a javadoc-only PR ever - thanks for working it through with us @lewismc > Provide Javadoc for ScriptInput/OutputFormat's > -- > > Key: TINKERPOP-1301 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1301 > Project: TinkerPop > Issue Type: Improvement > Components: documentation >Affects Versions: 3.2.0-incubating >Reporter: Lewis John McGibbney >Priority: Minor > > Right now > [ScriptInputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptInputFormat.html] > and > [ScriptOutputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptOutputFormat.html] > are not documented. Descriptions are however present over on some old > [TitanDB > documentation|http://s3.thinkaurelius.com/docs/titan/0.5.0/script-io-format.html] > which can be used to provide Javadoc level documentation for developers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TINKERPOP-1278) Implement Gremlin-Python and general purpose language variant test infrastructure
[
https://issues.apache.org/jira/browse/TINKERPOP-1278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marko A. Rodriguez reassigned TINKERPOP-1278:
-
Assignee: Marko A. Rodriguez
> Implement Gremlin-Python and general purpose language variant test
> infrastructure
> -
>
> Key: TINKERPOP-1278
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1278
> Project: TinkerPop
> Issue Type: Improvement
> Components: language-variant
>Affects Versions: 3.2.0-incubating
>Reporter: Marko A. Rodriguez
>Assignee: Marko A. Rodriguez
>
> As discussed on dev@...
> Apache TinkerPop should provide, out-of-the-box, at least 3 Gremlin language
> variants. It would be cool if these were:
> * Python (Mark Henderson)
> * PHP ([~PommeVerte])
> * Ruby (?[~okram])
> I think each of these should be generated using the reflection-model
> presented in
> http://tinkerpop.apache.org/docs/3.2.1-SNAPSHOT/tutorials/gremlin-language-variants/.
> Moreover, on every {{mvn clean install}}, the code for these variants is
> generated.
> Given the desire to separate language variants from language drivers, I think
> that a language driver for each variant above should be "plugable." Moreover,
> we should provide one driver implementation for each -- simple GremlinServer
> REST.
> {code}
> gremlin-variants/
> gremlin-ruby/
> gremlin_ruby.rb
> gremlin_ruby_rest_driver.rb
> gremlin-php/
> Gremlin_PHP.php
> Gremlin_PHP_REST_Driver.php
> gremlin-python/
> gremlin-python.py
> gremlin-python-rest-driver.py
> {code}
> Next, each variant implementation should be testable. This is PAINFUL if we
> have to implement each {{g_V_out_repeatXasXaXX}} test case in
> {{ProcessXXXSuite}}. Perhaps some RegEx transducer magic could be used to
> convert all those tests from Gremlin-Java to the respective host language?
> However, even if we do that, we still have the problem of how to test the
> returned results.
> I think what we should test the returned results using the JVM. For instance,
> JRuby, Jython, JPHP (does it exist?). If we do this, we will save ourselves a
> massive headache. All we have to do is create a {{GraphProvider}} that uses
> {{TinkerGraph}} and whose {{TraversalSource}} is some sort of wrapper around
> reflection-generated Ruby (e.g.).
> {code}
> g.V.out_e("knows") // returns a Ruby iterator
> {code}
> That Ruby iterator should be converted to a Java iterator and then the
> {{ProcessXXXSuite}} can verify the results.
> With this, most everything is reflectively constructed.
> {code}
> gremlin_ruby.rb // generated via Java reflection
> gremlin_ruby_rest_driver.rb // manually coded
> match_test.rb // generated via RegEx transducer
> has_test.rb // generated via RegEx transducer
> ...
> RubyGraphProvider.java// manually coded
> RubyProcessStandardSuite.java // manually coded
> RubyProcessComputerSuite.java // manually coded
> {code}
> Thus, the testing data flow would be:
> {code}
> MatchTest.Traversals.java --transducer-> match_test.rb
> match-test.rb --REST--> GremlinServer
> GremlinServer --GraphSON-->match-test.rb
> GraphSON --JRuby/GraphSONReader-->Java objects
> Java objects --JRuby-->MatchTest.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TINKERPOP-1278) Implement Gremlin-Python and general purpose language variant test infrastructure
[
https://issues.apache.org/jira/browse/TINKERPOP-1278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marko A. Rodriguez updated TINKERPOP-1278:
--
Summary: Implement Gremlin-Python and general purpose language variant test
infrastructure (was: Support at least 3 Gremlin language variants.)
> Implement Gremlin-Python and general purpose language variant test
> infrastructure
> -
>
> Key: TINKERPOP-1278
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1278
> Project: TinkerPop
> Issue Type: Improvement
> Components: language-variant
>Affects Versions: 3.2.0-incubating
>Reporter: Marko A. Rodriguez
>
> As discussed on dev@...
> Apache TinkerPop should provide, out-of-the-box, at least 3 Gremlin language
> variants. It would be cool if these were:
> * Python (Mark Henderson)
> * PHP ([~PommeVerte])
> * Ruby (?[~okram])
> I think each of these should be generated using the reflection-model
> presented in
> http://tinkerpop.apache.org/docs/3.2.1-SNAPSHOT/tutorials/gremlin-language-variants/.
> Moreover, on every {{mvn clean install}}, the code for these variants is
> generated.
> Given the desire to separate language variants from language drivers, I think
> that a language driver for each variant above should be "plugable." Moreover,
> we should provide one driver implementation for each -- simple GremlinServer
> REST.
> {code}
> gremlin-variants/
> gremlin-ruby/
> gremlin_ruby.rb
> gremlin_ruby_rest_driver.rb
> gremlin-php/
> Gremlin_PHP.php
> Gremlin_PHP_REST_Driver.php
> gremlin-python/
> gremlin-python.py
> gremlin-python-rest-driver.py
> {code}
> Next, each variant implementation should be testable. This is PAINFUL if we
> have to implement each {{g_V_out_repeatXasXaXX}} test case in
> {{ProcessXXXSuite}}. Perhaps some RegEx transducer magic could be used to
> convert all those tests from Gremlin-Java to the respective host language?
> However, even if we do that, we still have the problem of how to test the
> returned results.
> I think what we should test the returned results using the JVM. For instance,
> JRuby, Jython, JPHP (does it exist?). If we do this, we will save ourselves a
> massive headache. All we have to do is create a {{GraphProvider}} that uses
> {{TinkerGraph}} and whose {{TraversalSource}} is some sort of wrapper around
> reflection-generated Ruby (e.g.).
> {code}
> g.V.out_e("knows") // returns a Ruby iterator
> {code}
> That Ruby iterator should be converted to a Java iterator and then the
> {{ProcessXXXSuite}} can verify the results.
> With this, most everything is reflectively constructed.
> {code}
> gremlin_ruby.rb // generated via Java reflection
> gremlin_ruby_rest_driver.rb // manually coded
> match_test.rb // generated via RegEx transducer
> has_test.rb // generated via RegEx transducer
> ...
> RubyGraphProvider.java// manually coded
> RubyProcessStandardSuite.java // manually coded
> RubyProcessComputerSuite.java // manually coded
> {code}
> Thus, the testing data flow would be:
> {code}
> MatchTest.Traversals.java --transducer-> match_test.rb
> match-test.rb --REST--> GremlinServer
> GremlinServer --GraphSON-->match-test.rb
> GraphSON --JRuby/GraphSONReader-->Java objects
> Java objects --JRuby-->MatchTest.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1301) Provide Javadoc for ScriptInput/OutputFormat's
[ https://issues.apache.org/jira/browse/TINKERPOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314309#comment-15314309 ] ASF GitHub Bot commented on TINKERPOP-1301: --- Github user dkuppitz commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/327 VOTE: +1 > Provide Javadoc for ScriptInput/OutputFormat's > -- > > Key: TINKERPOP-1301 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1301 > Project: TinkerPop > Issue Type: Improvement > Components: documentation >Affects Versions: 3.2.0-incubating >Reporter: Lewis John McGibbney >Priority: Minor > > Right now > [ScriptInputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptInputFormat.html] > and > [ScriptOutputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptOutputFormat.html] > are not documented. Descriptions are however present over on some old > [TitanDB > documentation|http://s3.thinkaurelius.com/docs/titan/0.5.0/script-io-format.html] > which can be used to provide Javadoc level documentation for developers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-tinkerpop issue #327: TINKERPOP-1301 Provide Javadoc for ScriptInp...
Github user dkuppitz commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/327 VOTE: +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1301) Provide Javadoc for ScriptInput/OutputFormat's
[ https://issues.apache.org/jira/browse/TINKERPOP-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314306#comment-15314306 ] ASF GitHub Bot commented on TINKERPOP-1301: --- GitHub user lewismc opened a pull request: https://github.com/apache/incubator-tinkerpop/pull/327 TINKERPOP-1301 Provide Javadoc for ScriptInput/OutputFormat's ported to tp31 branch Hi @dkuppitz here is the TINKERPOP-1301 ported to tp31 branch :) You can merge this pull request into a Git repository by running: $ git pull https://github.com/lewismc/incubator-tinkerpop TINKERPOP-1301tp31 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-tinkerpop/pull/327.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #327 commit d5d2c48ef67e31671cb177d30c6a15c84b585d24 Author: Lewis John McGibbney Date: 2016-06-03T16:00:08Z TINKERPOP-1301 Provide Javadoc for ScriptInput/OutputFormat's ported to tp31 branch > Provide Javadoc for ScriptInput/OutputFormat's > -- > > Key: TINKERPOP-1301 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1301 > Project: TinkerPop > Issue Type: Improvement > Components: documentation >Affects Versions: 3.2.0-incubating >Reporter: Lewis John McGibbney >Priority: Minor > > Right now > [ScriptInputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptInputFormat.html] > and > [ScriptOutputFormat|http://tinkerpop.apache.org/javadocs/3.2.1-SNAPSHOT/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptOutputFormat.html] > are not documented. Descriptions are however present over on some old > [TitanDB > documentation|http://s3.thinkaurelius.com/docs/titan/0.5.0/script-io-format.html] > which can be used to provide Javadoc level documentation for developers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-tinkerpop pull request #327: TINKERPOP-1301 Provide Javadoc for Sc...
GitHub user lewismc opened a pull request: https://github.com/apache/incubator-tinkerpop/pull/327 TINKERPOP-1301 Provide Javadoc for ScriptInput/OutputFormat's ported to tp31 branch Hi @dkuppitz here is the TINKERPOP-1301 ported to tp31 branch :) You can merge this pull request into a Git repository by running: $ git pull https://github.com/lewismc/incubator-tinkerpop TINKERPOP-1301tp31 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-tinkerpop/pull/327.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #327 commit d5d2c48ef67e31671cb177d30c6a15c84b585d24 Author: Lewis John McGibbney Date: 2016-06-03T16:00:08Z TINKERPOP-1301 Provide Javadoc for ScriptInput/OutputFormat's ported to tp31 branch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (TINKERPOP-1324) Better error for invalid args to addV()
stephen mallette created TINKERPOP-1324:
---
Summary: Better error for invalid args to addV()
Key: TINKERPOP-1324
URL: https://issues.apache.org/jira/browse/TINKERPOP-1324
Project: TinkerPop
Issue Type: Bug
Components: process
Affects Versions: 3.1.2-incubating
Reporter: stephen mallette
Priority: Minor
Fix For: 3.1.3, 3.2.1
Could use a better error for when the wrong number of arguments are being
passed to {{addV()}}:
{code}
gremlin> g = TinkerGraph.open().traversal()
==>graphtraversalsource[tinkergraph[vertices:0 edges:0], standard]
gremlin> g.addV('person','x',1)
3
Display stack trace? [yN]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo
[
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314057#comment-15314057
]
ASF GitHub Bot commented on TINKERPOP-1321:
---
Github user dalaro commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/325
@spmallette You're right about overrides being the problem.
The last thing I did before opening this PR was to rebase on latest master,
when I encountered conflicts solely involving the GryoMapper registration
override feature added in d7684757734732f39970265a425b921a48d8f0fb. I
attempted to resolve those conflicts too hastily and got it wrong.
I went back to Stephen's original commit reworked my code around
registration overrides, and I now think it conforms to the old behavior
precedent. Specifically, when handling overrides of GryoMapper type
registrations, I departed from the precedent set by Stephen's commit in two
ways, both of which I've fixed locally:
* Override registrations allocated a new registration ID. Your commit used
the old ID from the registration being overridden.
* As a side effect of the preceding point, I incremented the shared
`currentSerializationId` for all registration calls. Your commit incremented
this shared counter only for non-override registration calls.
The test passes locally for me now. I'm still doing some registrator
refactoring and haven't pushed yet though.
@okram The problem with custom registrations is that GryoMapper allows the
user to provide a serializer written against shaded Kryo. This is not
compatible with Spark's KryoSerializer. I don't see a way to make it
compatible without putting fake `org.apache.tinkerpop.shaded.kryo.*` classes on
the classpath, which could get pretty ugly.
Instead, I'm adding a registrator that includes:
* all default mappings from GryoMapper (I am changing the
shaded-Kryo-serializers to be shim serializers so that they are reusable)
* the TinkerPop classes registered in GryoSerializer's constructors (but
not the scala and Spark classes registered in there)
If the user wants to register custom types while using KryoSerializer, they
can extend the registrator in a subclass, then set `spark.kryo.registrator` to
the subclass. The interface in question is `KryoRegistrator` and it has only
one method, `registerClasses(Kryo)`. It's about as simple -- maybe even less
so -- than implementing TP's `IoRegistry` interface, which has two methods.
If the user decides to continue using GryoSerializer, they can also
continue using `GryoPool.CONFIG_IO_REGISTRY`, which should still affect
GryoSerializer/HadoopPools as it always has.
WDYT?
> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
> Issue Type: Improvement
> Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting
> spark.serializer=GryoSerializer. This makes GryoSerializer responsible for
> serializing not just TinkerPop types but also scala runtime types and Spark
> internals. GryoSerializer doesn't extend either of the two serializers
> provided by Spark. It effectively assumes responsibility for reimplementing
> them.
> This is problematic. It is not totally trivial to replicate the
> functionality of Spark's standard serializers. It is also not easy to
> empirically test all meaningful cases. For instance, there is a conditional
> within Spark that selects between two concrete Map implementations depending
> on whether the current RDD partition count exceeds 2k
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
> The implementation used below this threshold serializes fine on
> GryoSerializer. The implementation used above the threshold does not. Above
> the partition threshold, I've started getting
> {{org.apache.spark.SparkException: Job aborted due to stage failure:
> Exception while getting task result:
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed
> to find one of the right cookies.}} Google leads to
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64. However, just
> switching to Spark's {{KryoSerializer}} without changing anything somehow
> fixes the problem in my environment, implying that Spark has done something
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a gr
[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...
Github user dalaro commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/325 @spmallette You're right about overrides being the problem. The last thing I did before opening this PR was to rebase on latest master, when I encountered conflicts solely involving the GryoMapper registration override feature added in d7684757734732f39970265a425b921a48d8f0fb. I attempted to resolve those conflicts too hastily and got it wrong. I went back to Stephen's original commit reworked my code around registration overrides, and I now think it conforms to the old behavior precedent. Specifically, when handling overrides of GryoMapper type registrations, I departed from the precedent set by Stephen's commit in two ways, both of which I've fixed locally: * Override registrations allocated a new registration ID. Your commit used the old ID from the registration being overridden. * As a side effect of the preceding point, I incremented the shared `currentSerializationId` for all registration calls. Your commit incremented this shared counter only for non-override registration calls. The test passes locally for me now. I'm still doing some registrator refactoring and haven't pushed yet though. @okram The problem with custom registrations is that GryoMapper allows the user to provide a serializer written against shaded Kryo. This is not compatible with Spark's KryoSerializer. I don't see a way to make it compatible without putting fake `org.apache.tinkerpop.shaded.kryo.*` classes on the classpath, which could get pretty ugly. Instead, I'm adding a registrator that includes: * all default mappings from GryoMapper (I am changing the shaded-Kryo-serializers to be shim serializers so that they are reusable) * the TinkerPop classes registered in GryoSerializer's constructors (but not the scala and Spark classes registered in there) If the user wants to register custom types while using KryoSerializer, they can extend the registrator in a subclass, then set `spark.kryo.registrator` to the subclass. The interface in question is `KryoRegistrator` and it has only one method, `registerClasses(Kryo)`. It's about as simple -- maybe even less so -- than implementing TP's `IoRegistry` interface, which has two methods. If the user decides to continue using GryoSerializer, they can also continue using `GryoPool.CONFIG_IO_REGISTRY`, which should still affect GryoSerializer/HadoopPools as it always has. WDYT? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo
[
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314048#comment-15314048
]
ASF GitHub Bot commented on TINKERPOP-1321:
---
Github user okram commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/325
@dalaro -- you mention:
```
spark.serializer=KryoSerializer
spark.kryo.registrator=com.tinkerpop.something.TinkerPopRegistrator
```
I think we need:
```
spark.serializer=KryoSerializer
spark.kryo.registrator=org.apache.tinkerpop.gremlin.structure.io.gryo.GryoRegistrar
```
`GryoRegistrar` would have all the `GryoMapper` classes registered which,
when people do custom registrations (e.g. `Point`,
`MyStupidStringImplementation`), it will be in `GryoMapper`. Can you provide
something like that in this PR? Hard? @spmallette knows more about how to
interact with `GryoMapper` to get our the serializers...
> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
> Issue Type: Improvement
> Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting
> spark.serializer=GryoSerializer. This makes GryoSerializer responsible for
> serializing not just TinkerPop types but also scala runtime types and Spark
> internals. GryoSerializer doesn't extend either of the two serializers
> provided by Spark. It effectively assumes responsibility for reimplementing
> them.
> This is problematic. It is not totally trivial to replicate the
> functionality of Spark's standard serializers. It is also not easy to
> empirically test all meaningful cases. For instance, there is a conditional
> within Spark that selects between two concrete Map implementations depending
> on whether the current RDD partition count exceeds 2k
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
> The implementation used below this threshold serializes fine on
> GryoSerializer. The implementation used above the threshold does not. Above
> the partition threshold, I've started getting
> {{org.apache.spark.SparkException: Job aborted due to stage failure:
> Exception while getting task result:
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed
> to find one of the right cookies.}} Google leads to
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64. However, just
> switching to Spark's {{KryoSerializer}} without changing anything somehow
> fixes the problem in my environment, implying that Spark has done something
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.
> These still serialize, but KryoSerializer uses its default behavior
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's
> StarGraphSerializer. TinkerPop's reliance on its own internal shaded Kryo
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just
> by tweaking the configuration. Besides spark.serializer, there is also a
> setting spark.closure.serializer for which the only supported value is
> JavaSerializer. Key TP classes that make it into the object reference graphs
> of Spark closures implement Serializable by resorting to TinkerPop's shaded
> Kryo via HadoopPools (looking at Object/VertexWritable). This leads to
> surprises with custom property data types. It doesn't matter if those types
> implement Serializable, and it doesn't matter if Spark's KryoSerializer is
> configured to accept those types. If those types are reachable from
> Object/VertexWritable, then they must be registered with TinkerPop's internal
> shaded Kryo, or else it will choke on them (unless it was explicitly
> configured to allow unregistered classes).
> I suggest the following change to give users more flexibility in their choice
> of spark.serializer, and to allow them to reuse TinkerPop's serializers if
> they choose not to use GryoSerializer: introduce lightweight interfaces that
> decouple TinkerPop's serialization logic from the exact Kryo shaded/unshaded
> package doing the work. TinkerPop's serialization logic would be written
> against interfaces th
[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...
Github user okram commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/325 @dalaro -- you mention: ``` spark.serializer=KryoSerializer spark.kryo.registrator=com.tinkerpop.something.TinkerPopRegistrator ``` I think we need: ``` spark.serializer=KryoSerializer spark.kryo.registrator=org.apache.tinkerpop.gremlin.structure.io.gryo.GryoRegistrar ``` `GryoRegistrar` would have all the `GryoMapper` classes registered which, when people do custom registrations (e.g. `Point`, `MyStupidStringImplementation`), it will be in `GryoMapper`. Can you provide something like that in this PR? Hard? @spmallette knows more about how to interact with `GryoMapper` to get our the serializers... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-tinkerpop issue #322: TINKERPOP-1196 Fix for Result.one() which co...
Github user dkuppitz commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/322 20 runs in a row, all of them succeeded. VOTE: +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1196) Calls to Result.one() might block indefinitely
[ https://issues.apache.org/jira/browse/TINKERPOP-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313981#comment-15313981 ] ASF GitHub Bot commented on TINKERPOP-1196: --- Github user dkuppitz commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/322 20 runs in a row, all of them succeeded. VOTE: +1 > Calls to Result.one() might block indefinitely > -- > > Key: TINKERPOP-1196 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1196 > Project: TinkerPop > Issue Type: Bug > Components: driver >Affects Versions: 3.1.1-incubating >Reporter: stephen mallette >Assignee: stephen mallette > Fix For: 3.1.3 > > > There are no reproduction steps for this but it does seem to happen from on > very rare occasion for some unknown reason. Need to investigate more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TINKERPOP-1321) Loosen coupling between TinkerPop serialization logic and shaded Kryo
[
https://issues.apache.org/jira/browse/TINKERPOP-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313973#comment-15313973
]
ASF GitHub Bot commented on TINKERPOP-1321:
---
Github user spmallette commented on the issue:
https://github.com/apache/incubator-tinkerpop/pull/325
I've confirmed that this test:
```text
GremlinResultSetIntegrateTest.shouldHandleVertexResultWithLiteSerialization:174
? Execution
```
is only failing after these changes and not on master. I sense it has
something to do with this:
```java
private void addOrOverrideRegistration(TypeRegistration
newRegistration)
```
not preserving the registration identifier, but i could be wrong.
> Loosen coupling between TinkerPop serialization logic and shaded Kryo
> -
>
> Key: TINKERPOP-1321
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1321
> Project: TinkerPop
> Issue Type: Improvement
> Components: io
>Affects Versions: 3.2.0-incubating
>Reporter: Dan LaRocque
> Fix For: 3.2.1
>
>
> When running jobs on Spark, TinkerPop currently recommends setting
> spark.serializer=GryoSerializer. This makes GryoSerializer responsible for
> serializing not just TinkerPop types but also scala runtime types and Spark
> internals. GryoSerializer doesn't extend either of the two serializers
> provided by Spark. It effectively assumes responsibility for reimplementing
> them.
> This is problematic. It is not totally trivial to replicate the
> functionality of Spark's standard serializers. It is also not easy to
> empirically test all meaningful cases. For instance, there is a conditional
> within Spark that selects between two concrete Map implementations depending
> on whether the current RDD partition count exceeds 2k
> (https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala#L47-L53).
> The implementation used below this threshold serializes fine on
> GryoSerializer. The implementation used above the threshold does not. Above
> the partition threshold, I've started getting
> {{org.apache.spark.SparkException: Job aborted due to stage failure:
> Exception while getting task result:
> org.apache.tinkerpop.shaded.kryo.KryoException: java.io.IOException: I failed
> to find one of the right cookies.}} Google leads to
> https://github.com/RoaringBitmap/RoaringBitmap/issues/64. However, just
> switching to Spark's {{KryoSerializer}} without changing anything somehow
> fixes the problem in my environment, implying that Spark has done something
> to address this problem that may not be fully replicated in TinkerPop.
> However, "just switching to Spark's KryoSerializer" is not a great approach.
> For one thing, we lose the benefit of TinkerPop's space-efficient StarGraph
> serializer, and Spark traversals can produce a lot of little ego-StarGraphs.
> These still serialize, but KryoSerializer uses its default behavior
> (FieldSerializer), which is not as clever about StarGraphs as TinkerPop's
> StarGraphSerializer. TinkerPop's reliance on its own internal shaded Kryo
> means that its serializers cannot be registered with Spark's unshaded Kryo.
> More concerning, it's impossible to completely switch to KryoSerializer just
> by tweaking the configuration. Besides spark.serializer, there is also a
> setting spark.closure.serializer for which the only supported value is
> JavaSerializer. Key TP classes that make it into the object reference graphs
> of Spark closures implement Serializable by resorting to TinkerPop's shaded
> Kryo via HadoopPools (looking at Object/VertexWritable). This leads to
> surprises with custom property data types. It doesn't matter if those types
> implement Serializable, and it doesn't matter if Spark's KryoSerializer is
> configured to accept those types. If those types are reachable from
> Object/VertexWritable, then they must be registered with TinkerPop's internal
> shaded Kryo, or else it will choke on them (unless it was explicitly
> configured to allow unregistered classes).
> I suggest the following change to give users more flexibility in their choice
> of spark.serializer, and to allow them to reuse TinkerPop's serializers if
> they choose not to use GryoSerializer: introduce lightweight interfaces that
> decouple TinkerPop's serialization logic from the exact Kryo shaded/unshaded
> package doing the work. TinkerPop's serialization logic would be written
> against interfaces that replicate a minimal subset of Kryo, and then TP's
> shaded Kryo or Spark's unshaded Kryo could be plugged in underneath without
> having to touch the source, recompile any TinkerPop code, or munge bytecode
> at runtime.
[GitHub] incubator-tinkerpop issue #325: TINKERPOP-1321 Introduce Kryo shim to suppor...
Github user spmallette commented on the issue: https://github.com/apache/incubator-tinkerpop/pull/325 I've confirmed that this test: ```text GremlinResultSetIntegrateTest.shouldHandleVertexResultWithLiteSerialization:174 ? Execution ``` is only failing after these changes and not on master. I sense it has something to do with this: ```java private void addOrOverrideRegistration(TypeRegistration newRegistration) ``` not preserving the registration identifier, but i could be wrong. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
