[
https://issues.apache.org/jira/browse/TINKERPOP-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485516#comment-15485516
]
Marko A. Rodriguez commented on TINKERPOP-1389:
-----------------------------------------------
I tested SparkServer/SparkGraphComputer manually. If there were any
{{<dependency/>}}-problems, then it would show up here. Typically, in the past,
Netty and javax.servlet versioning issues popped up here. No issues. Sweet.
{code}
gremlin> hdfs
==>storage[DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1105121616_1,
ugi=marko (auth:SIMPLE)]]]
gremlin> hdfs.copyFromLocal('data/tinkerpop-modern.kryo',
'tinkerpop-modern.kryo')
==>null
gremlin> hdfs.copyFromLocal('data/grateful-dead.kryo', 'grateful-dead.kryo')
==>null
gremlin> hdfs.ls()
==>rw-r--r-- marko supergroup 332226 grateful-dead.kryo
==>rw-r--r-- marko supergroup 781 tinkerpop-modern.kryo
gremlin> graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
==>hadoopgraph[gryoinputformat->gryooutputformat]
gremlin> g = graph.traversal().withComputer(SparkGraphComputer)
==>graphtraversalsource[hadoopgraph[gryoinputformat->gryooutputformat],
sparkgraphcomputer]
gremlin> g.V().count()
WARN org.apache.spark.SparkContext - Use an existing SparkContext, some
configuration may not take effect.
WARN org.apache.spark.SparkContext - Use an existing SparkContext, some
configuration may not take effect.
WARN org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer -
HADOOP_GREMLIN_LIBS is not set -- proceeding regardless
==>6
gremlin>
graph.configuration.setProperty('gremlin.hadoop.inputLocation','grateful-dead.kryo')
==>null
gremlin> g.V().count()
WARN org.apache.spark.SparkContext - Use an existing SparkContext, some
configuration may not take effect.
WARN org.apache.spark.SparkContext - Use an existing SparkContext, some
configuration may not take effect.
WARN org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer -
HADOOP_GREMLIN_LIBS is not set -- proceeding regardless
==>808
gremlin> g.V().out().out().count()
WARN org.apache.spark.SparkContext - Use an existing SparkContext, some
configuration may not take effect.
WARN org.apache.spark.SparkContext - Use an existing SparkContext, some
configuration may not take effect.
WARN org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer -
HADOOP_GREMLIN_LIBS is not set -- proceeding regardless
==>327370
gremlin> g.V().repeat(both()).times(5).groupCount().by('name')
WARN org.apache.spark.SparkContext - Use an existing SparkContext, some
configuration may not take effect.
WARN org.apache.spark.SparkContext - Use an existing SparkContext, some
configuration may not take effect.
WARN org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer -
HADOOP_GREMLIN_LIBS is not set -- proceeding regardless
==>[YOUNG BLOOD:36345386,BANKS OF
OHIO:85127885,FRANCE:84769579,Van_Morrison:16102028,RIP IT UP:44381195,TOUGH
MAMA:47380220,AROUND AND AROUND:9244685325,MARDI GRAS
PARADE:98599196,WEREWOLVES OF LONDON:1449989105,Winwood_Davis:39344,SHE BELONGS
TO ME:1225165040,LADY WITH A FAN:92857446,BROKEN
ARROW:3612302287,Jones_Edwards:393117,OH BOY:352130336,YOU CANT CATCH
ME:6048018,CHINA DOLL:3771061788,RIVER DEEP MOUNTAIN
HIGH:463420,Jimmy_Reed:6460189,Welnick_Bralove:26062666,BALLAD OF FRANKIE LEE
AND JUDAS PRIEST:6008372,Leiber_Stoller:5290286,HELL IN A
BUCKET:9199936179,Marvin_Baumgardner:393117,Naomi_Neville:15064,Pickett_Cropper:27195,Williamson:14199208,Grateful_Dead:273516191,YOU
AINT WOMAN ENOUGH:2310156219,THROWING STONES:4242415158,OH BABE IT AINT NO
LIE:1066021928,NEIGHBORHOOD GIRLS:73005367,KEEP ON GROWING:606596608,ONE MORE
SATURDAY NIGHT:9000871029,RAMBLE ON
ROSE:10904428685,West_Tilghman_Holly:3850855,ROSA LEE MCFALL:44381195,ME AND
BOBBY
MCGEE:4634389961,Owens_Howard:393117,Bruce_Hornsby:13822011,Stephen_Stills:6888060,
...
{code}
> Support Spark 2.0.0
> -------------------
>
> Key: TINKERPOP-1389
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1389
> Project: TinkerPop
> Issue Type: Improvement
> Components: hadoop
> Affects Versions: 3.2.1
> Reporter: Chen Xin Yu
> Assignee: Marko A. Rodriguez
> Fix For: 3.3.0
>
> Attachments: TINKERPOP-1389.patch
>
>
> Spark 2.0.0 was released:
> http://spark.apache.org/news/spark-2-0-0-released.html
> There are lots of improvement and changes compared to 1.6.1, we should better
> bump to it for TinkerPop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
