Re: Build keeps failing at giraph-accumulo
Please see GIRAPH-481, depending on where your input data is coming from you might be able to run just fine while we fix this. As long as your build report is SUCCESS for Giraph-Core, you are in good shape. Sorry about the confusion, we're looking into it... On Tue, Jan 15, 2013 at 12:45 PM, Ηλίας Καπουράνης ikapo...@csd.auth.grwrote: Hello there, I recently started reading about Giraph for my Diploma Thesis for graph computations. So I went in to build and install it. I have everything configured (Java and Hadoop) but there is an error which keeps bumping. I used many commands such as mvn test, compile, clean install or even keep the tests out but no luck. The first Warning says that it cannot find POM for org.apache.giraph:giraph:jar:**test:0,2-SNAPSHOT because there are some dependencies missing. Then after the FAILURE message it prints: Failed to execute goal on project giraph-accumulo: Could not resolve dependencies for project org.apache.giraph:giraph-**accumulo:jar:0.2-SNAPSHOT: Failure to find org.apache.giraph:giraph:jar:**tests:0.2-SNAPSHOT in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced - [Help 1] I searched in the mailing issues but no luck there too. Can you please give me any suggestions on how to fix it? Thanks in advance!
Re: Using HiveGiraphRunner with dependencies
Yeah, this is where things get a bit tricky. You'll have to experiment
with what works for you, but we are using Hive to launch the job with
the jar.sh script. This gets the environment straight from the Hive side.
jar_help () {
echo Used for applications that require Hadoop and Hive classpath
and environment.
echo ./hive --service jar yourjar yourclass HIVE_OPTS your_args
}
Avery
On 1/17/13 4:49 PM, pradeep kumar wrote:
Hi,
Actually we are trying to use giraph in our project for graph analysis
with hive, so far it was good build was successful shortestpath
example ran fine but working with hive is been a real issue. we
started with command line
hadoop jar giraph-hcatalog-0.2-SNAPSHOT-jar-with-dependencies.jar
org.apache.giraph.io.hcatalog.HiveGiraphRunner -db default
-vertexClass org.apache.giraph.vertex.Vertex -vertexInputFormatClass
org.apache.giraph.io.hcatalog.HCatalogVertexInputFormat
-vertexOutputFormatClass
org.apache.giraph.io.hcatalog.HCatalogVertexOutputFormat -w 1 -vi
testinput -o testoutput -hiveconf
javax.jdo.option.ConnectionURL=jdbc:mysql://localhost/metastore
-hiveconf javax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver
-hiveconf javax.jdo.option.ConnectionUserName=root -hiveconf
javax.jdo.option.ConnectionPassword=root -hiveconf
datanucleus.autoCreateSchema=false -hiveconf
datanucleus.fixedDatastore=true
is it a wrong way of doing it.. because we are running into exception
while doing so..
and if its wrong,
then any suggestion on how can we proceed will be a great help.
Regards,
Pradeep
Re: Using HiveGiraphRunner with dependencies
Hi avery, thanks for reply and suggestion, while running commands we were getting some exceptions, we are not sure of reasons, may be path or config.. did you guys faced such an issue while developing, if then, what can be wrong..? 13/01/16 15:01:11 WARN conf.HiveConf: DEPRECATED: Ignoring hive-default.xml found on the CLASSPATH at /etc/hive/conf.dist/hive-default.xml 13/01/16 15:01:16 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 13/01/16 15:01:17 INFO metastore.ObjectStore: ObjectStore, initialize called Exception in thread main com.google.common.util.concurrent.UncheckedExecutionException: javax.jdo.JDOFatalInternalException: Unexpected exception caught. NestedThrowables: java.lang.reflect.InvocationTargetException at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2256) at com.google.common.cache.LocalCache.get(LocalCache.java:3980) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4783) at org.apache.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:166) at org.apache.hcatalog.common.HiveClientCache.get(HiveClientCache.java:142) at org.apache.hcatalog.common.HCatUtil.getHiveClient(HCatUtil.java:542) at org.apache.hcatalog.mapreduce.HCatUtils.getInputJobInfo(HCatUtils.java:75) at org.apache.giraph.io.hcatalog.GiraphHCatInputFormat.setVertexInput(GiraphHCatInputFormat.java:81) at org.apache.giraph.io.hcatalog.HiveGiraphRunner.run(HiveGiraphRunner.java:174) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.giraph.io.hcatalog.HiveGiraphRunner.main(HiveGiraphRunner.java:147) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception caught. NestedThrowables: java.lang.reflect.InvocationTargetException at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186) at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803) at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698) at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:246) at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:275) at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:208) at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:183) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.hive.metastore.RetryingRawStore.init(RetryingRawStore.java:62) at org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:346) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:333) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:371) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:278) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:248) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.init(HiveMetaStoreClient.java:114) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.init(HiveMetaStoreClient.java:98) at org.apache.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.init(HiveClientCache.java:245) at org.apache.hcatalog.common.HiveClientCache$4.call(HiveClientCache.java:169) at org.apache.hcatalog.common.HiveClientCache$4.call(HiveClientCache.java:166) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4786) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3579) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2372) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2335) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2250) ... 16 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958) at java.security.AccessController.doPrivileged(Native Method) at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953) at
Re: Using HiveGiraphRunner with dependencies
we just tried with jar_help()... actually earlier error are gone now, unfortunately we getting a new error now.!! Exception in thread main java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.sameThreadExecutor()Lcom/google/common/util/concurrent/ListeningExecutorService; at com.google.common.cache.LocalCache.clinit(LocalCache.java:156) at com.google.common.cache.LocalCache$LocalManualCache.init(LocalCache.java:4765) at com.google.common.cache.CacheBuilder.build(CacheBuilder.java:821) at org.apache.hcatalog.common.HiveClientCache.init(HiveClientCache.java:88) at org.apache.hcatalog.common.HCatUtil.getHiveClient(HCatUtil.java:536) at org.apache.hcatalog.mapreduce.HCatUtils.getInputJobInfo(HCatUtils.java:75) at org.apache.giraph.io.hcatalog.GiraphHCatInputFormat.setVertexInput(GiraphHCatInputFormat.java:81) at org.apache.giraph.io.hcatalog.HiveGiraphRunner.run(HiveGiraphRunner.java:174) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.giraph.io.hcatalog.HiveGiraphRunner.main(HiveGiraphRunner.java:147) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) On Fri, Jan 18, 2013 at 11:23 AM, pradeep kumar pradeep0...@gmail.comwrote: Hi avery, thanks for reply and suggestion, while running commands we were getting some exceptions, we are not sure of reasons, may be path or config.. did you guys faced such an issue while developing, if then, what can be wrong..? 13/01/16 15:01:11 WARN conf.HiveConf: DEPRECATED: Ignoring hive-default.xml found on the CLASSPATH at /etc/hive/conf.dist/hive-default.xml 13/01/16 15:01:16 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 13/01/16 15:01:17 INFO metastore.ObjectStore: ObjectStore, initialize called Exception in thread main com.google.common.util.concurrent.UncheckedExecutionException: javax.jdo.JDOFatalInternalException: Unexpected exception caught. NestedThrowables: java.lang.reflect.InvocationTargetException at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2256) at com.google.common.cache.LocalCache.get(LocalCache.java:3980) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4783) at org.apache.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:166) at org.apache.hcatalog.common.HiveClientCache.get(HiveClientCache.java:142) at org.apache.hcatalog.common.HCatUtil.getHiveClient(HCatUtil.java:542) at org.apache.hcatalog.mapreduce.HCatUtils.getInputJobInfo(HCatUtils.java:75) at org.apache.giraph.io.hcatalog.GiraphHCatInputFormat.setVertexInput(GiraphHCatInputFormat.java:81) at org.apache.giraph.io.hcatalog.HiveGiraphRunner.run(HiveGiraphRunner.java:174) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.giraph.io.hcatalog.HiveGiraphRunner.main(HiveGiraphRunner.java:147) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception caught. NestedThrowables: java.lang.reflect.InvocationTargetException at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186) at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803) at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698) at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:246) at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:275) at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:208) at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:183) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.hive.metastore.RetryingRawStore.init(RetryingRawStore.java:62) at org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:346) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:333) at
