[jira] [Commented] (SPARK-1884) Shark failed to start
[ https://issues.apache.org/jira/browse/SPARK-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049877#comment-14049877 ] Pete MacKinnon commented on SPARK-1884: --- This is due to the version of protobuf-java provided by Shark being older (2.4.1) than what's needed by Hadoop 2.4 (2.5.0). See SPARK-2338. Shark failed to start - Key: SPARK-1884 URL: https://issues.apache.org/jira/browse/SPARK-1884 Project: Spark Issue Type: Bug Affects Versions: 0.9.1 Environment: ubuntu 14.04, spark 0.9.1, hive 0.13.0, hadoop 2.4.0 (stand alone), scala 2.11.0 Reporter: Wei Cui Priority: Blocker the hadoop, hive, spark works fine. when start the shark, it failed with the following messages: Starting the Shark Command Line Client 14/05/19 16:47:21 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive 14/05/19 16:47:21 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize 14/05/19 16:47:21 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize 14/05/19 16:47:21 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack 14/05/19 16:47:21 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node 14/05/19 16:47:21 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 14/05/19 16:47:21 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 14/05/19 16:47:22 WARN conf.Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@48c724c:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 14/05/19 16:47:22 WARN conf.Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@48c724c:an attempt to override final parameter: mapreduce.cluster.local.dir; Ignoring. 14/05/19 16:47:22 WARN conf.Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@48c724c:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 14/05/19 16:47:22 WARN conf.Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@48c724c:an attempt to override final parameter: mapreduce.cluster.temp.dir; Ignoring. Logging initialized using configuration in jar:file:/usr/local/shark/lib_managed/jars/edu.berkeley.cs.shark/hive-common/hive-common-0.11.0-shark-0.9.1.jar!/hive-log4j.properties Hive history file=/tmp/root/hive_job_log_root_14857@ubuntu_201405191647_897494215.txt 6.004: [GC 279616K-18440K(1013632K), 0.0438980 secs] 6.445: [Full GC 59125K-7949K(1013632K), 0.0685160 secs] Reloading cached RDDs from previous Shark sessions... (use -skipRddReload flag to skip reloading) 7.535: [Full GC 104136K-13059K(1013632K), 0.0885820 secs] 8.459: [Full GC 61237K-18031K(1013632K), 0.0820400 secs] 8.662: [Full GC 29832K-8958K(1013632K), 0.0869700 secs] 8.751: [Full GC 13433K-8998K(1013632K), 0.0856520 secs] 10.435: [Full GC 72246K-14140K(1013632K), 0.1797530 secs] Exception in thread main org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1072) at shark.memstore2.TableRecovery$.reloadRdds(TableRecovery.scala:49) at shark.SharkCliDriver.init(SharkCliDriver.scala:283) at shark.SharkCliDriver$.main(SharkCliDriver.scala:162) at shark.SharkCliDriver.main(SharkCliDriver.scala) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1139) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:51) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2288) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2299) at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1070) ... 4 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
[jira] [Commented] (SPARK-1884) Shark failed to start
[ https://issues.apache.org/jira/browse/SPARK-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14002427#comment-14002427 ] Wei Cui commented on SPARK-1884: It's Caused by: java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$SetOwnerRequestProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; Shark failed to start - Key: SPARK-1884 URL: https://issues.apache.org/jira/browse/SPARK-1884 Project: Spark Issue Type: Bug Affects Versions: 0.9.1 Environment: ubuntu 14.04, spark 0.9.1, hive 0.13.0, hadoop 2.4.0 (stand alone), scala 2.11.0 Reporter: Wei Cui Priority: Blocker the hadoop, hive, spark works fine. when start the shark, it failed with the following messages: Starting the Shark Command Line Client 14/05/19 16:47:21 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive 14/05/19 16:47:21 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize 14/05/19 16:47:21 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize 14/05/19 16:47:21 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack 14/05/19 16:47:21 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node 14/05/19 16:47:21 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 14/05/19 16:47:21 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 14/05/19 16:47:22 WARN conf.Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@48c724c:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 14/05/19 16:47:22 WARN conf.Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@48c724c:an attempt to override final parameter: mapreduce.cluster.local.dir; Ignoring. 14/05/19 16:47:22 WARN conf.Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@48c724c:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 14/05/19 16:47:22 WARN conf.Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@48c724c:an attempt to override final parameter: mapreduce.cluster.temp.dir; Ignoring. Logging initialized using configuration in jar:file:/usr/local/shark/lib_managed/jars/edu.berkeley.cs.shark/hive-common/hive-common-0.11.0-shark-0.9.1.jar!/hive-log4j.properties Hive history file=/tmp/root/hive_job_log_root_14857@ubuntu_201405191647_897494215.txt 6.004: [GC 279616K-18440K(1013632K), 0.0438980 secs] 6.445: [Full GC 59125K-7949K(1013632K), 0.0685160 secs] Reloading cached RDDs from previous Shark sessions... (use -skipRddReload flag to skip reloading) 7.535: [Full GC 104136K-13059K(1013632K), 0.0885820 secs] 8.459: [Full GC 61237K-18031K(1013632K), 0.0820400 secs] 8.662: [Full GC 29832K-8958K(1013632K), 0.0869700 secs] 8.751: [Full GC 13433K-8998K(1013632K), 0.0856520 secs] 10.435: [Full GC 72246K-14140K(1013632K), 0.1797530 secs] Exception in thread main org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1072) at shark.memstore2.TableRecovery$.reloadRdds(TableRecovery.scala:49) at shark.SharkCliDriver.init(SharkCliDriver.scala:283) at shark.SharkCliDriver$.main(SharkCliDriver.scala:162) at shark.SharkCliDriver.main(SharkCliDriver.scala) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1139) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:51) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2288) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2299) at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1070) ... 4 more Caused by: java.lang.reflect.InvocationTargetException at
