Here is my command: aws emr create-cluster --release-label emr-4.3.0 --name "ClusterJava8" --use-default-roles --applications Name=Ganglia Name=Hive Name=Hue Name=Mahout Name=Pig Name=Spark --ec2-attributes KeyName=CC-ES-Demo --instance-count 3 --instance-type m3.xlarge --use-default-roles --bootstrap-action Path=s3://crayon-emr-scripts/emr_java_8.sh
I am using bootstrap script to install java 8. When I choose applications (Name=Ganglia Name=Hive Name=Hue Name=Mahout Name=Pig Name=Spark) problem is gone. I fixed on the way Lzo not found exception. Now I have another problem that I have no idea why it happens: I tries to copy file to hdfs and got this exception (file is very small , just couple of kb). org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /input/test.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1550) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3110) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3034) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:723) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:632) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1407) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:238) at com.sun.proxy.$Proxy9.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1441) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454) put: File /input/test.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. On Wed, Mar 2, 2016 at 4:09 AM, Gourav Sengupta <gourav.sengu...@gmail.com> wrote: > Hi, > > which region are you using the EMR clusters from? Is there any tweaking of > the HADOOP parameters that you are doing before starting the clusters? > > If you are using AWS CLI to start the cluster just send across the command. > > I have, never till date, faced any such issues in the Ireland region. > > > Regards, > Gourav Sengupta > > On Tue, Mar 1, 2016 at 9:15 AM, Oleg Ruchovets <oruchov...@gmail.com> > wrote: > >> Hi , I am installed EMR 4.3.0 with spark. I tries to enter spark shell >> but it looks it does't work and throws exceptions. >> Please advice: >> >> [hadoop@ip-172-31-39-37 conf]$ cd /usr/bin/ >> [hadoop@ip-172-31-39-37 bin]$ ./spark-shell >> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; >> support was removed in 8.0 >> 16/03/01 09:11:48 INFO SecurityManager: Changing view acls to: hadoop >> 16/03/01 09:11:48 INFO SecurityManager: Changing modify acls to: hadoop >> 16/03/01 09:11:48 INFO SecurityManager: SecurityManager: authentication >> disabled; ui acls disabled; users with view permissions: Set(hadoop); users >> with modify permissions: Set(hadoop) >> 16/03/01 09:11:49 INFO HttpServer: Starting HTTP Server >> 16/03/01 09:11:49 INFO Utils: Successfully started service 'HTTP class >> server' on port 47223. >> Welcome to >> ____ __ >> / __/__ ___ _____/ /__ >> _\ \/ _ \/ _ `/ __/ '_/ >> /___/ .__/\_,_/_/ /_/\_\ version 1.6.0 >> /_/ >> >> Using Scala version 2.10.5 (OpenJDK 64-Bit Server VM, Java 1.8.0_71) >> Type in expressions to have them evaluated. >> Type :help for more information. >> 16/03/01 09:11:53 INFO SparkContext: Running Spark version 1.6.0 >> 16/03/01 09:11:53 INFO SecurityManager: Changing view acls to: hadoop >> 16/03/01 09:11:53 INFO SecurityManager: Changing modify acls to: hadoop >> 16/03/01 09:11:53 INFO SecurityManager: SecurityManager: authentication >> disabled; ui acls disabled; users with view permissions: Set(hadoop); users >> with modify permissions: Set(hadoop) >> 16/03/01 09:11:54 INFO Utils: Successfully started service 'sparkDriver' >> on port 52143. >> 16/03/01 09:11:54 INFO Slf4jLogger: Slf4jLogger started >> 16/03/01 09:11:54 INFO Remoting: Starting remoting >> 16/03/01 09:11:54 INFO Remoting: Remoting started; listening on addresses >> :[akka.tcp://sparkDriverActorSystem@172.31.39.37:42989] >> 16/03/01 09:11:54 INFO Utils: Successfully started service >> 'sparkDriverActorSystem' on port 42989. >> 16/03/01 09:11:54 INFO SparkEnv: Registering MapOutputTracker >> 16/03/01 09:11:54 INFO SparkEnv: Registering BlockManagerMaster >> 16/03/01 09:11:54 INFO DiskBlockManager: Created local directory at >> /mnt/tmp/blockmgr-afaf0e7f-086e-49f1-946d-798e605a3fdc >> 16/03/01 09:11:54 INFO MemoryStore: MemoryStore started with capacity >> 518.1 MB >> 16/03/01 09:11:55 INFO SparkEnv: Registering OutputCommitCoordinator >> 16/03/01 09:11:55 INFO Utils: Successfully started service 'SparkUI' on >> port 4040. >> 16/03/01 09:11:55 INFO SparkUI: Started SparkUI at >> http://172.31.39.37:4040 >> 16/03/01 09:11:55 INFO RMProxy: Connecting to ResourceManager at / >> 172.31.39.37:8032 >> 16/03/01 09:11:55 INFO Client: Requesting a new application from cluster >> with 2 NodeManagers >> 16/03/01 09:11:55 INFO Client: Verifying our application has not >> requested more than the maximum memory capability of the cluster (11520 MB >> per container) >> 16/03/01 09:11:55 INFO Client: Will allocate AM container, with 896 MB >> memory including 384 MB overhead >> 16/03/01 09:11:55 INFO Client: Setting up container launch context for >> our AM >> 16/03/01 09:11:55 INFO Client: Setting up the launch environment for our >> AM container >> 16/03/01 09:11:55 INFO Client: Preparing resources for our AM container >> 16/03/01 09:11:56 INFO Client: Uploading resource >> file:/usr/lib/spark/lib/spark-assembly-1.6.0-hadoop2.7.1-amzn-0.jar -> >> hdfs:// >> 172.31.39.37:8020/user/hadoop/.sparkStaging/application_1456818849676_0005/spark-assembly-1.6.0-hadoop2.7.1-amzn-0.jar >> 16/03/01 09:11:56 INFO MetricsSaver: MetricsConfigRecord >> disabledInCluster: false instanceEngineCycleSec: 60 clusterEngineCycleSec: >> 60 disableClusterEngine: false maxMemoryMb: 3072 maxInstanceCount: 500 >> lastModified: 1456818856695 >> 16/03/01 09:11:56 INFO MetricsSaver: Created MetricsSaver >> j-2FT6QNFSPTHNX:i-5f6bcadb:SparkSubmit:04807 period:60 >> /mnt/var/em/raw/i-5f6bcadb_20160301_SparkSubmit_04807_raw.bin >> 16/03/01 09:11:56 WARN DFSClient: DataStreamer Exception >> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File >> /user/hadoop/.sparkStaging/application_1456818849676_0005/spark-assembly-1.6.0-hadoop2.7.1-amzn-0.jar >> could only be replicated to 0 nodes instead of minReplication (=1). There >> are 0 datanode(s) running and no node(s) are excluded in this operation. >> at >> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1550) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3110) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3034) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:723) >> at >> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) >> at >> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:632) >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:422) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1476) >> at org.apache.hadoop.ipc.Client.call(Client.java:1407) >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:238) >> at com.sun.proxy.$Proxy16.addBlock(Unknown Source) >> at >> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) >> at com.sun.proxy.$Proxy17.addBlock(Unknown Source) >> at >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1441) >> at >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237) >> at >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454) >> 16/03/01 09:11:56 INFO Client: Deleting staging directory >> .sparkStaging/application_1456818849676_0005 >> 16/03/01 09:11:56 ERROR SparkContext: Error initializing SparkContext. >> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File >> /user/hadoop/.sparkStaging/application_1456818849676_0005/spark-assembly-1.6.0-hadoop2.7.1-amzn-0.jar >> could only be replicated to 0 nodes instead of minReplication (=1). There >> are 0 datanode(s) running and no node(s) are excluded in this operation. >> at >> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1550) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3110) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3034) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:723) >> at >> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) >> at >> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:632) >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:422) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1476) >> at org.apache.hadoop.ipc.Client.call(Client.java:1407) >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:238) >> at com.sun.proxy.$Proxy16.addBlock(Unknown Source) >> at >> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) >> at com.sun.proxy.$Proxy17.addBlock(Unknown Source) >> at >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1441) >> at >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237) >> at >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454) >> 16/03/01 09:11:56 INFO SparkUI: Stopped Spark web UI at >> http://172.31.39.37:4040 >> 16/03/01 09:11:56 INFO YarnClientSchedulerBackend: Stopped >> 16/03/01 09:11:56 INFO MapOutputTrackerMasterEndpoint: >> MapOutputTrackerMasterEndpoint stopped! >> 16/03/01 09:11:56 ERROR Utils: Uncaught exception in thread main >> java.lang.NullPointerException >> at >> org.apache.spark.network.shuffle.ExternalShuffleClient.close(ExternalShuffleClient.java:152) >> at org.apache.spark.storage.BlockManager.stop(BlockManager.scala:1231) >> at org.apache.spark.SparkEnv.stop(SparkEnv.scala:96) >> at >> org.apache.spark.SparkContext$$anonfun$stop$12.apply$mcV$sp(SparkContext.scala:1756) >> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1229) >> at org.apache.spark.SparkContext.stop(SparkContext.scala:1755) >> at org.apache.spark.SparkContext.<init>(SparkContext.scala:602) >> at >> org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017) >> at $line3.$read$$iwC$$iwC.<init>(<console>:15) >> at $line3.$read$$iwC.<init>(<console>:24) >> at $line3.$read.<init>(<console>:26) >> at $line3.$read$.<init>(<console>:30) >> at $line3.$read$.<clinit>(<console>) >> at $line3.$eval$.<init>(<console>:7) >> at $line3.$eval$.<clinit>(<console>) >> at $line3.$eval.$print(<console>) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at >> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) >> at >> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346) >> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) >> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) >> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) >> at >> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857) >> at >> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) >> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) >> at >> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:125) >> at >> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124) >> at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324) >> at >> org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124) >> at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974) >> at >> org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159) >> at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64) >> at >> org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108) >> at >> org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) >> at >> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) >> at org.apache.spark.repl.SparkILoop.org >> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945) >> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) >> at org.apache.spark.repl.Main$.main(Main.scala:31) >> at org.apache.spark.repl.Main.main(Main.scala) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at >> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) >> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> 16/03/01 09:11:56 INFO SparkContext: Successfully stopped SparkContext >> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File >> /user/hadoop/.sparkStaging/application_1456818849676_0005/spark-assembly-1.6.0-hadoop2.7.1-amzn-0.jar >> could only be replicated to 0 nodes instead of minReplication (=1). There >> are 0 datanode(s) running and no node(s) are excluded in this operation. >> at >> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1550) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3110) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3034) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:723) >> at >> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) >> at >> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:632) >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:422) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1476) >> at org.apache.hadoop.ipc.Client.call(Client.java:1407) >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:238) >> at com.sun.proxy.$Proxy16.addBlock(Unknown Source) >> at >> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) >> at com.sun.proxy.$Proxy17.addBlock(Unknown Source) >> at >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1441) >> at >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237) >> at >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454) >> >> java.lang.NullPointerException >> at >> org.apache.spark.sql.SQLContext$.createListenerAndUI(SQLContext.scala:1367) >> at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101) >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >> at >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >> at >> org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028) >> at $iwC$$iwC.<init>(<console>:15) >> at $iwC.<init>(<console>:24) >> at <init>(<console>:26) >> at .<init>(<console>:30) >> at .<clinit>(<console>) >> at .<init>(<console>:7) >> at .<clinit>(<console>) >> at $print(<console>) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at >> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) >> at >> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346) >> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) >> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) >> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) >> at >> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857) >> at >> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) >> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) >> at >> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:132) >> at >> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124) >> at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324) >> at >> org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124) >> at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974) >> at >> org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159) >> at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64) >> at >> org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108) >> at >> org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) >> at >> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) >> at org.apache.spark.repl.SparkILoop.org >> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945) >> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) >> at org.apache.spark.repl.Main$.main(Main.scala:31) >> at org.apache.spark.repl.Main.main(Main.scala) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at >> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) >> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> >> <console>:16: error: not found: value sqlContext >> import sqlContext.implicits._ >> ^ >> <console>:16: error: not found: value sqlContext >> import sqlContext.sql >> ^ >> >> >
