[
https://issues.apache.org/jira/browse/SPARK-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14276436#comment-14276436
]
Florian Verhein commented on SPARK-3185:
----------------------------------------
I'm also getting this, though with "Server IPC version 9" now that I'm using
hadoop 2.4.1 (modification of the various hadoop init.sh scripts). I'm also
using spark 1.2.0.
My understanding is that spark-1.2.0-bin-hadoop2.4.tgz is built against hadoop
2.4 and tachyon 0.4.1.
But I suspect the tachyon 0.4.1 that is installed in the spark-ec2 scripts is
built against hadoop 1...
Does this mean building tachyon against hadoop 2.4.1 would fix this?
> SPARK launch on Hadoop 2 in EC2 throws Tachyon exception when Formatting
> JOURNAL_FOLDER
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-3185
> URL: https://issues.apache.org/jira/browse/SPARK-3185
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.0.2
> Environment: Amazon Linux AMI
> [ec2-user@ip-172-30-1-145 ~]$ uname -a
> Linux ip-172-30-1-145 3.10.42-52.145.amzn1.x86_64 #1 SMP Tue Jun 10 23:46:43
> UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> https://aws.amazon.com/amazon-linux-ami/2014.03-release-notes/
> The build I used (and MD5 verified):
> [ec2-user@ip-172-30-1-145 ~]$ wget
> http://supergsego.com/apache/spark/spark-1.0.2/spark-1.0.2-bin-hadoop2.tgz
> Reporter: Jeremy Chambers
>
> {code}
> org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot
> communicate with client version 4
> {code}
> When I launch SPARK 1.0.2 on Hadoop 2 in a new EC2 cluster, the above tachyon
> exception is thrown when "Formatting JOURNAL_FOLDER".
> No exception occurs when I launch on Hadoop 1.
> Launch used:
> {code}
> ./spark-ec2 -k spark_cluster -i /home/ec2-user/kagi/spark_cluster.ppk
> --zone=us-east-1a --hadoop-major-version=2 --spot-price=0.0165 -s 3 launch
> sparkProd
> {code}
> {code}
> ----log snippet----
> Formatting Tachyon Master @ ec2-54-80-49-244.compute-1.amazonaws.com
> Formatting JOURNAL_FOLDER: /root/tachyon/libexec/../journal/
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot
> communicate with client version 4
> at tachyon.util.CommonUtils.runtimeException(CommonUtils.java:246)
> at tachyon.UnderFileSystemHdfs.<init>(UnderFileSystemHdfs.java:73)
> at tachyon.UnderFileSystemHdfs.getClient(UnderFileSystemHdfs.java:53)
> at tachyon.UnderFileSystem.get(UnderFileSystem.java:53)
> at tachyon.Format.main(Format.java:54)
> Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot
> communicate with client version 4
> at org.apache.hadoop.ipc.Client.call(Client.java:1070)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
> at
> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
> at tachyon.UnderFileSystemHdfs.<init>(UnderFileSystemHdfs.java:69)
> ... 3 more
> Killed 0 processes
> Killed 0 processes
> ec2-54-167-219-159.compute-1.amazonaws.com: Killed 0 processes
> ec2-54-198-198-17.compute-1.amazonaws.com: Killed 0 processes
> ec2-54-166-36-0.compute-1.amazonaws.com: Killed 0 processes
> ---end snippet---
> {code}
> *I don't have this problem when I launch without the
> "--hadoop-major-version=2" (which defaults to Hadoop 1.x).*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
