I compiled spark with SPARK_HADOOP_VERSION=2.4.0 sbt/sbt assembly, fixed the s3 dependencies, but I am still getting the same error... 14/05/05 00:32:33 WARN TaskSetManager: Loss was due to org.apache.hadoop.ipc.RemoteException org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on
Any ideas? thanks 2014-05-04 11:53 GMT-03:00 Andre Kuhnen <[email protected]>: > Thanks Mayur, the only think that my code is doing is: > > read from s3, and saveAsTextFile on hdfs. Like I said, everything is > written correctly, but at the end of the job there is this warnning, > I will try to compile with hadoop 2.4 > thanks > > > > > 2014-05-04 11:17 GMT-03:00 Mayur Rustagi <[email protected]>: > > You should compile Spark with every hadoop version you use. I am surprised >> its working otherwise as HDFS breaks compatibility quite often. >> As for this error it comes when your code writes/reads from file that has >> already deleted. Are you trying to update a single file in multiple >> mappers/reduce partitioners? >> >> >> Mayur Rustagi >> Ph: +1 (760) 203 3257 >> http://www.sigmoidanalytics.com >> @mayur_rustagi <https://twitter.com/mayur_rustagi> >> >> >> >> On Sun, May 4, 2014 at 5:30 PM, Andre Kuhnen <[email protected]>wrote: >> >>> Please, can anyone give a feedback? thanks >>> >>> Hello, I am getting this warning after upgrading Hadoop 2.4, when I try >>> to write something to the HDFS. The content is written correctly, but I >>> do not like this warning. >>> >>> DO I have to compile SPARK with hadoop 2.4? >>> >>> WARN TaskSetManager: Loss was due to >>> org.apache.hadoop.ipc.RemoteException >>> >>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException) >>> >>> thanks >>> >>> >>> 2014-05-03 13:09 GMT-03:00 Andre Kuhnen <[email protected]>: >>> >>> Hello, I am getting this warning after upgrading Hadoop 2.4, when I try >>>> to write something to the HDFS. The content is written correctly, but I >>>> do not like this warning. >>>> >>>> DO I have to compile SPARK with hadoop 2.4? >>>> >>>> WARN TaskSetManager: Loss was due to >>>> org.apache.hadoop.ipc.RemoteException >>>> >>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException) >>>> >>>> thanks >>>> >>>> >>> >> >
