*First Question:*
On Amazon S3 I have a directory with 1024 files, where each file size is
~9Mb; and each line in a file has two entries separated by '\t'.
Here is my program, which is calculating total number of entries in the
dataset
--
val inputId = sc.textFile(inputhPath, noParts).flatMap {line=>
val lineArray = line.split("\\t")
Iterator(lineArray(0).toLong, lineArray(1).toLong)
}.distinct(noParts)
println("######input-cnt = %s; ".
format(inputId.count))
--
Where inputpath =
"s3n://my-AWS_ACCESS_KEY_ID:myAWS_ACCESS_KEY_SECRET@bucket-id/directory"
When I run this program multiple times on EC2, "input-cnt" across
iterations is not consistent. FYI, I uploaded the data to S3 two days back;
so I assume by now data is properly replicated/(eventually-concistency).
*
Is this a known issue with S3? What it the solution?
*
Note: When I ran same experiment on my yarn cluster; where inputhPath is
hdfs-path, I got the results as expected.
*Second Question:*
How can I copy data from s3 to ephemeral-hdfs? I tried distcp, but I got
this error:
--
With failures, global counters are inaccurate; consider running with -i
Copy failed: java.net.ConnectException: Call to
ec2-54-227-208-124.compute-1.amazonaws.com/10.187.1y.120:9001 failed on
connection exception: java.net.ConnectException: Connection refused
--
(ideally I can use sc.textfile to read and write to hdfs; but I am not sure
if I will get all the data with sc.textFile due to the issue I mentioned
above)
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Readin-from-Amazon-S3-behaves-inconsistently-return-different-number-of-lines-tp11092.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
