Hi,
I'm trying to get data from S3 using sc.textFile("s3n://"+filenamePattern)
It seems that if a pattern gives out no result i get an exception like so:
org.apache.hadoop.mapred.InvalidInputException: Input Pattern
s3n://bucket/20140512/* matches 0 files
at
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:197)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
I'm actually doing a union of multiple RDD so some may have data in them but
some won't.
Is there anyway to say "ignore empty patterns" so that it can work with the
RDDs that actually found files ?
Thanks
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Ignoring-S3-0-files-exception-tp6101.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
