I was wondering about the same thing. Vadim ᐧ
On Tue, Apr 28, 2015 at 10:19 PM, bit1...@163.com <bit1...@163.com> wrote: > Looks to me that the same thing also applies to the SparkContext.textFile > or SparkContext.wholeTextFile, there is no way in RDD to figure out the > file information where the data in RDD is from > > ------------------------------ > bit1...@163.com > > > *From:* Saisai Shao <sai.sai.s...@gmail.com> > *Date:* 2015-04-29 10:10 > *To:* lokeshkumar <lok...@dataken.net> > *CC:* spark users <user@spark.apache.org> > *Subject:* Re: Spark streaming - textFileStream/fileStream - Get file name > I think currently there's no API in Spark Streaming you can use to get the > file names for file input streams. Actually it is not trivial to support > this, may be you could file a JIRA with wishes you want the community to > support, so anyone who is interested can take a crack on this. > > Thanks > Jerry > > > 2015-04-29 0:13 GMT+08:00 lokeshkumar <lok...@dataken.net>: > >> Hi Forum, >> >> Using spark streaming and listening to the files in HDFS using >> textFileStream/fileStream methods, how do we get the fileNames which are >> read by these methods? >> >> I used textFileStream which has file contents in JavaDStream and I got no >> success with fileStream as it is throwing me a compilation error with >> spark >> version 1.3.1. >> >> Can someone please tell me if we have an API function or any other way to >> get the file names that these streaming methods read? >> >> Thanks >> Lokesh >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-textFileStream-fileStream-Get-file-name-tp22692.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
