It is possible to access the filename, its a bit tricky though.

 val fstream = ssc.fileStream[LongWritable, IntWritable,
      SequenceFileInputFormat[LongWritable,
IntWritable]]("/home/akhld/input/")

    fstream.foreach(x =>{
      //You can get it with this object.
      println(x.values.toDebugString)

    } )

[image: Inline image 1]

Thanks
Best Regards

On Wed, Apr 29, 2015 at 8:33 AM, [email protected] <[email protected]> wrote:

> For the SparkContext#textFile, if a directory is given as the path
> parameter  ,then it will pick up the files in the directory, so the same
> thing will occur.
>
> ------------------------------
> [email protected]
>
>
> *From:* Saisai Shao <[email protected]>
> *Date:* 2015-04-29 10:54
> *To:* Vadim Bichutskiy <[email protected]>
> *CC:* [email protected]; lokeshkumar <[email protected]>; user
> <[email protected]>
> *Subject:* Re: Re: Spark streaming - textFileStream/fileStream - Get file
> name
> I think it might be useful in Spark Streaming's file input stream, but not
> sure is it useful in SparkContext#textFile, since we specify the file by
> our own, so why we still need to know the file name.
>
> I will open up a JIRA to mention about this feature.
>
> Thanks
> Jerry
>
>
> 2015-04-29 10:49 GMT+08:00 Vadim Bichutskiy <[email protected]>:
>
>> I was wondering about the same thing.
>>
>> Vadim
>> ᐧ
>>
>> On Tue, Apr 28, 2015 at 10:19 PM, [email protected] <[email protected]>
>> wrote:
>>
>>> Looks to me  that the same thing also applies to the
>>> SparkContext.textFile or SparkContext.wholeTextFile, there is no way in RDD
>>> to figure out the file information where the data in RDD is from
>>>
>>> ------------------------------
>>> [email protected]
>>>
>>>
>>> *From:* Saisai Shao <[email protected]>
>>> *Date:* 2015-04-29 10:10
>>> *To:* lokeshkumar <[email protected]>
>>> *CC:* spark users <[email protected]>
>>> *Subject:* Re: Spark streaming - textFileStream/fileStream - Get file
>>> name
>>> I think currently there's no API in Spark Streaming you can use to get
>>> the file names for file input streams. Actually it is not trivial to
>>> support this, may be you could file a JIRA with wishes you want the
>>> community to support, so anyone who is interested can take a crack on this.
>>>
>>> Thanks
>>> Jerry
>>>
>>>
>>> 2015-04-29 0:13 GMT+08:00 lokeshkumar <[email protected]>:
>>>
>>>> Hi Forum,
>>>>
>>>> Using spark streaming and listening to the files in HDFS using
>>>> textFileStream/fileStream methods, how do we get the fileNames which are
>>>> read by these methods?
>>>>
>>>> I used textFileStream which has file contents in JavaDStream and I got
>>>> no
>>>> success with fileStream as it is throwing me a compilation error with
>>>> spark
>>>> version 1.3.1.
>>>>
>>>> Can someone please tell me if we have an API function or any other way
>>>> to
>>>> get the file names that these streaming methods read?
>>>>
>>>> Thanks
>>>> Lokesh
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-textFileStream-fileStream-Get-file-name-tp22692.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [email protected]
>>>> For additional commands, e-mail: [email protected]
>>>>
>>>>
>>>
>>
>

Reply via email to