Re: get and append file name in record being reading

2016-06-02 Thread Sun Rui
You can use RDD.wholeTextFiles(). For example, suppose all your files are under /tmp/ABC_input/, val rdd = sc.wholeTextFiles("file:///tmp/ABC_input”) val rdd1 = rdd.flatMap { case (path, content) => val fileName = new java.io.File(path).getName content.split("\n").map { line => (lin

get and append file name in record being reading

2016-06-01 Thread Vikash Kumar
How I can get the file name of each record being reading? suppose input file ABC_input_0528.txt contains 111,abc,234 222,xyz,456 suppose input file ABC_input_0531.txt contains 100,abc,299 200,xyz,499 and I need to create one final output with file name in each record using dataframes my output f