thanks Ajay but I have this below code to generate dataframes, So I wanted to change in df only to achieve this. I thought inputFileName will work but it's not working.
private def getPaths: String = { val regex = (conf.namingConvention + conf.extension).replace("?", ".?").replace("**", ".**?") val files = FileUtilities.getFiles(conf.filePath).filter(x => x.getName.matches(regex)) println(s"${files.length} files matched:\n${files.map( x => "-- " + x.getName ).mkString("\n")}") files.map(_.getPath).mkString(",") } private def readTextFile(sqlContext: SQLContext): DataFrame = { System.out.println(s"Reading ${conf.filePath}") sqlContext.read .format("com.databricks.spark.csv") .option("delimiter", conf.delimiter.getOrElse(defaultDelimiter)) .option("header", if (conf.hasHeader.getOrElse(defaultHasHeader)) "true" else "false") .option("quote", if (conf.textQualifier.getOrElse(defaultTextQualifier)) "\"" else null) .schema(conf.schema.toStruct) .load(getPaths) } println("Intaking text file(s)...") *val df: DataFrame = readTextFile(sqlContext)* On Tue, May 31, 2016 at 11:26 PM, Ajay Chander <itsche...@gmail.com> wrote: > Hi Vikash, > > These are my thoughts, read the input directory using wholeTextFiles() > which would give a paired RDD with key as file name and value as file > content. Then you can apply a map function to read each line and append > key to the content. > > Thank you, > Aj > > > On Tuesday, May 31, 2016, Vikash Kumar <vikashsp...@gmail.com> wrote: > >> I have a requirement in which I need to read the input files from a >> directory and append the file name in each record while output. >> >> e.g. I have directory /input/files/ which have folllowing files: >> ABC_input_0528.txt >> ABC_input_0531.txt >> >> suppose input file ABC_input_0528.txt contains >> 111,abc,234 >> 222,xyz,456 >> >> suppose input file ABC_input_0531.txt contains >> 100,abc,299 >> 200,xyz,499 >> >> and I need to create one final output with file name in each record using >> dataframes >> my output file should looks like this: >> 111,abc,234,ABC_input_0528.txt >> 222,xyz,456,ABC_input_0528.txt >> 100,abc,299,ABC_input_0531.txt >> 200,xyz,499,ABC_input_0531.txt >> >> I am trying to use this inputFileName function but it is showing blank. >> >> https://spark.apache.org/docs/1.6.1/api/java/org/apache/spark/sql/functions.html#inputFileName() >> >> Can anybody help me? >> >>