Hi Anchit, this is not my expected, because you specified the HDFS directory in your code. I've solved like this:
val text = sc.hadoopFile(Args.input, classOf[TextInputFormat], classOf[LongWritable], classOf[Text], 2) val hadoopRdd = text.asInstanceOf[HadoopRDD[LongWritable, Text]] hadoopRdd.mapPartitionsWithInputSplit((inputSplit, iterator) => { val file = inputSplit.asInstanceOf[FileSplit] terator.map ( tp => {tp._1, new Text(file.toString + “,” + tp._2.toString)}) } > On Sep 25, 2015, at 13:12, Anchit Choudhry <anchit.choud...@gmail.com> wrote: > > Hi Fengdong, > > So I created two files in HDFS under a test folder. > > test/dt=20100101.json > { "key1" : "value1" } > > test/dt=20100102.json > { "key2" : "value2" } > > Then inside PySpark shell > > rdd = sc.wholeTextFiles('./test/*') > rdd.collect() > [(u'hdfs://localhost:9000/user/hduser/test/dt=20100101.json', u'{ "key1" : > "value1" }), (u'hdfs://localhost:9000/user/hduser/test/dt=20100102.json', u'{ > "key2" : "value2" })] > import json > def editMe(y, x): > j = json.loads(y) > j['source'] = x > return j > > rdd.map(lambda (x,y): editMe(y,x)).collect() > [{'source': u'hdfs://localhost:9000/user/hduser/test/dt=20100101.json', > u'key1': u'value1'}, {u'key2': u'value2', 'source': > u'hdfs://localhost:9000/user/hduser/test/dt=20100102.json'}] > > Similarly you could modify the function to return 'source' and 'date' with > some string manipulation per your requirements. > > Let me know if this helps. > > Thanks, > Anchit > > > On 24 September 2015 at 23:55, Fengdong Yu <fengdo...@everstring.com > <mailto:fengdo...@everstring.com>> wrote: > > yes. such as I have two data sets: > > date set A: /data/test1/dt=20100101 > data set B: /data/test2/dt=20100202 > > > all data has the same JSON format , such as: > {“key1” : “value1”, “key2” : “value2” } > > > my output expected: > {“key1” : “value1”, “key2” : “value2” , “source” : “test1”, “date” : > “20100101"} > {“key1” : “value1”, “key2” : “value2” , “source” : “test2”, “date” : > “20100202"} > > >> On Sep 25, 2015, at 11:52, Anchit Choudhry <anchit.choud...@gmail.com >> <mailto:anchit.choud...@gmail.com>> wrote: >> >> Sure. May I ask for a sample input(could be just few lines) and the output >> you are expecting to bring clarity to my thoughts? >> >> On Thu, Sep 24, 2015, 23:44 Fengdong Yu <fengdo...@everstring.com >> <mailto:fengdo...@everstring.com>> wrote: >> Hi Anchit, >> >> Thanks for the quick answer. >> >> my exact question is : I want to add HDFS location into each line in my JSON >> data. >> >> >> >>> On Sep 25, 2015, at 11:25, Anchit Choudhry <anchit.choud...@gmail.com >>> <mailto:anchit.choud...@gmail.com>> wrote: >>> >>> Hi Fengdong, >>> >>> Thanks for your question. >>> >>> Spark already has a function called wholeTextFiles within sparkContext >>> which can help you with that: >>> >>> Python >>> hdfs://a-hdfs-path/part-00000 >>> hdfs://a-hdfs-path/part-00001 >>> ... >>> hdfs://a-hdfs-path/part-nnnnn >>> rdd = sparkContext.wholeTextFiles(“hdfs://a- <>hdfs-path”) >>> (a-hdfs-path/part-00000, its content) >>> (a-hdfs-path/part-00001, its content) >>> ... >>> (a-hdfs-path/part-nnnnn, its content) >>> More info: http://spark >>> <http://spark/>.apache.org/docs/latest/api/python/pyspark.html?highlight=wholetext#pyspark.SparkContext.wholeTextFiles >>> >>> ------------ >>> >>> Scala >>> >>> val rdd = sparkContext.wholeTextFile("hdfs://a-hdfs-path") >>> >>> More info: >>> https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkContext@wholeTextFiles(String,Int):RDD[(String,String)] >>> >>> Let us know if this helps or you need more help. >>> >>> Thanks, >>> Anchit Choudhry >>> >>> On 24 September 2015 at 23:12, Fengdong Yu <fengdo...@everstring.com >>> <mailto:fengdo...@everstring.com>> wrote: >>> Hi, >>> >>> I have multiple files with JSON format, such as: >>> >>> /data/test1_data/sub100/test.data >>> /data/test2_data/sub200/test.data >>> >>> >>> I can sc.textFile(“/data/*/*”) >>> >>> but I want to add the {“source” : “HDFS_LOCATION”} to each line, then save >>> it the one target HDFS location. >>> >>> how to do it, Thanks. >>> >>> >>> >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>> <mailto:dev-unsubscr...@spark.apache.org> >>> For additional commands, e-mail: dev-h...@spark.apache.org >>> <mailto:dev-h...@spark.apache.org> >>> >>> >> > >