If your file is not very large, try
sc.wholeTextFiles("...").values.flatMap(_.split("\n").grouped(4).map(_.mkString("\n")))
-Xiangrui
On Sat, Oct 25, 2014 at 12:57 AM, Parthus <peng.wei....@gmail.com> wrote:
> Hi,
>
> It might be a naive question, but I still wish that somebody could help me
> handle it.
>
> I have a textFile, in which every 4 lines represent a record. Since
> SparkContext.textFile() API deems of one line as a record, it does not fit
> into my case. I know that SparkContext.hadoopFile or newAPIHadoopFile API
> can read a file in an arbitrary format, but I do not know how to use them. I
> think that there must be some API which can easily solve this problem, but I
> am kind of a bad googler and cannot find it by myself online.
>
> Would it be possible for somebody to tell me how to use the API? I run Spark
> based on Hadoop 1.2.1 rather than Hadoop 2.x. I wish that I could get
> several lines of code which actually works, if possible.
>
> Thanks very much.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Read-a-TextFile-1-record-contains-4-lines-into-a-RDD-tp17256.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
