You can flatMap:
rdd.flatMap { in =>
if (condition(in)) {
Some(transformation(in))
} else {
None
}
}
On Thu, Feb 26, 2015 at 6:39 PM, Crystal Xing <crystalxin...@gmail.com> wrote:
> Hi,
> I have a text file input and I want to parse line by line and map each line
> to another format. But at the same time, I want to filter out some lines I
> do not need.
>
> I wonder if there is a way to filter out those lines in the map function.
>
> Do I have to do two steps filter and map? In that way, I have to scan and
> parse the lines twice in order to filter and map.
>
> If I map those unwanted line to null and filter out null, will that work?
> never tried yet.
>
> Thanks,
>
> Zheng zheng
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
