RE: Checking for null values when mapping

2016-02-20 Thread Mich Talebzadeh
ll not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Technology Ltd, its subsidiaries nor their employees acce

Re: Checking for null values when mapping

2016-02-20 Thread Chandeep Singh
message shall not be understood as given or endorsed by Peridale Technology >>> Ltd, its subsidiaries or their employees, unless expressly so stated. It is >>> the responsibility of the recipient to ensure that this email is virus >>> free, therefore neither Peridale Technol

Re: Checking for null values when mapping

2016-02-20 Thread Chandeep Singh
ty. >> >> >> From: Chandeep Singh [mailto:c...@chandeep.com <mailto:c...@chandeep.com>] >> Sent: 20 February 2016 13:47 >> To: Mich Talebzadeh mailto:m...@peridale.co.uk>> >> Cc: user @spark mailto:user@spark.apache.org>> >&g

RE: Checking for null values when mapping

2016-02-20 Thread Mich Talebzadeh
Singh [mailto:c...@chandeep.com] Sent: 20 February 2016 14:27 To: Mich Talebzadeh Cc: user @spark Subject: Re: Checking for null values when mapping Also, have you looked into Dos2Unix (http://dos2unix.sourceforge.net/) Has helped me in the past to deal with special characters while using w

Re: Checking for null values when mapping

2016-02-20 Thread Chandeep Singh
y Ltd, its subsidiaries nor their > employees accept any responsibility. > > > From: Chandeep Singh [mailto:c...@chandeep.com] > Sent: 20 February 2016 13:47 > To: Mich Talebzadeh > Cc: user @spark > Subject: Re: Checking for null values when mapping > > Looks l

RE: Checking for null values when mapping

2016-02-20 Thread Mich Talebzadeh
From: Chandeep Singh [mailto:c...@chandeep.com] Sent: 20 February 2016 13:47 To: Mich Talebzadeh Cc: user @spark Subject: Re: Checking for null values when mapping Looks like you’re using substring just to get rid of the ‘?’. Why not use replace for that as well? And then you wouldn’t run into

Re: Checking for null values when mapping

2016-02-20 Thread Chandeep Singh
Looks like you’re using substring just to get rid of the ‘?’. Why not use replace for that as well? And then you wouldn’t run into issues with index out of bound. val a = "?1,187.50" val b = "" println(a.substring(1).replace(",", "”)) —> 1187.50 println(a.replace("?", "").replace(",", "”)) —

Re: Checking for null values when mapping

2016-02-20 Thread Ted Yu
by Peridale Technology Ltd, its > subsidiaries or their employees, unless expressly so stated. It is the > responsibility of the recipient to ensure that this email is virus free, > therefore neither Peridale Technology Ltd, its subsidiaries nor their > employees accept any

RE: Checking for null values when mapping

2016-02-20 Thread Mich Talebzadeh
responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Technology Ltd, its subsidiaries nor their employees accept any responsibility. From: Michał Zieliński [mailto:zielinski.mich...@gmail.com] Sent: 20 February 2016 08:59 To: Mich Talebzadeh C

Re: Checking for null values when mapping

2016-02-20 Thread Michał Zieliński
You can use filter and isNotNull on Column before the map. On 20 February 2016 at 08:24, Mich Talebzadeh wrote: > > > I have a DF like below reading a csv file > > > > > > val df = > HiveContext.read.format("com.databr