RE: Spark 1.6.0: substring on df.select

Thanks Raghav. I have 5+ million records. I feel creating multiple come is not an optimal way. Please suggest any other alternate solution. Can’t we do any string operation in DF.Select? Regards, Raja From: Raghavendra Pandey Sent: 11 May 2016 09:04 PM To: Bharathi Raja Cc: User Subject: Re

Spark 1.6.0: substring on df.select

Hi, I have a dataframe column col1 with values something like “/client/service/version/method”. The number of “/” are not constant. Could you please help me to extract all methods from the column col1? In Pig i used SUBSTRING with LAST_INDEX_OF(“/”). Thanks in advance. Regards, Raja

How to ignore case in dataframe groupby?

Hi, Values in a dataframe column named countrycode are in different cases. Eg: (US, us). groupBy & count gives two rows but the requirement is to ignore case for this operation. 1) Is there a way to ignore case in groupBy? Or 2) Is there a way to update the dataframe column countrycode to

RE: How to Parse & flatten JSON object in a text file using Spark into Dataframe

Thanks Eran, I'll check the solution. Regards, Raja -Original Message- From: "Eran Witkon" <eranwit...@gmail.com> Sent: ‎12/‎24/‎2015 4:07 PM To: "Bharathi Raja" <raja...@yahoo.com>; "Gokula Krishnan D" <email2...@gmail.com> Cc: "u

RE: How to Parse & flatten JSON object in a text file using Spark into Dataframe

Thanks Gokul, but the file I have had the same format as I have mentioned. First two columns are not in Json format. Thanks, Raja -Original Message- From: "Gokula Krishnan D" Sent: ‎12/‎24/‎2015 2:44 AM To: "Eran Witkon" Cc: "raja kbv"

RE: How to Parse & flatten JSON object in a text file using Spark into Dataframe

Hi Eran, I didn't get the solution yet. Thanks, Raja -Original Message- From: "Eran Witkon" Sent: ‎12/‎23/‎2015 8:17 PM To: "raja kbv" ; "user@spark.apache.org" Subject: Re: How to Parse & flatten JSON object in a text