something like this
df.filter('transactiontype > " ").filter(not('transactiontype ==="DEB") &&
not('transactiontype ==="BGC")).select('transactiontype).*distinct*
.collect.foreach(println)
HTH
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6
Hi
If you want column wise distinct, you may need to define it. Will it be
possible to demonstrate your problem with an example? Like what's the input
and output. Maybe with few columns..
On 19 Sep 2016 20:36, "Abhishek Anand" wrote:
> Hi Ayan,
>
> How will I get column wise distinct items using
Hi Ayan,
How will I get column wise distinct items using this approach ?
On Mon, Sep 19, 2016 at 3:31 PM, ayan guha wrote:
> Create an array out of cilumns, convert to Dataframe,
> explode,distinct,write.
> On 19 Sep 2016 19:11, "Saurav Sinha" wrote:
>
>> You can use distinct over you data fra
Create an array out of cilumns, convert to Dataframe,
explode,distinct,write.
On 19 Sep 2016 19:11, "Saurav Sinha" wrote:
> You can use distinct over you data frame or rdd
>
> rdd.distinct
>
> It will give you distinct across your row.
>
> On Mon, Sep 19, 2016 at 2:35 PM, Abhishek Anand
> wrote:
You can use distinct over you data frame or rdd
rdd.distinct
It will give you distinct across your row.
On Mon, Sep 19, 2016 at 2:35 PM, Abhishek Anand
wrote:
> I have an rdd which contains 14 different columns. I need to find the
> distinct across all the columns of rdd and write it to hdfs.
I have an rdd which contains 14 different columns. I need to find the
distinct across all the columns of rdd and write it to hdfs.
How can I acheive this ?
Is there any distributed data structure that I can use and keep on updating
it as I traverse the new rows ?
Regards,
Abhi
