Mike, Any suggestions on doing it for consequitive id's? On Aug 5, 2016 9:08 AM, "Tony Lane" <tonylane....@gmail.com> wrote:
> Mike. > > I have figured how to do this . Thanks for the suggestion. It works > great. I am trying to figure out the performance impact of this. > > thanks again > > > On Fri, Aug 5, 2016 at 9:25 PM, Tony Lane <tonylane....@gmail.com> wrote: > >> @mike - this looks great. How can i do this in java ? what is the >> performance implication on a large dataset ? >> >> @sonal - I can't have a collision in the values. >> >> On Fri, Aug 5, 2016 at 9:15 PM, Mike Metzger <m...@flexiblecreations.com> >> wrote: >> >>> You can use the monotonically_increasing_id method to generate >>> guaranteed unique (but not necessarily consecutive) IDs. Calling something >>> like: >>> >>> df.withColumn("id", monotonically_increasing_id()) >>> >>> You don't mention which language you're using but you'll need to pull in >>> the sql.functions library. >>> >>> Mike >>> >>> On Aug 5, 2016, at 9:11 AM, Tony Lane <tonylane....@gmail.com> wrote: >>> >>> Ayan - basically i have a dataset with structure, where bid are unique >>> string values >>> >>> bid: String >>> val : integer >>> >>> I need unique int values for these string bid''s to do some processing >>> in the dataset >>> >>> like >>> >>> id:int (unique integer id for each bid) >>> bid:String >>> val:integer >>> >>> >>> >>> -Tony >>> >>> On Fri, Aug 5, 2016 at 6:35 PM, ayan guha <guha.a...@gmail.com> wrote: >>> >>>> Hi >>>> >>>> Can you explain a little further? >>>> >>>> best >>>> Ayan >>>> >>>> On Fri, Aug 5, 2016 at 10:14 PM, Tony Lane <tonylane....@gmail.com> >>>> wrote: >>>> >>>>> I have a row with structure like >>>>> >>>>> identifier: String >>>>> value: int >>>>> >>>>> All identifier are unique and I want to generate a unique long id for >>>>> the data and get a row object back for further processing. >>>>> >>>>> I understand using the zipWithUniqueId function on RDD, but that would >>>>> mean first converting to RDD and then joining back the RDD and dataset >>>>> >>>>> What is the best way to do this ? >>>>> >>>>> -Tony >>>>> >>>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> Ayan Guha >>>> >>> >>> >> >