Anyone who can guide me how to reduce the Size from Long to Int since I dont need Long index. I am huge data and this index talking 8 bytes, if i can reduce it to 4 bytes will be great help?
On 22 April 2015 at 22:46, Jeetendra Gangele <gangele...@gmail.com> wrote: > Sure thanks. if you can guide me how to do this will be great help. > > On 17 April 2015 at 22:05, Ted Yu <yuzhih...@gmail.com> wrote: > >> I have some assignments on hand at the moment. >> >> Will try to come up with sample code after I clear the assignments. >> >> FYI >> >> On Thu, Apr 16, 2015 at 2:00 PM, Jeetendra Gangele <gangele...@gmail.com> >> wrote: >> >>> Can you please guide me how can I extend RDD and convert into this way >>> you are suggesting. >>> >>> On 16 April 2015 at 23:46, Jeetendra Gangele <gangele...@gmail.com> >>> wrote: >>> >>>> I type T i already have Object ... I have RDD<Object> and then I am >>>> calling ZipWithIndex on this RDD and getting RDD<Object,Long> on this I am >>>> running MapToPair and converting into RDD<Long,Object> so that i can use it >>>> later for other operation like lookup and join. >>>> >>>> >>>> On 16 April 2015 at 23:42, Ted Yu <yuzhih...@gmail.com> wrote: >>>> >>>>> The Long in RDD[(T, Long)] is type parameter. You can create RDD with >>>>> Integer as the first type parameter. >>>>> >>>>> Cheers >>>>> >>>>> On Thu, Apr 16, 2015 at 11:07 AM, Jeetendra Gangele < >>>>> gangele...@gmail.com> wrote: >>>>> >>>>>> Hi Ted. >>>>>> This works for me. But since Long takes here 8 bytes. Can I reduce it >>>>>> to 4 bytes. its just a index and I feel 4 bytes was more than >>>>>> enough.is there any method which takes Integer or similar for Index? >>>>>> >>>>>> >>>>>> On 13 April 2015 at 01:59, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>> >>>>>>> bq. will return something like JavaPairRDD<Object, long> >>>>>>> >>>>>>> The long component of the pair fits your description of index. What >>>>>>> other requirement does ZipWithIndex not provide you ? >>>>>>> >>>>>>> Cheers >>>>>>> >>>>>>> On Sun, Apr 12, 2015 at 1:16 PM, Jeetendra Gangele < >>>>>>> gangele...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi All I have an RDD JavaRDD<Object> and I want to convert it to >>>>>>>> JavaPairRDD<Index,Object>.. Index should be unique and it should >>>>>>>> maintain >>>>>>>> the order. For first object It should have 1 and then for second 2 like >>>>>>>> that. >>>>>>>> >>>>>>>> I tried using ZipWithIndex but it will return something like >>>>>>>> JavaPairRDD<Object, long> >>>>>>>> I wanted to use this RDD for lookup and join operation later in my >>>>>>>> workflow so ordering is important. >>>>>>>> >>>>>>>> >>>>>>>> Regards >>>>>>>> jeet >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>>> >>> >>> >>> >>> >> > > > >
