If Arsalan publishes the modified version of MultiHFileoutputFormat, that would expedite adding support in 0.94 After some validation, this can be backported.
Cheers On Wed, Nov 2, 2011 at 10:14 AM, Christopher Dorner < [email protected]> wrote: > Will HBase 0.92 support MultiHFileoutputFormat and IncrementalLoad for > different Tables? > > Is there a comfortable way to make it work for HBase 0.90.4 as well? I am > using Cloudera's CDH3u2. > > > Am 30.10.2011 12:57, schrieb Christopher Dorner: > > Hi, >> >> i am facing a similar problem. I Need to read a large file to put into >> different hbase tables. Until now i have done it with >> MultiTableOutputFormat directly from the Mapper. Thats works ok, but i >> believe it will become quite slow when i try larger files. But I thought >> it is a good chance to try bulk load as well and compare them because i >> think it can be much faster in my scenario. >> >> It works fast and like a charm for one table. But i want to avoid >> reading the file a few times. >> >> Did you finally manage to make it work? Also with the lack of >> "configureIncrementalLoad" in the MultiHFileOutputFormat class? I am >> having only one family. >> >> Regards, >> Christopher >> >> >> >> Am -10.01.-28163 20:59, schrieb Arsalan Bilal: >> >>> Now I have redefine "MultiHfileOutputFormat" and "HfileOutputFormat" >>> code. >>> And it works fine for my solution. It generate data for multiple >>> families, >>> for multiple table and create multiple HFiles accordingly to related >>> Family >>> and save it at different paths. >>> >>> Thanks Andrew for indicating right direction and Thanks all community for >>> help >>> >>> >>> >>> On Tue, Sep 27, 2011 at 11:45 AM, Arsalan >>> Bilal<[email protected]**>wrote: >>> >>> Now this time, i am trying to achieve my objective by playing with >>>> HfileOutputFormat and MultiHfileOutputFormat. But did not achieve my >>>> target. >>>> So i need your help >>>> I am using two >>>> context.write(key, kv1); >>>> context.write(key, kv2); >>>> then it should write/generate data/Hfile on two different paths for two >>>> different tables. I am trying to achieve this. Plz help. >>>> >>>> >>>> >>>> On Mon, Sep 26, 2011 at 8:26 PM, Stack<[email protected]> wrote: >>>> >>>> Can you play around w/ the class to make it work for your case? >>>>> St.Ack >>>>> >>>>> On Mon, Sep 26, 2011 at 9:24 AM, Arsalan >>>>> Bilal<[email protected]**> >>>>> wrote: >>>>> >>>>>> @Andrew >>>>>> >>>>>> Can u help me? U suggest that Try this: >>>>>> https://gist.github.com/**1237770 <https://gist.github.com/1237770>, >>>>>> i >>>>>> saw it, How we can we use it actually? >>>>>> i try to use it as >>>>>> >>>>> job.setOutputFormatClass(**MultiHFileOutputFormat.class); >>>>> >>>>>> but it generate different column Families(not correct) inside one >>>>>> folder(recursively). I need to generate different column families in >>>>>> different folders. >>>>>> And there is also lack of configureIncrementalLoad Function in your >>>>>> >>>>> pointed >>>>> >>>>>> file. >>>>>> >>>>>> Plz help me Andrew, Stack, i need your help. >>>>>> >>>>>> >>>>>> On Mon, Sep 26, 2011 at 6:48 PM, Shuja Rehman<[email protected]> >>>>>> >>>>> wrote: >>>>> >>>>>> >>>>>> @arslan, >>>>>>> I did not get chance to look at it. >>>>>>> >>>>>>> >>>>>>> On Mon, Sep 26, 2011 at 12:11 PM, Arsalan Bilal< >>>>>>> >>>>>> [email protected]>**wrote: >>>>> >>>>>> >>>>>>> I am also facing same kind of problem, I am using one Map only and >>>>>>>> >>>>>>> need to >>>>> >>>>>> generate multiple Hfiles for multiple table for insertion in Hbase? >>>>>>>> Have you found any solution ? Can u help me to find solution of this >>>>>>>> >>>>>>> one? >>>>> >>>>>> >>>>>>>> >>>>>>>> On Fri, Sep 23, 2011 at 8:11 PM, Andrew Purtell<[email protected] >>>>>>>> >>>>>>> wrote: >>>>>> >>>>>>> >>>>>>>> Try this: >>>>>>>> https://gist.github.com/**1237770<https://gist.github.com/1237770> >>>>>>>>> >>>>>>>>> See line 135. >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> >>>>>>>>> >>>>>>>>> - Andy >>>>>>>>> >>>>>>>>> >>>>>>>>> Problems worthy of attack prove their worth by hitting back. - Piet >>>>>>>>> >>>>>>>> Hein >>>>> >>>>>> (via Tom White) >>>>>>>>> >>>>>>>>> >>>>>>>>> ______________________________**__ >>>>>>>>>> From: Shuja Rehman<[email protected]> >>>>>>>>>> To: [email protected]; Andrew Purtell<[email protected]> >>>>>>>>>> Sent: Friday, September 23, 2011 8:18 AM >>>>>>>>>> Subject: Re: Creation of Hfiles for multiple tables using Single >>>>>>>>>> >>>>>>>>> Bulk >>>>> >>>>>> Load Job? >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> andrew, >>>>>>>>>> >>>>>>>>>> can you guide me where to change and what to change? it will help. >>>>>>>>>> thnx >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Sep 22, 2011 at 9:35 PM, Shuja Rehman< >>>>>>>>>> >>>>>>>>> [email protected]> >>>>> >>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Can you update it so I can use it? :) >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Sep 22, 2011 at 8:40 PM, Andrew Purtell< >>>>>>>>>>> >>>>>>>>>> [email protected]> >>>>> >>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> From: Shuja Rehman<[email protected]> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I am using bulk load to insert data into hbase. Its runs >>>>>>>>>>>>> fine if >>>>>>>>>>>>> >>>>>>>>>>>> I >>>>> >>>>>> need to >>>>>>>>> >>>>>>>>>> insert in one table. But Now, I have the requirement in which I >>>>>>>>>>>>> >>>>>>>>>>>> need >>>>> >>>>>> to >>>>>>>>> >>>>>>>>>> insert data into more than one table. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> We started some support for this here: >>>>>>>>>>>> >>>>>>>>>>> https://issues.apache.org/**jira/browse/HBASE-3727<https://issues.apache.org/jira/browse/HBASE-3727> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> It needs to be updated for HFile v2, I think, a couple of single >>>>>>>>>>>> >>>>>>>>>>> line >>>>> >>>>>> changes. >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> Best regards, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> - Andy >>>>>>>>>>>> >>>>>>>>>>>> Problems worthy of attack prove their worth by hitting back. - >>>>>>>>>>>> >>>>>>>>>>> Piet >>>>> >>>>>> Hein (via Tom White) >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> >>>>>>>>>>>>> From: Shuja Rehman<[email protected]> >>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>> Cc: >>>>>>>>>>>>> Sent: Thursday, September 22, 2011 8:01 AM >>>>>>>>>>>>> Subject: Creation of Hfiles for multiple tables using Single >>>>>>>>>>>>> >>>>>>>>>>>> Bulk >>>>> >>>>>> Load Job? >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>> Hi >>>>>>>>>>>>> >>>>>>>>>>>>> I am using bulk load to insert data into hbase. Its runs >>>>>>>>>>>>> fine if >>>>>>>>>>>>> >>>>>>>>>>>> I >>>>> >>>>>> need to >>>>>>>>> >>>>>>>>>> insert in one table. But Now, I have the requirement in which I >>>>>>>>>>>>> >>>>>>>>>>>> need >>>>> >>>>>> to >>>>>>>>> >>>>>>>>>> insert data into more than one table. But it seems that using >>>>>>>>>>>>> >>>>>>>>>>>> bulk >>>>> >>>>>> load, we >>>>>>>>> >>>>>>>>>> can generate hfiles for 1 table only. So the question is that >>>>>>>>>>>>> >>>>>>>>>>>> is >>>>> >>>>>> there any >>>>>>>>> >>>>>>>>>> solution which enable us to generate hfiles in multiple output >>>>>>>>>>>>> >>>>>>>>>>>> paths >>>>> >>>>>> and >>>>>>>>> >>>>>>>>>> then we can load these different files in different hbase >>>>>>>>>>>>> >>>>>>>>>>>> tables. >>>>> >>>>>> >>>>>>>>>>>>> Thanks in advance >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Regards >>>>>>>>>>>>> Shuja-ur-Rehman Baig >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> <http://pk.linkedin.com/in/**shujamughal<http://pk.linkedin.com/in/shujamughal> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Regards >>>>>>>>>>> Shuja-ur-Rehman Baig >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Regards >>>>>>>>>> Shuja-ur-Rehman Baig >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Best Regards, >>>>>>>> Arsalan Bilal >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards >>>>>>> Shuja-ur-Rehman Baig >>>>>>> <http://pk.linkedin.com/in/**shujamughal<http://pk.linkedin.com/in/shujamughal> >>>>>>> > >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Best Regards, >>>>>> Arsalan Bilal >>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> Arsalan Bilal >>>> >>>> >>>> >>>> >>> >>> >> >
