Its for performance optimisation .There are 2 requirements 1.I am gonna consume data on daily basis. Gonna run query on hive table and fetch today's incremental data which I got from RDBMS and query on that. 2.Gonna run cumulative distinct user query on whole set.
Shall I merge output of each mapper to reduce no of files.if yes then how? Does hadoop has some api for that. On Tue, May 6, 2014 at 4:08 AM, Db-Blog <[email protected]> wrote: > In general it is recommended to have Millions of Large files rather than > billions of small files in hadoop. > > Please describe your issues in detail. Say for ex. > -How are you planning to consume the data stored in this partition table? > - Are you looking for storage and performance optimizations? Etc. > > Thanks > Saurabh > > Sent from my iPhone, please avoid typos. > > > On 05-May-2014, at 3:33 pm, Shushant Arora <[email protected]> > wrote: > > > > I have a hive table in which data is populated from RDBMS on daily basis. > > > > After map reduce each mapper write its data in hive table partitioned at > month level. > > Issue is daily when job runs it fetches data of last day and each mapper > writes its output in seperate file. Shall I merge those files in single one > ? > > > > What should be file format? Sequence file or text is better ? > > > > >
