Null map step (at a guess?), 3 step reduce. No problem. Suspect 3 may be rather long running?
*.......* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud of smoke,thoroughly used up, totally worn out, and loudly proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* On Tue, Feb 3, 2015 at 6:44 PM, 임정택 <kabh...@gmail.com> wrote: > Hello all. > > We're periodically scan HBase tables to aggregate statistic information, > and store it to MySQL. > > We have 3 kinds of CP (kind of data source), each has one Channel and one > Article table. > (Channel : Article is 1:N relation.) > > All CPs table schema are different a bit, so in order to aggregate we > should apply different logics, with joining Channel and Article. > > I've thought about workflow like this, but I wonder it can make sense. > > 1. run single process which initializes MySQL by creating table, deleting > row, etc. > 2. run 3 M/Rs simultaneously to aggregate statistic information for each > CP, and insert rows per Channel to MySQL. > 3. run single process which finalizes whole aggregation - runs aggregation > query from MySQL to insert new row to MySQL, rolling table, etc. > > Definitely 1,2,3 should be run in a row. > > Any helps are really appreciated! > Thanks. > > Regards. > Jungtaek Lim (HeartSaVioR) >
