Here are exact numbers:
# of (k,v) pairs = 1.2 Mil this is same. 
# of unique k = 1000, k is integer.
# of  unique v = 1Mil, v is a big big string.
For a given k, cumulative size of all v associated to it is about 30 Mb.
(That is each v is about 25~30Kb)
# of Mappers = 30
# of Reducers = 10

(v,k) is atleast 4/5 times faster than (k,v).

-Gyanit


Scott Carey wrote:
> 
> Well if the smaller keys are producing fewer unique values, there should
> be some more significant differences.
> 
> I had assumed that your test produced the same number of unique values.
> 
> I'm still not sure why there would be that significant of a difference as
> long as the total number of unique values in the small key test is a good
> deal larger than the number of reducers and there is not too much skew in
> the bucket sizes.  If there are a small subset of keys in the small key
> test that contain a large subset of the values, then the reducers will
> have very skewed work sizes and this could explain your observation.
> 
> 
> On 3/11/09 11:50 AM, "Gyanit" <[email protected]> wrote:
> 
> 
> 
> I notices one more thing. Lighter keys tend to make smaller number of
> unique
> keys.
> For example (key,value) pairs may be 10Mil, but if key is lighter unique
> keys might be just 1000.
> In other case if keys are heavier unique keys might be 5 mil.
> I think this might have something to do with it.
> Bottom line: If your reduce is simple dump and no combining, the put data
> in
> keys than values.
> 
> I need to put data in values. Any suggestions on how to make it faster.
> 
> -Gyanit.
> 
> 
> Scott Carey wrote:
>>
>> That is a fascinating question.  I would also love to know the reason
>> behind this.
>>
>> If I were to guess I would have thought that smaller keys and heavier
>> values would slightly outperform, rather than significantly underperform.
>> (assuming total pair count at each phase is the same).   Perhaps there is
>> room for optimization here?
>>
>>
>>
>> On 3/10/09 6:44 PM, "Gyanit" <[email protected]> wrote:
>>
>>
>>
>> I have large number of key,value pairs. I don't actually care if data
>> goes
>> in
>> value or key. Let me be more exact.
>> (k,v) pair after combiner is about 1 mil. I have approx 1kb data for each
>> pair. I can put it in keys or values.
>> I have experimented with both options (heavy key , light value)  vs
>> (light
>> key, heavy value). It turns out that hk,lv option is much much better
>> than
>> (lk,hv).
>> Has someone else also noticed this?
>> Is there a way to make things faster in light key , heavy value option.
>> As
>> some application will need that also.
>> Remember in both cases we are talking about atleast dozen or so million
>> pairs.
>> There is a difference of time in shuffle phase. Which is weird as amount
>> of
>> data transferred is same.
>>
>> -gyanit
>> --
>> View this message in context:
>> http://www.nabble.com/Why-is-large-number-of---%28heavy%29-keys-%2C-%28light%29-value--faster-than-%28light%29key-%2C-%28heavy%29-value-tp22447877p22447877.html
>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>
>>
>>
>>
> 
> --
> View this message in context:
> http://www.nabble.com/Why-is-large-number-of---%28heavy%29-keys-%2C-%28light%29-value--faster-than-%28light%29key-%2C-%28heavy%29-value-tp22447877p22463049.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
> 
> 
> 
> 

-- 
View this message in context: 
http://www.nabble.com/Why-is-large-number-of---%28heavy%29-keys-%2C-%28light%29-value--faster-than-%28light%29key-%2C-%28heavy%29-value-tp22447877p22463784.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Reply via email to