Re: writable class to be used to read floating point values from input?

2008-10-26 Thread pols cut 
Thanks ..

I converted the text-->string --> Float.

I am trying to calculate the average of a very large set of numbers. You are 
right...I plan to use a dummy key (its not null as i said before) as input to 
reduce. Then in reduce when sorted, i will have a single record as 
> which i will use to calculate the avg.

Regards,

Pavan







From: Owen O'Malley <[EMAIL PROTECTED]>
To: core-user@hadoop.apache.org
Sent: Sunday, 26 October, 2008 1:24:43 AM
Subject: Re: writable class to be used to read floating point values from input?


On Oct 25, 2008, at 8:32 PM, pols cut wrote:

> I am trying to write a map reduce function which takes take the  
> following types of  pairs
>
>
> Map function -- should read floating point values (i dont really  
> care about  key)
> it should output 

If the input is stored in a text file, using TextInputFormat is right.  
Your map inputs will be:

LongWritable, Text

Just use the Text and convert it to a Double.

> reduce -- input- 
>output 

This doesn't make any sense. How should the input to the reduce be  
sorted? By the float? In that case, it would be:

FloatWritable, NullWritable

You will get one call to the reduce for each distinct float value the  
maps generate. The reduce can iterate through the NullWritables to see  
how many times that key was generated.

-- Owen



  Bollywood news, movie reviews, film trailers and more! Go to 
http://in.movies.yahoo.com/

Re: writable class to be used to read floating point values from input?

2008-10-25 Thread Owen O'Malley 


On Oct 25, 2008, at 8:32 PM, pols cut wrote:

I am trying to write a map reduce function which takes take the  
following types of  pairs



Map function -- should read floating point values (i dont really  
care about  key)

it should output 


If the input is stored in a text file, using TextInputFormat is right.  
Your map inputs will be:


LongWritable, Text

Just use the Text and convert it to a Double.


reduce -- input- 
   output 


This doesn't make any sense. How should the input to the reduce be  
sorted? By the float? In that case, it would be:


FloatWritable, NullWritable

You will get one call to the reduce for each distinct float value the  
maps generate. The reduce can iterate through the NullWritables to see  
how many times that key was generated.


-- Owen

Re: writable class to be used to read floating point values from input?

2008-10-25 Thread Jeremy Chow 
Hi pols,

>
>
> I Have set the input format to be TextInputFormat.class   (is this right?)


> Please tell me what signature am I supposed to use for the Map/ reduce
> methods?
>  as of now I am trying to write the Map function as
>
> public static class Map extends MapReduceBase implements
> Mapper
>{
>public void map(LongWritable key,  FloatWritable value,
> OutputCollector output, Reporter reporter) throws
> IOException
> { ...
> ..
> }
> }
>
> But I am getting an error saying text cannot be cast to longwritable.


Unfortunately no, you are wrong. The default key-value type for
TextInputFormat is . A  type named LineRecordReader,
which implmented a method LineRecordeReader.next(LongWritable key, Text
value), is the default reader of TextInputFormat.  If you wanna change the
key-value type that reveived in map function, you can define your own
InputFormat and RecordReader.

-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

http://coderplay.javaeye.com

RE: writable class to be used to read floating point values from input?

2008-10-25 Thread Malcolm Matalka 
What are you collecting in your map?  Are you just passing 'key'?  And are you 
sure you want to use map reduce for this job?  It doesn't seem to make sense to 
me, but I'm rather new.



-Original Message-
From: pols cut [mailto:[EMAIL PROTECTED]
Sent: Sat 10/25/2008 8:32 PM
To: core-user@hadoop.apache.org
Subject: writable class to be used to read floating point values from input?
 
I am trying to write a map reduce function which takes take the following types 
of  pairs


Map function -- should read floating point values (i dont really care about  
key)
it should output 

reduce -- input- 
output 

I Have set the input format to be TextInputFormat.class   (is this right?)

Please tell me what signature am I supposed to use for the Map/ reduce methods?
  as of now I am trying to write the Map function as 

public static class Map extends MapReduceBase implements Mapper
{
public void map(LongWritable key,  FloatWritable value, 
OutputCollector output, Reporter reporter) throws 
IOException
{ ...
..
}
}

But I am getting an error saying text cannot be cast to longwritable. 



  Add more friends to your messenger and enjoy! Go to 
http://messenger.yahoo.com/invite/

writable class to be used to read floating point values from input?

2008-10-25 Thread pols cut 
I am trying to write a map reduce function which takes take the following types 
of  pairs


Map function -- should read floating point values (i dont really care about  
key)
it should output 

reduce -- input- 
output 

I Have set the input format to be TextInputFormat.class   (is this right?)

Please tell me what signature am I supposed to use for the Map/ reduce methods?
  as of now I am trying to write the Map function as 

public static class Map extends MapReduceBase implements Mapper
{
public void map(LongWritable key,  FloatWritable value, 
OutputCollector output, Reporter reporter) throws 
IOException
{ ...
..
}
}

But I am getting an error saying text cannot be cast to longwritable. 



  Add more friends to your messenger and enjoy! Go to 
http://messenger.yahoo.com/invite/