to
keep it within the same rack.
All of this is dependent a proper topology configuration, both in your
NameNode and JobTracker.
On Thu, Sep 12, 2013 at 3:02 PM, Adrian CAPDEFIER
chivas314...@gmail.comwrote:
Thanks Bryan.
Yes, I am using hadoop + hdfs.
If I understand your point, hadoop
can learn about it more here: http://tez.incubator.apache.org/
HTH,
+Vinod
On Sep 12, 2013, at 6:36 AM, Adrian CAPDEFIER wrote:
Howdy,
My application requires 2 distinct processing steps (reducers) to be
performed on the input data. The first operation generates changes the key
values
Howdy,
My application requires 2 distinct processing steps (reducers) to be
performed on the input data. The first operation generates changes the key
values and, records that had different keys in step 1 can end up having the
same key in step 2.
The heavy lifting of the operation is in step1
logic and they map it to the various mapper/reducer steps automatically.
Hope this helps,
Chris
On Thu, Sep 12, 2013 at 9:36 AM, Adrian CAPDEFIER
chivas314...@gmail.comwrote:
Howdy,
My application requires 2 distinct processing steps (reducers) to be
performed on the input data
that the mappers in the second job do as good a job as
possible to be data or rack-local.
On Thu, Sep 12, 2013 at 12:35 PM, Adrian CAPDEFIER chivas314...@gmail.com
wrote:
Thank you, Chris. I will look at Cascading and Pig, but for starters I'd
prefer to keep, if possible, everything as close to the hadoop
own custom types here (note that you are restricted by size
of byte):
http://hadoop.sourcearchive.com/documentation/0.20.2plus-pdfsg1-1/AbstractMapWritable_8java-source.html
Regards,
Shahab
On Sat, Aug 31, 2013 at 5:38 AM, Adrian CAPDEFIER
chivas314...@gmail.comwrote:
Thank you for your
, 2013 at 3:42 AM, Shahab Yunus shahab.yu...@gmail.comwrote:
What I meant was that you might have to split or redesign your logic or
your usecase (which we don't know about)?
Regards,
Shahab
On Fri, Aug 30, 2013 at 10:31 PM, Adrian CAPDEFIER chivas314...@gmail.com
wrote:
But how would
Howdy,
I apologise for the lack of code in this message, but the code is fairly
convoluted and it would obscure my problem. That being said, I can put
together some sample code if really needed.
I am trying to pass some metadata between the map reduce steps. This
metadata is read and generated
at 9:01 PM, Adrian CAPDEFIER
chivas314...@gmail.comwrote:
Howdy,
I apologise for the lack of code in this message, but the code is fairly
convoluted and it would obscure my problem. That being said, I can put
together some sample code if really needed.
I am trying to pass some metadata
