Hi Shahab,
Sorry about the late reply, a personal matter came up and it took most of
my time. Thank you for your replies.
The solution I chose was to temporarily transfer the metadata along with
the data and then restore it on the reduce nodes. This works from a
functional perspective as long as
Thank you for your help Shahab.
I guess I wasn't being too clear. My logic is that I use a custom type as
key and in order to deserialize it on the compute nodes, I need an extra
piece of information (also a custom type).
To use an analogy, a Text is serialized by writing the length of the
Personally, I don't know a way to access job configuration parameters in
custom implementation of Writables ( at least not an elegant and
appropriate one. Of course hacks of various kinds be done.) Maybe experts
can chime in?
One idea that I thought about was to use MapWritable (if you have not
Howdy,
I apologise for the lack of code in this message, but the code is fairly
convoluted and it would obscure my problem. That being said, I can put
together some sample code if really needed.
I am trying to pass some metadata between the map reduce steps. This
metadata is read and generated
I think you have to override/extend the Comparator to achieve that,
something like what is done in Secondary Sort?
Regards,
Shahab
On Fri, Aug 30, 2013 at 9:01 PM, Adrian CAPDEFIER chivas314...@gmail.comwrote:
Howdy,
I apologise for the lack of code in this message, but the code is fairly
But how would the comparator have access to the job config?
On Sat, Aug 31, 2013 at 2:38 AM, Shahab Yunus shahab.yu...@gmail.comwrote:
I think you have to override/extend the Comparator to achieve that,
something like what is done in Secondary Sort?
Regards,
Shahab
On Fri, Aug 30, 2013
What I meant was that you might have to split or redesign your logic or
your usecase (which we don't know about)?
Regards,
Shahab
On Fri, Aug 30, 2013 at 10:31 PM, Adrian CAPDEFIER
chivas314...@gmail.comwrote:
But how would the comparator have access to the job config?
On Sat, Aug 31, 2013
