Hi Rajen,
you can write stuff to the task attempt directory and it will be included in
the output of your MapReduce job.
You can get the directory from the Mapper context:
FileOutputFormat.getWorkOutputPath(context)
In that path, you can just open new files via the FileSystem methods.
Hope th
On Friday 30 September 2011 04:17 PM, Rajen Bhatt (RBEI/EST1) wrote:
Dear Experts:
We are scaling down an image to few partitions and processing each
partition in the MAP class. We want to save the output of MAP class
only to demonstrate and present how this output sub images are looking
like.
Arsalan,
This isn't a HBase question. This belongs on
mapreduce-user@hadoop.apache.org lists. I'm moving it there, and lets
carry on on that list. I've added you to cc in case you are not
subscribed to the mentioned list :)
Also, to reply to your original question - No, there isn't anything in
Ha
I have a small hadoop task which is running out of memory on a colleague's
cluster.
I looked at has mapred-site.xml and find
mapred.child.java.opts
-Xmx512M
mapred.child.ulimit
1572864
I set mapred.child.java.opts -Xmx1024M
but in my cluster there is no value for map
The ulimit should be set to 1.5 times the heap. One thing to note is the
unit is on KB.
-Joey
On Sep 30, 2011 1:24 PM, "Steve Lewis" wrote:
> I have a small hadoop task which is running out of memory on a colleague's
> cluster.
> I looked at has mapred-site.xml and find
>
>
> mapred.child.java.o
We are planning to enable secure Hadoop using Kerberos.
Our users reside in the active directory. We read that there are two options
to use Kerberos for securing Hadoop.
1) You run Kerberos on machine local to the cluster and create service
principals here
2) Use Active Directory itself as the k
Hi,
Is there a way to stop an entire job when a certain condition is met in the
map/reduce function? Like looking for a particular key or value.
Thanks,
Praveen
Not sure what you are trying, or if MR is the right tool...
Anyway, the basic idea is to write a map-only job with a custom RecordReader
to stop emitting inputs to your map when the mapper/recordreader finds the
needle in the haystack.
Arun
Sent from my iPhone
On Sep 30, 2011, at 8:39 PM, Prave
Arun,
Let's say there is 1PB of data in HDFS and the requirement is to look for a
particular sequence in the whole data. If the sequence is found then the MR
job could be aborted. There is no meaning in keeping the Map/Reduce tasks
running.
There was a query in one of the forums on how a MR job c
