Kris,
Here's sample PathFilter which can been configured, The only thing you need
to do is add the following line to configure the job
*job.getConfiguration().set("pathfilter.pattern", "your_patter");*
Not sure whether this is what you want.
public class MyPathFilter implements PathFilter ,Con
Whoops, so much for that idea. The Configuration instance being passed
to setConf is null.
I am utterly baffled. Is there seriously nobody out there using
PathFilter in this way? Everyone's just using dumb PathFilter
instances that don't have any configurable functionality?
/me boggles.
Kris
On
I just dove into the source, and it looks like the PathFilter instance
is instantiated using ReflectionUtils, and setConf is called so if the
resulting PathFilter instance implements Configurable, then
configuration will be available.
Kris
On Mon, Apr 12, 2010 at 1:52 PM, Kris Nuttycombe
wrote:
static void setInputPathFilter(Job job, Class filter)
This indicates that reflection will be used to instantiate the
required PathFilter object, and I need to be able to access the
minimum and maximum date for a given run. I don't want to have to
implement a separate PathFilter class for each
Thanks Arun!
From: Arun C Murthy [mailto:a...@yahoo-inc.com]
Sent: Wednesday, April 07, 2010 4:22 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Setting Committer for a map reduce job
In the new context-objects api the OutputFormat has the necessary apis to setup
the OutputCommitter.
Arun
Hi Philip,
So, there is no way to get the jobtracker metrics through jmx?
Currently, I use hyperic (which uses jmx) to monitor HDFS. Basically, anything
with jmx, can be monitored through hyperic.
-Harold
--- On Fri, 4/9/10, Philip Zeyliger wrote:
From: Philip Zeyliger
Subject: Re: map re
Hi Kris,
Do you mean you want to use the PathFilter in map or reduce task ? Or you
mean using the PathFilter in InputFormat ?
I guess you mean the second case, if so you only need to call
FileInputFormat.setInputPathFilter(,) to provide the filter information.
On Mon, Apr 12, 2010 at 8:13 AM, K
Hi, all, quick question about using PathFilter.
Is there any way to provide information from the job configuration to
a PathFilter instance? In my case, I want to limit the date range of
the files being selected by the filter, and don't want to have to
hard-code a separate PathFilter instance for
