Hi Harsh
I rechecked the code, it seems to be fine :(
public static final String HDFS_STOPWORD_LIST = "/data/stop_words.txt";
public static final String HDFS_POSITIVE_LIST =
"/data/Positive_Words.txt";
public static final String HDFS_NEGATIVE_LIST =
"/data/Negative_Words.txt";
public static final String HDFS_SEARCH_LIST =
"/data/Search_Criteria_Twitter.txt";
@Override
public void configure(JobConf conf) {
try {
super.configure(conf);
String stopwordCacheName = new
Path(HDFS_STOPWORD_LIST).getName();
String positiveCacheName = new
Path(HDFS_POSITIVE_LIST).getName();
String negativeCacheName = new
Path(HDFS_NEGATIVE_LIST).getName();
String searchCacheName = new
Path(HDFS_SEARCH_LIST).getName();
sLogger.debug("*********Inside Configure");
Path [] cacheFiles =
DistributedCache.getLocalCacheFiles(conf);
if (null != cacheFiles && cacheFiles.length > 0) {
for (Path cachePath : cacheFiles) {
if
(cachePath.getName().equals(stopwordCacheName)) {
loadStopWords(cachePath);
break;
}
if
(cachePath.getName().equals(positiveCacheName)) {
loadPositiveWords(cachePath);
break;
}
if
(cachePath.getName().equals(negativeCacheName)) {
loadNegativeWords(cachePath);
break;
}
if
(cachePath.getName().equals(searchCacheName)) {
loadSearchWords(cachePath);
break;
}
}
}else {
System.out.println("Got no cache files...");
}
} catch (IOException ioe) {
System.err.println("IOException reading from distributed
cache");
System.err.println(ioe.toString());
}
}
Regards,
Shreya
-----Original Message-----
From: Harsh J [mailto:ha...@cloudera.com]
Sent: Thursday, September 15, 2011 6:23 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Error calling config method in Mapper
Shreya,
On Thu, Sep 15, 2011 at 5:37 PM, <shreya....@cognizant.com> wrote:
> I used @override does not seem to work.
Its just to guarantee that you're actually overriding things. Its a
good practice, not a trouble fixer at all times :)
> Configure is not getting called as I populate some collections there
that are used later in the mapper and I am getting those collections as
blank in the mapper.
FWIW, here's my results of your snippet, and it apparently looks like
configure() is called just fine:
https://gist.github.com/bae64045e55377089afd. I think the trouble is
with your code, on how you're trying to load the distributed cache
elements out and back in. My guess is that you haven't initialized
some of the variables you may be .equals()-ing against and thereby
they're being treated as nulls?
Its definitely not a Hadoop-side issue afaict.
--
Harsh J
This e-mail and any files transmitted with it are for the sole use of the
intended recipient(s) and may contain confidential and privileged information.
If you are not the intended recipient(s), please reply to the sender and
destroy all copies of the original message. Any unauthorized review, use,
disclosure, dissemination, forwarding, printing or copying of this email,
and/or any action taken in reliance on the contents of this e-mail is strictly
prohibited and may be unlawful.
