[
https://issues.apache.org/jira/browse/HADOOP-2578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558286#action_12558286
]
Thomas Garner commented on HADOOP-2578:
---------------------------------------
I'm not sure I understand your logic. Are you saying that when hbase is
reading hbase-default and hbase-site, that it's not finding them on disk (they
are there and properly configured in /usr/local/hadoop/src/contrib/hbase/conf
on both of my mapreduce nodes), and then injecting defaults from somewhere
other than the xml files?
To test this, I edited hbase-default on one node, and I am now seeing the
change (hbase.master to default:60000):
2008-01-12 19:56:59,675 DEBUG org.apache.hadoop.mapred.TaskTracker: Child
starting
2008-01-12 19:57:00,038 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=MAP, sessionId=
2008-01-12 19:57:00,284 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1
2008-01-12 19:57:00,353 INFO org.apache.hadoop.conf.Configuration:
HBaseConfiguration after super(c): hbase.master: master:60000
2008-01-12 19:57:00,420 INFO org.apache.hadoop.conf.Configuration:
HBaseConfiguration after hbase-default.xml: hbase.master: default:60000
2008-01-12 19:57:00,420 INFO org.apache.hadoop.conf.Configuration:
HBaseConfiguration after hbase-site.xml: hbase.master: default:60000
2008-01-12 19:57:00,428 DEBUG org.apache.hadoop.hbase.HConnectionManager:
DEBUG: HConnection getConnection(): hbase.master: default:60000
2008-01-12 19:57:00,438 INFO org.apache.hadoop.conf.Configuration:
HBaseConfiguration after super(c): hbase.master: default:60000
2008-01-12 19:57:00,444 DEBUG
org.apache.hadoop.hbase.HConnectionManager$TableServers: No servers for
wordcount. Doing a find...
2008-01-12 19:57:10,119 INFO
org.apache.hadoop.hbase.HConnectionManager$TableServers: Attempt 0 of 5 failed
with . Retrying after sleep of 10000
2008-01-12 19:57:20,132 INFO
org.apache.hadoop.hbase.HConnectionManager$TableServers: Attempt 1 of 5 failed
with . Retrying after sleep of 10000
2008-01-12 19:57:30,144 INFO
org.apache.hadoop.hbase.HConnectionManager$TableServers: Attempt 2 of 5 failed
with . Retrying after sleep of 10000
2008-01-12 19:57:40,153 INFO
org.apache.hadoop.hbase.HConnectionManager$TableServers: Attempt 3 of 5 failed
with . Retrying after sleep of 10000
Here's the log from an earlier mr job/task, where you see the original
hbase.master value be correct (master:60000), then change to local (as was the
value in the hbase-default file):
2008-01-12 19:16:49,005 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=MAP, sessionId=
2008-01-12 19:16:49,253 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1
2008-01-12 19:16:49,333 INFO org.apache.hadoop.conf.Configuration:
HBaseConfiguration after super(c): hbase.master: master:60000
2008-01-12 19:16:49,403 INFO org.apache.hadoop.conf.Configuration:
HBaseConfiguration after hbase-default.xml: hbase.master: local
2008-01-12 19:16:49,404 INFO org.apache.hadoop.conf.Configuration:
HBaseConfiguration after hbase-site.xml: hbase.master: local
2008-01-12 19:16:49,421 INFO org.apache.hadoop.conf.Configuration:
HBaseConfiguration after super(c): hbase.master: local
2008-01-12 19:16:49,453 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: localhost/127.0.0.1:60000. Already tried 1 time(s).
2008-01-12 19:16:50,458 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: localhost/127.0.0.1:60000. Already tried 2 time(s).
2008-01-12 19:16:51,462 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: localhost/127.0.0.1:60000. Already tried 3 time(s).
2008-01-12 19:16:52,466 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: localhost/127.0.0.1:60000. Already tried 4 time(s).
I don't have an issue allowing mr nodes to read hbase default and site configs
(say for optimizations applicable to all hbase mr jobs that are pushed to disk
for every node), so long as config values specific to my job override any
default values.
I'm surprised that there's no set method in the configuration class that takes
a configuration as a parameter, then all you'd have to do is super on the
default, then set the site and set the passed config, each overriding any
existing parameters.
> HBaseConfiguration(Configuration c) constructor shouldn't overwrite passed
> conf with defaults
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-2578
> URL: https://issues.apache.org/jira/browse/HADOOP-2578
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Reporter: Thomas Garner
>
> While testing out mapreduce with hbase, in the map portion of a task, the map
> would try to connect to an hbase master at localhost/127.0.0.1. The config
> passed to the hbaseconfiguration contained the necessary hbase configuration
> information, but I assume was being overwritten by the defaults in the config
> files during addResource, as commenting out addHbaseResources in the
> constructor fixed the symptom. I would expect the configs to be layered on
> top of each other, e.g. default, then site, then the config passed as a
> parameter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
