See inline.
On 12/06/2013 01:02 AM, Steven Núñez wrote:
Gents,
I’m starting an install from BigTop 0.70: HDFS, YARN and Hue, with the
goal of building up from there to a minimal stack. In theory, this
should be as simple as ‘yum install hadoop\* hue\*’; in practice this
turns out to be surprisingly broken. For example, after the yum install
hue is reporting a configuration error:
hadoop.mapred_clusters.default.hadoop_mapred_home Current value:
/usr/lib/hadoop-0.20-mapreduce
Path does not exist on the filesystem.
Nowhere is this being set from the files that BigTop installed. Why is
Hue looking for MR1 stuff?
Attempting to fix another misconfiguration reported by hue:
hadoop.hdfs_clusters.default.webhdfs_url Current value: None
Failed to access filesystem root
Packages do not configure services for you. Not because it is broken,
but because it is not their responsibility.
In order to configure services correctly, packages would have to know
about the cluster topology and probably a few more things.
The closest thing you could find to that are the "pseudo conf"
configurations which assume everything is run locally.
In your example, you are just looking at the default Hue's configuration.
If you want to deploy Apache Bigtop on multiple nodes, I would recommend
you to take a look at the puppet recipes, which do configure services.
Even if you do not use puppet, you can still look at the configuration
templates to help you derive one for your cluster.
Would that be helpful to you if more packages were to provide "pseudo
conf" configurations where everything is assumed to run locally?
Is simple, however when restarting the daemons with: "for i in
hadoop-hdfs-namenode hadoop-hdfs-datanode ; do service $i restart ;
done", both daemons fail to restart (say, what happened to
start-dfs.sh?). Extracts from the logs show the reasons:
* Datanode because:java.net.BindException: Problem binding to
[0.0.0.0:50010] java.net.BindException: Address already in use
* Namenode because:java.io.IOException: Cannot lock storage
/var/lib/hadoop-hdfs/cache/hdfs/dfs/name. The directory is already
locked
So what I’ve done is a fresh installed, modified two config files, as
described in the hue configuration section
<http://cloudera.github.io/hue/docs-3.5.0/manual.html#usage>, and tried
to restart the HDFS daemons and they’re failing. This probably isn’t
what I’d call ‘working’.
start-dfs.sh is not meant to be used with packages. It would not work.
Can you describe *exactly* what you did from the beginning along with
the output?
The easier it is to reproduce, the easier it will be to fix/help you :)
I know that BigTop is at 0.70, but then again version numbers don’t mean
much these days. Is this kind of error to be expected after a fresh
install? When will it be safe to assume that a simple stack (HDFS, YARN,
Hue, Oozie, Zookeeper) installation would be working ‘out of the box’
without loads of manual configuration?
Perhaps I’m missing the point (I’m definitely missing the
documentation), but at this stage it seems BigTop’s primary advantage is
to ensure that the collection of packages is version compatible. Would
that be fair to say? I’m not unappreciative of the value there, but just
want to set expectations with people on what they’re getting with BigTop
in its current state. I recall some discussion of configuration before,
and there were somewhat different opinions. One that I recall was (my
words, summarizing my understanding):
BigTop doesn’t do any configuration, that’s all left to the
individual packages; BigTop just places them on the filesystem in
(somewhat, there’s still .cfg, .ini, .conf files) a consistent manner.
Would that be a fair statement?
Cheers,
- Steve
The packages do not set the configuration. But the Apache Bigtop puppet
recipes will set the configuration.
Packages will, as you describe, put files on the filesystem in a
consistent manner but also create users, and other tasks to provide a
rich integration with the system (init scripts, ulimit, etc.).
Most devs on this list probably either use the puppet recipes or have a
set of config files ready to be reused for their different installations.
So we do not always encounter the same issues as you do from a fresh start.
A hackathon is coming up this week end and I would like to tackle some
tasks to make Apache Bigtop more user friendly. So besides some more
consistent documentation and pseudo conf packages, would there be
anything else you would like to see happening?
Thanks,
Bruno
