I guess the last thing I would add is a bit of QA around the init scripts and other installed items. They seem to be frayed around the edges.
On 2013-12-6 18:42 , "Bruno Mahé" <[email protected]> wrote: >See inline. > > >On 12/06/2013 01:02 AM, Steven Núñez wrote: >> Gents, >> >> I¹m starting an install from BigTop 0.70: HDFS, YARN and Hue, with the >> goal of building up from there to a minimal stack. In theory, this >> should be as simple as Œyum install hadoop\* hue\*¹; in practice this >> turns out to be surprisingly broken. For example, after the yum install >> hue is reporting a configuration error: >> >> hadoop.mapred_clusters.default.hadoop_mapred_home Current value: >> /usr/lib/hadoop-0.20-mapreduce >> Path does not exist on the filesystem. >> >> >> Nowhere is this being set from the files that BigTop installed. Why is >> Hue looking for MR1 stuff? >> >> Attempting to fix another misconfiguration reported by hue: >> >> hadoop.hdfs_clusters.default.webhdfs_url Current value: None >> Failed to access filesystem root >> >> > >Packages do not configure services for you. Not because it is broken, >but because it is not their responsibility. >In order to configure services correctly, packages would have to know >about the cluster topology and probably a few more things. >The closest thing you could find to that are the "pseudo conf" >configurations which assume everything is run locally. >In your example, you are just looking at the default Hue's configuration. >If you want to deploy Apache Bigtop on multiple nodes, I would recommend >you to take a look at the puppet recipes, which do configure services. >Even if you do not use puppet, you can still look at the configuration >templates to help you derive one for your cluster. > > >Would that be helpful to you if more packages were to provide "pseudo >conf" configurations where everything is assumed to run locally? > > > >> Is simple, however when restarting the daemons with: "for i in >> hadoop-hdfs-namenode hadoop-hdfs-datanode ; do service $i restart ; >> done", both daemons fail to restart (say, what happened to >> start-dfs.sh?). Extracts from the logs show the reasons: >> >> * Datanode because:java.net.BindException: Problem binding to >> [0.0.0.0:50010] java.net.BindException: Address already in use >> * Namenode because:java.io.IOException: Cannot lock storage >> /var/lib/hadoop-hdfs/cache/hdfs/dfs/name. The directory is already >> locked >> >> So what I¹ve done is a fresh installed, modified two config files, as >> described in the hue configuration section >> <http://cloudera.github.io/hue/docs-3.5.0/manual.html#usage>, and tried >> to restart the HDFS daemons and they¹re failing. This probably isn¹t >> what I¹d call Œworking¹. >> > >start-dfs.sh is not meant to be used with packages. It would not work. > >Can you describe *exactly* what you did from the beginning along with >the output? >The easier it is to reproduce, the easier it will be to fix/help you :) > > > >> I know that BigTop is at 0.70, but then again version numbers don¹t mean >> much these days. Is this kind of error to be expected after a fresh >> install? When will it be safe to assume that a simple stack (HDFS, YARN, >> Hue, Oozie, Zookeeper) installation would be working Œout of the box¹ >> without loads of manual configuration? >> >> Perhaps I¹m missing the point (I¹m definitely missing the >> documentation), but at this stage it seems BigTop¹s primary advantage is >> to ensure that the collection of packages is version compatible. Would >> that be fair to say? I¹m not unappreciative of the value there, but just >> want to set expectations with people on what they¹re getting with BigTop >> in its current state. I recall some discussion of configuration before, >> and there were somewhat different opinions. One that I recall was (my >> words, summarizing my understanding): >> >> BigTop doesn¹t do any configuration, that¹s all left to the >> individual packages; BigTop just places them on the filesystem in >> (somewhat, there¹s still .cfg, .ini, .conf files) a consistent >>manner. >> >> >> Would that be a fair statement? >> >> Cheers, >> - Steve >> >> > >The packages do not set the configuration. But the Apache Bigtop puppet >recipes will set the configuration. > >Packages will, as you describe, put files on the filesystem in a >consistent manner but also create users, and other tasks to provide a >rich integration with the system (init scripts, ulimit, etc.). > >Most devs on this list probably either use the puppet recipes or have a >set of config files ready to be reused for their different installations. >So we do not always encounter the same issues as you do from a fresh >start. >A hackathon is coming up this week end and I would like to tackle some >tasks to make Apache Bigtop more user friendly. So besides some more >consistent documentation and pseudo conf packages, would there be >anything else you would like to see happening? > > >Thanks, >Bruno
